Synchronizers

nodeshot.interop.sync is a django-app that enables nodeshot to build an abstraction layer between itself and other third party web-applications which deal with georeferenced data.

There are mainly four strategies through which we can achieve interoperability with third party web apps:

  • periodic synchronization: data is imported periodically into the local database by a background job which reads an external source
  • event driven synchronization: data is exported to a third party API whenever local data is added, changed or deleted
  • RESTful translator: data is retrieved on the fly and converted to json/geojson, no data is saved in the database
  • mixed: custom synchronizers might implement mixed strategies

These strategies are implemented through “Synchronizers”.

New synchronizers can be written ad-hoc for each application that need to be supported.

Internal dependencies

To enable this feature, the following apps must be listed in settings.INSTALLED_APPS:

  • nodeshot.core.layers
  • nodeshot.core.nodes
  • nodeshot.interop.sync

By default these three apps are installed.

Required settings

The module nodeshot.interop.sync is activated by default in nodeshot.conf.settings.INSTALLED_APPS.

For periodic synchronization CELERYBEAT_SCHEDULE must be uncommented in your settings.py:

from datetime import timedelta

CELERYBEAT_SCHEDULE.update({
    'synchronize': {
        'task': 'nodeshot.interop.sync.tasks.synchronize_external_layers',
        'schedule': timedelta(hours=12),
    },
    # ... other tasks ...
})

You might want to tweak how often data is synchronized, in the previous example we configured the task to run every 12 hours.

Layer configuration

Interoperability is configured at layer level in the admin interface.

A layer must be flagged as “external”, after doing so a new box labeled “External layer info” will appear in the bottom of the page. If the layer is new and not saved in the database proceed to save and reload the admin page.

Then change the synchronizer field and the configuration fields will appear.

Each synchronizer has different fields, an brief explaination of the default synchronizers follows.

Nodeshot (RESTful translator)

This synchronizer is a RESTful translator and allows to reference the nodes of an external nodeshot instance.

There are two required configuration keys:

  • layer url: URL of the layer API resource, eg: https://test.map.ninux.org/api/v1/layers/rome/
  • verify ssl: indicates wether the SSL certificate of the external layer should be verified or not; if checked self signed certificates won’t work

There is no periodic synchronization needed because this synchronizer grabs the data on the fly.

GeoJSON (periodic sync)

This synchronizer implements the periodic synchronization strategy and therefore needs to be enabled in the CELERYBEAT_SCHEDULE setting.

The main configuration keys are:

  • url: URL to retrieve the geojson file
  • verify_ssl: indicates wether the SSL certificate of the external layer should be verified or not; if checked self signed certificates won’t work
  • default status: status to be used for new nodes, to use the system default leave blank

There are other configuration keys which enable to parse geojson files which use radically different names for corresponding fields.

  • name: corresponding name field, for example, on the data source file the name field could be labeled title
  • status: corresponding status field, if present
  • description: corresponding description field, if present
  • address: corresponding address field, if present
  • is_published: corresponding is_published field, if present
  • user: corresponding user field, if present
  • elev: corresponding elev field, if present
  • notes: corresponding notes field, if present
  • added: corresponding added field, if present
  • updated: corresponding updated field, if present

GeoRSS (periodic sync)

This synchronizer implements the periodic synchronization strategy and therefore needs to be enabled in the CELERYBEAT_SCHEDULE setting.

The main configuration keys are:

  • url: URL to retrieve the georss file
  • verify_ssl: indicates wether the SSL certificate of the external layer should be verified or not; if checked self signed certificates won’t work
  • default status: status to be used for new nodes, to use the system default leave blank

There are other configuration keys which enable to parse georss files which use radically different names for corresponding fields.

  • name: corresponding name field, defaults to title
  • status: corresponding status field, if present
  • description: corresponding description field, if present
  • address: corresponding address field, if present
  • is_published: corresponding is_published field, if present
  • user: corresponding user field, if present
  • elev: corresponding elev field, if present
  • notes: corresponding notes field, if present
  • added: corresponding added field, defaults to pubDate
  • updated: corresponding updated field, if present

OpenWisp (periodic sync)

This synchronizer inherits from the GeoRSS synchronizer, the available options and configurations are the same.

The only difference is that this synchronizer is designed to grab data from the GeoRSS file produced by OpenWISP Geographic Monitoring.

Sync management command

This is the command which is used to perform periodic synchronization, use --help to know its options:

python manage.py sync --help

Sync a specific layer:

python manage.py sync layer-slug

Sync multiple layers by specifying space separated layer slugs:

python manage.py sync layer1-slug layer2-slug

Sync all layers is as simple as:

python manage.py sync

Sync all layers except those specified in –exclude:

python manage.py sync --exclude=layer1-slug,layer2-slug

# spaces are allowed as long as string is wrapped in quotes/doublequotes

python manage.py sync --exclude="layer1-slug, layer2-slug"

Writing new synchronizers

To write new synchronizers, you should extend the class GenericGisSynchronizer in /nodeshot/interoperability/synchronizers/base.py:

# my_very_cool_app.py

from nodeshot.interop.sync.synchronizer.base import GenericGisSynchronizer

class MyVeryCoolApp(GenericGisSynchronizer):
    """ Synchronizer for my MyVeryCoolApp """
    pass

Note

this section is a work in progress.

Once the file is saved and you are sure it’s on your pythonpath you have to add a tuple in settings.NODESHOT_SYNCHRONIZERS in which the first element is the path to the file and the second element is the name you want to show in the admin interface in the “Synchronizer” select:

NODESHOT_SYNCHRONIZERS = [
    ('myproject.synchronizers.my_very_cool_app.MyVeryCoolApp', 'MyVeryCoolApp'),
]

This will add your new synchronizer to the default list.

Third party packages