Modules

Data Processing contains 5 modules: 1. Authentication 2. Extract 3. Transform 4. Load 5. Helpers

Authentication

Fill in the config.ini.example with the proper user credentials for the project. Rename this file to config.ini. Do not store passwords in this file Use .gitignore to ignore .ini files to prevent uploading them to github.

Extract

Functions to extract data from the Objectstore where raw files are stored, from API’s or from the data catalog.

Transform Geospatial

Functions to spatial transform and enrich datasets.

Transform Enrichment

Functions to enrich datasets.

Load

Common functions to load data into the Objectstore, CKAN (our public data.amsterdam.nl) or PostgreSQL.

Helpers

Generic helper functions for most commonly used data operations.