load package

Submodules

load.load_file_to_ckan module

load.load_file_to_ckan.find_resource_id_if_exists(url, dataset_name, file_name)
load.load_file_to_ckan.main()
load.load_file_to_ckan.parser()

Parser function to run arguments from commandline and to add description to sphinx docs. To see possible styling options: https://pythonhosted.org/an_example_pypi_project/sphinx.html

load.load_file_to_ckan.upload_file_to_ckan(url, dataset_name, file_path)

Upload a file to the CKAN datastore.

Args:
  1. url: url of the catalog:

    https://api.data.amsterdam.nl/catalogus
    
  2. dataset_name: name of the dataset, which can be found on the ckan page url:

    https://api.data.amsterdam.nl/catalogus/dataset/afvalcontainers
    
  3. api_key: your private user key, which can be found on the user profile page.

  4. file_path: location of the file including filename:

    /path/to/file/to/upload.csv
    
Returns:
An uploaded file to the CKAN datastore.

load.load_file_to_objectstore module

load.load_file_to_objectstore.check_existence_object(connection, container_path, filename)

Check if the file is present on the objectstore container_path,

Args:
  1. connection = Objectstore connection based on from helpers.connection import objectstore_connection
  2. container_path = Name of container/prefix/subfolder
  3. filename = Name of file, for example test.csv
Returns:
  • ‘The object was successfully created’
  • ‘The object was not found’
  • ‘Error finding the object’
load.load_file_to_objectstore.get_object(connection, container_path, filename, output_folder)

Download file from objectstore container.

Args:
  1. connection: Objectstore connection based on from helpers.connection import objectstore_connection
  2. container_path: Name of container/prefix/subfolder
  3. filename: Name of file, for example test.csv
  4. output_folder: Define the path to write the file to for example app/data when using docker.
Returns:
A file from the objectstore into the specified output_folder.
load.load_file_to_objectstore.main()
load.load_file_to_objectstore.parser()

Parser function to run arguments from commandline and to add description to sphinx docs.

load.load_file_to_objectstore.put_object(connection, container: str, filename: str, contents, content_type: str) → None

Upload a file to objectstore.

Args:
  1. container: path/in/store
  2. filename: your_file_name.txt
  3. contents: contents of file with use of with open(‘your_file_name.txt’, ‘rb’) as contents:
  4. content_type:’text/csv’,’application/json’, … Is retrievd by using the mime package.
Returns:
A saved file in the container of the objectstore.
load.load_file_to_objectstore.upload_file(connection, container_path, filename_path)

Upload file to the objectstore.

Args:
  1. connection = Objectstore connection based on from helpers.connection import objectstore_connection
  2. container_path = Name of container/prefix/subfolder, for example Dataservices/aanvalsplan_schoon/crow
  3. filename_path = full path including the name of file, for example: data/test.csv

Uses mime for content_type: https://stackoverflow.com/questions/43580/how-to-find-the-mime-type-of-a-file-in-python

Result:
Uploads a file to the objectstore and checks if it exists on in the defined container_path.

load.load_wfs_to_postgres module

exception load.load_wfs_to_postgres.NonZeroReturnCode

Bases: Exception

Used for subprocess error messages.

load.load_wfs_to_postgres.load_layers(pg_str)

Load layers into Postgres using a list of titles of each layer within the WFS service.

Args:

pg_str: psycopg2 connection string:

'PG:host= port= user= dbname= password='
Returns:
Loaded layers into postgres using ogr2ogr.
load.load_wfs_to_postgres.main()
load.load_wfs_to_postgres.parser()

Parser function to run arguments from commandline and to add description to sphinx.

load.load_wfs_to_postgres.run_command_sync(cmd, allow_fail=False)

Run a string in the command line.

Args:
  1. cmd: command line code formatted as a list:

    ['ogr2ogr', '-overwrite', '-t_srs', 'EPSG:28992','-nln',layer_name,'-F' ,'PostgreSQL' ,pg_str ,url]
    
  2. Optional: allow_fail: True or false to return error code

Returns:
Excuted program or error message.
load.load_wfs_to_postgres.scrub(line)

Hide the login credentials of Postgres in the console.

load.load_wfs_to_postgres.wfs2psql(url, pg_str, layer_name, **kwargs)

Command line ogr2ogr string to load a WFS into PostGres.

load.load_xls_to_postgres module

load.load_xls_to_postgres.load_xls(datadir, config_path, db_config_name)

Load xlsx into postgres for multiple files

load.load_xls_to_postgres.main()
load.load_xls_to_postgres.parser()

Module contents