sensortoolkit.testing_attrib_objs._referencemonitor.ReferenceMonitor
- class ReferenceMonitor(data_source=None, site_name=None, site_id=None)[source]
Bases:
object
Object for storing and accessing reference monitor (FRM/FEM) data as well as instrument and monitoring site attributes.
- Parameters
data_source (str, optional) –
The name of the data service that reference measurements were acquired from. Defaults to None.
Options:
airnow
: Reference data from the AirNow APIaqs
: Reference data from U.S. EPA’s Air Quality Systemairnowtech
: Reference data from AirNowTech, files downloaded locally.local
: A catch-all category for datasets stored locally on the user’s system.
site_name (str, optional) – The name of the air monitoring site. Defaults to None.
site_id (str, optional) – The unique identification number for the air monitoring site. Typically the AQS Site ID if applicable. Defaults to None.
- Raises
ValueError – If both the site name and site ID are not None but the data source is None, raise ValueError as the source must be specified to indicate where reference data are located. Also, a ValueError will be raised if the passed data source, site name, and site id do not point to a valid reference data subdirectory.
Methods
Return the name of the reference method (instrument name) for the passed parameter.
Load reference data from locally acquired files.
Send a data query to the AirNow API.
Send a data query to the AQS API.
Interactive method for configuring reference data ingestion.
Attributes
data_source
default_proj_path
project_path
- get_method_name(param)[source]
Return the name of the reference method (instrument name) for the passed parameter.
- Parameters
param (str) – The parameter for which the reference monitor name will be retrieved.
- Returns
Name of the reference monitor used to collect measurements for the passed parameter.
- Return type
method_name (str)
- load_data(bdate, edate, param_list, path=None, met_data=True)[source]
Load reference data from locally acquired files.
- Parameters
bdate (str) – The beginning date (“YYYY-MM-DD” format) for the sensor testing period.
edate (str) – The ending date (“YYYY-MM-DD” format) for the sensor testing period.
param_list (list) – A list of SDFS parameters measured by reference monitors at the monitoring site for which data will be loaded.
path (str, optional) –
Full path to the data directory where processed reference datasets are located. If None (default), will attempt to construct path from an existing setup configuration if the
sensortoolkit.ReferenceMonitor.reference_setup()
method was run prior to callingload_data()
.Note
If path is not null (i.e., a valid directory path is passed) and a reference monitor configuration was created with
sensortoolkit.ReferenceMonitor.reference_setup()
prior to runningload_data()
, the path will point to datasets located at the indicated path instead of datasets pointed to by the path configured by thereference_setup()
method.met_data (bool, optional) – If true, meteorological data will be loaded in addition to datasets corresponding to parameters passed to param_list. Defaults to true (U.S. EPA’s documents for recommended performance testing protocols, metrics, and target values encourage users to report meteorological conditions for sensor performance evaluations and reports).
- Returns
None.
- query_airnow(key, param_list, bdate, edate, bbox=None, bbox_size=0.01)[source]
Send a data query to the AirNow API.
- Parameters
key (str) – The API authentication key code.
param_list (list) – A list of SDFS parameters measured by reference monitors at the monitoring site for which data will be loaded.
bdate (str) – The beginning date (“YYYY-MM-DD” format) for the sensor testing period.
edate (str) – The ending date (“YYYY-MM-DD” format) for the sensor testing period.
bbox (dict, optional) –
A bounding box of coordinates within which data will be queried. Defaults to None.
bbox
accepts four key-value pair entries, where the following keys should be specified:'minLat'
: The northern-most latitude for the bounding box.'maxLat'
: The southern-most latitude for the bounding box.'minLong'
: The western-most longitude for the bounding box.'maxLong'
: The eastern-most longitude for the bounding box.
Values for each key should be type float.
bbox_size (float, optional) – DESCRIPTION. Defaults to 0.01.
- Returns
None.
- query_aqs(username, key, param_list, bdate, edate, site_id=None, query_met_data=True)[source]
Send a data query to the AQS API.
- Parameters
username (str) – The email account registered with the API service.
key (str) – The API authentication key code.
param_list (list) – A list of SDFS parameters measured by reference monitors at the monitoring site for which data will be loaded.
bdate (str) – The beginning date (“YYYY-MM-DD” format) for the sensor testing period.
edate (str) – The ending date (“YYYY-MM-DD” format) for the sensor testing period.
site_id (dict, optional) –
The AQS site ID for the air monitoring site from which reference measurements will be returned by the API. The site ID is passed as a dictionary with three key-value pairs:
'state'
: The two digit FIPS code for the state (leading zero included).'county'
: The three digit FIPS code for the county (leading zeros included) located within the state.'site'
: The four digit AQS site number within the county (leading zeros included).
Note
If site_id is not null, the site ID passed will be used instead of any previously configured reference monitor configuration via the
sensortoolkit.ReferenceMonitor.reference_setup()
method.query_met_data (bool, optional) – If true, meteorological data for temperature and relative humidity measurements will be queried in addition to parameters passed to param_list. Defaults to true (U.S. EPA’s documents for recommended performance testing protocols, metrics, and target values encourage users to report meteorological conditions for sensor performance evaluations and reports).
- Returns
None.
- reference_setup()[source]
Interactive method for configuring reference data ingestion.
Setup for Local Datasets:
Selecting the Reference Data Service or Source:
Choose the service or source from which reference data were acquired. Choose from the following options:
'local'
: Data files aqcuired locally (e.g., local transfer from agency overseeing reference instrumentation at air monitoring site).'airnowtech'
: User has downloaded files from the AirNowTech system and has saved files locally to the user’s system.'aqs'
: User will query EPA’s Air Quality System (AQS) API for reference data.'airnow'
: User will query the AirNow API for reference data.
Adding Monitoring Site Information:
Users input information about the ambient monitoring site at which the collocation study was conducted.
Selecting File Data Type:
Choose the corresponding data file type for recorded reference datasets from
'.csv'
,'.txt'
,'.xlsx'
.Selecting Data Files:
Choose the selection scheme for pointing to recorded data files from the following options:
'directory'
, which will locate and copy all of the data files in the specified directory for the indicated data type'recursive directory'
, which will locate and copy all data files within the specified directory and any subdirectories contained within the indicated folder path'files'
which copies over files that the user manually selects within a directory.
Copying Data Files:
Recorded reference datasets are copied from the selected file or folder location to the
../data/reference_data/[reference_source]/[site_name]_[site_id]/raw
directory path.Selecting the Column Header Index:
Users indicate the integer index position for the row of header data in reference datasets.
Parsing Reference Datasets:
The first few rows of recorded reference datasets located in the
../data/reference_data/[reference_source]/[site_name]_[site_id]/raw
directory path are imported and the names of column headers are located based on the indicated head index. A list of unique column headers is stored for subsequent reassignment of column header names.Specifying Timestamp Columns:
Users indicate the column(s) containing date/timestamp information.
Specifying the Parameter Renaming Scheme and Monitor Information:
Users indicate the SDFS parameters corresponding to column names discovered in step 5. This creates a parameter renaming dictionary for reassigning the names of header labels. Details about the reference monitors used to measure the selected SDFS parameters are also indicated at this step.
Configuring Timestamp Column Formatting:
Users indicate the date/time formatting for date/time column(s) indicated in step 6. Formatting should correspond to the
strftime
formatting keywords located at https://strftime.org/Specifying the DateTime Index Time Zone:
Users indicate the time zone associated with the date/time column(s). Timezones should be valid timezone names recognized by the
pytz
library.Saving the Setup Configuration to
setup.json
:The setup configuration specified by the user is saved to a
setup.json
file for subsequent use by the ingestion module for importing recorded reference datasets and conversion to SDFS datasets.Reference Data Ingestion and Saving Processed Datasets:
Recorded reference datasets are ingested via the configuration specified and processed version of these datasets that have been converted to SDFS format are saved as comma-separated value files to the
/data/reference_data/[source]/processed/[site_name]_[site_id]
.
Setup for AirNowTech Datsets:
Setup process is the same as steps #1-5 for the local data file setup process. Subsequently, these steps are followed by:
Pre-processing AirNowTech Datasets:
Local AirNowTech files that the user selected in the previous step are imported and processed versions of these datasets are ingested into the SDFS format via the
sensortoolkit.reference.preprocess_airnowtech()
method. Processed datasets are subsequently saved as comma-separated value files to/data/reference_data/airnowtech/processed/[site_name]_[site_id]
Saving the Setup Configuration to
setup.json
:The setup configuration specified by the user is saved to a
setup.json
file.
Setup for Querying the AQS API
Setup process is the same as steps 1 and 2 for the local data file setup process.
Important
Users must provide a site AQS ID in step 2 to successfully query the API service.
The setup configuration is subsequently saved to a
reference_setup.json
. Users should then query the AQS API via theReferenceMonitor.query_aqs()
method.Setup for Querying the AirNow API
Setup process is the same as steps 1 and 2 for the local data file setup process.
Important
In step 2, users must provide the latitude and longitude coordinates for the monitoring site where air sensors were collocated to successfully query the API service.
The setup configuration is subsequently saved to a
reference_setup.json
. Users should then query the AirNow API via theReferenceMonitor.query_airnow()
method.- Returns
None.