Here you can find exact details regarding app’s components.
Components
Config
- class components.config.ConfigProtocol(*args, **kwargs)[source]
Protocol class for config object.
- Parameters:
domain (str) – argument for SFDC domain
reports_csv_path (str) – argument for input report csv path.
reports_list (list[dict[str, Any]]) – argument for list of reports as dictionary
summary_filepath (str) – argument for summary file path
log_filepath (str) – argument for log file path
report (str) – argument for single report parameters
path (str) – argument for save location path override
threads (int) – argument for number of threads to use
stdout_loglevel (str) – argument for stdout log level
file_loglevel (str) – argument for stdout log level
verbose (bool) – Flag, toggles progress bar/stdout
- class components.config.Config(*, domain: str, reports_csv_path: str, reports_list: list[dict[str, Any]] = [], summary_filepath: str | None = None, log_path: str | None = None, report: str = '', path: str = '', threads: int = 0, stdout_loglevel: str = 'WARNNING', file_loglevel: str = 'INFO', verbose=False)[source]
Concrete class representing Config object. Contains entire configuration required for a program.
- Parameters:
domain (str) – argument for SFDC domain
reports_csv_path (str) – argument for input report csv path.
reports_list (list[dict[str, Any]], optional) – argument for list of reports as dictionary, defaults to []
summary_filepath (str, optional) – argument for summary file path, defaults to None
log_filepath (str, optional) – argument for log file path, defaults to None
report (str, optional) – argument for single report parameters, defaults to “”
path (str, optional) – argument for save location path override, defaults to “”
threads (int, optional) – argument for number of threads to use, defaults to 0
stdout_loglevel (str, optional) – argument for stdout log level, defaults to “WARNING”
file_loglevel (str, optional) – argument for stdout log level, defaults to “INFO”
verbose (bool, optional) – Flag, toggles progress bar/stdout, defaults to False
- _define_number_of_threads()[source]
Defines number of threads. By default number of threads is set to half of available threads. If threads value is not available number of threds will be set to 2. If
report
parameter is filled (single report mode) then number of threads will be automatically set to 1.
- _input_report_path_cast(object_kwargs: list[dict[str, Any]]) list[dict[str, str | os.PathLike]] [source]
Casts value of
path
intoPath
object.- Parameters:
object_kwargs (list[dict[str, Any]]) – colection of report parameters
- Returns:
collection of object parameters with path casted to
Path
object- Return type:
list[dict[str, str |
os.PathLike
]]
- _input_report_single_mode_override() list[dict[str, str]] [source]
Creates report’s parameters in single report mode. Returns parsed parameters as object kwargs.
- Returns:
collection of report’s parameters in single report mode
- Return type:
list[dict[str, str]]
- _input_report_csv_standard_file_mode() list[dict[str, str]] [source]
Reads parameteres taken from input csv . Returns parsed parameters as objects kwargs.
- Returns:
collection of report’s parameters in standard mode
- Return type:
list[dict[str, str]]
- _input_report_path_override(object_kwargs: list[dict[str, str]]) list[dict[str, str]] [source]
Replaces value of path key in object_kwargs dict with path parameter value. path override.
- Parameters:
object_kwargs (list[dict[str, str]]) – Colection of report parameters
- Returns:
Collection of report’s parameters with replaced path
- Return type:
list[dict[str, str]]
Connectors
- class components.connectors.ConnectorProtocol(*args, **kwargs)[source]
Protocol class for connector object.
- Parameters:
queue (
Queue
) – shared queue objecttimeout (int) – request’s timeout value in seconds
headers (dict[str, str]) – headers required to establish the connection
- check_connection() bool [source]
Checks connection with given domain.
- Returns:
flag,
True
if connection is established,False
otherwise.- Return type:
bool
- async report_gathering(reports: list[components.containers.ReportProtocol], session: ClientSession) None [source]
Collects asynchronous responses from the servers.
- Parameters:
reports (list[
ReportProtocol
]) – collection ofReportProtocol
objects.session (
ClientSession
) – HTTP client session object to handle request in transaction.
- class components.connectors.SfdcConnector(*, queue: Queue, domain: str, verbose: bool = False, timeout: int = 900, headers: dict[str, str] = {'Content-Type': 'application/csv', 'X-PrettyPrint': '1'})[source]
Concrete class representing SFDC Connector object.
- Parameters:
queue (
Queue
) – shared queue objectdomain (str) – SalesForce domain of your organization -> “https://corp.my.salesforce.com/”
verbose – flag, parameter used as switch between progress bar and logging to stdout, defaults to False
timeout (int, optional) – Request’s timeout value in seconds, defaults to 900
headers (dict[str, str], optional) – headers required connection, defaults to {‘Content-Type’: ‘application/csv’, ‘X-PrettyPrint’: ‘1’}
export_params (str, optional) – GET request parameters required by SFDC, defaults to ‘?export=csv&enc=UTF-8&isdtp=p1’
- _convert_domain_for_cookies_lookup() str [source]
Converts domain string according to
sid
CookieJar lookup requirements.- Returns:
converted url complaiant with CookieJar schema
- Return type:
str
- _intercept_sid() str [source]
Intercepts sid from MS Edge’s CookieJar.
- Returns:
Intercepted
sid
or empty string ifsid
doesn’t exist.- Return type:
str
- _open_sfdc_site() None [source]
Opens SFDC website on given domain url if
sid
is not present or not valid.
- _sid_check() bool [source]
Checks
sid
valididty for given SFDC domain.- Returns:
Flag,
True
ifsid
was valid,False
when wasn’t.- Return type:
bool
- check_connection() bool [source]
Checks the connection with given domain and
sid
validity.- Returns:
flag,
True
if connection was successful,False
if wasn’t.- Return type:
bool
- _parse_report_url(report: ReportProtocol) str [source]
Parses report object url.
- Parameters:
report (
ReportProtocol
) – instance ofReportProtocol
- Returns:
parsed url
- Return type:
str
- async _request_report(report: ReportProtocol, session: ClientSession) None [source]
- Sends asynchronous request to given domain with given parameters within shared session. Checks response status:
- 200: response is saved in
ReportProtocol.response()
,ReportProtocol.valid()
set toTrue
,ReportProtocol
is being put to thequeue
.404: error in response,ReportProtocol.valid()
set toFalse
, no retries.500: request timeour,ReportProtocol.valid()
set toFalse
, another attempt.
- Parameters:
report (
ReportProtocol
) – instance ofReportProtocol
session (
aiohttp.ClientSession
) – shared session object
- async _toggle_progress_bar(tasks: list[_asyncio.Task]) None [source]
Toggles between showing progress bar or stdout logging.
- Parameters:
tasks (list[
asyncio.Task
]) – collection of asynchronous request tasks
- _create_async_tasks(reports: list[components.containers.ReportProtocol], session: ClientSession) list[_asyncio.Task] [source]
Creates collection of asynchronous request tasks.
- Parameters:
reports (list[
ReportProtocol
]) – collection ofReportProtocol
instancessession (
aiohttp.ClientSession
) – shared, asynchronous session
- Returns:
collection of asynchronous request tasks
- Return type:
list[
asyncio.Task
]
- async _report_request_all(reports: list[components.containers.ReportProtocol], session: ClientSession) None [source]
Orchestrates entire process of processing tasks.
- Parameters:
reports (list[
ReportProtocol
]) – collection ofReportProtocol
instancessession (
aiohttp.ClientSession
) – shared, asynchronous session
- async handle_requests(reports: list[components.containers.ReportProtocol]) None [source]
Creates session and processes asynchronous tasks.
- Parameters:
reports (list[
ReportProtocol
]) – collection ofReportProtocol
instances
Containers
- class components.containers.ReportProtocol(*args, **kwargs)[source]
Protocol class for report object.
- Parameters:
name (str) – report name, propagated to report file name
id (str) – report id
path (
os.PathLike
) – report path, save location for the report in form ofpathlib.Path
objecttype (str) – report type, drives connector selection
export_params (str) – default GET parameters
downloaded (bool) – flag indicating whether the reports has been succesfully downloaded or not
valid (bool) – flag indicating whether the response has been succesfully retrieved or not
created_date (datetime) – report save completition date
pull_date (timedelta) – report response completition date
processing_time – the time it took to process the report in seconds
attempt_count (int) – number of attempts to process the report
size (float) – size of saved report file in Mb
response (str) – container for request response
content (
pd.DataFrame
) – Pandaspd.DataFrame
based on response
- class components.containers.ReportsContainerProtocol(*args, **kwargs)[source]
Protocol class for report container object.
- Parameters:
report_params_list (list[dict[str, Any]]) – collection of dictionaries with parameters for object crafting
summary_report_path (
os.PathLike
) – path to save location of summary report
- create_reports() list[components.containers.ReportProtocol] [source]
Orchestrating method to handle report objects factory
- Returns:
collection of
ReportProtocol
objects- Return type:
list[
ReportProtocol
]
- create_summary_report() None [source]
Creates summary report which consist of all important details regarding
ReportProtocol
objects. Summary report is generated once all the reports are completed.
- class components.containers.SfdcReport(name: str, id: str, path: ~os.PathLike, type: str = 'SFDC', export_params: str = '?export=csv&enc=UTF-8&isdtp=p1', downloaded: bool = False, valid: bool = False, created_date: ~datetime.datetime = datetime.datetime(2023, 4, 7, 3, 40, 58, 22998), pull_date: ~datetime.datetime = datetime.datetime(2023, 4, 7, 3, 40, 58, 22998), processing_time: ~datetime.timedelta = datetime.timedelta(0), attempt_count: int = 0, size: float = 0.0, response: str = '', content: ~pandas.core.frame.DataFrame = <factory>)[source]
Concrete class representing SFDC Report object.
- Parameters:
name (str) – report name, propagated to report file name
id (str) – report id, identification number of the report in SFDC
path (
os.PathLike
) – report path, save location for the report in form ofpathlib.Path
objecttype (str, optional) – report type, type drives connector and report objects selectiond, defaults to ‘SFDC’
export_params (str, optional) – default GET parameters required by SFDC, defaults to ‘?export=csv&enc=UTF-8&isdtp=p1’
downloaded (bool, optional) – flag indicating whether the reports has been succesfully downloaded or not, defaults to False
valid (bool, optional) – flag indicating whether the response has been succesfully retrieved or not, defaults to False
created_date (datetime, optional) – report save completition date, defaults to current datetime
pull_date (timedelta, optional) – report response completition date, defaults to current datetime
processing_time – the time it took to process the report in seconds, defaults to 0 microseconds
attempt_count (int, optional) – number of attempts to process the report, defaults to 0
size (float, optional) – size of saved report file in Mb, defaults to 0.0
response (str, optional) – container for request response, defaults “”
content (
pd.DataFrame
, optional) – Pandas DataFrame based on response, defaults to emptypd.DataFrame
- class components.containers.ReportsContainer(reports_params_list: list[dict[str, Any]], summary_path: PathLike | None)[source]
Concrete class representing ReportContainer object.
- _create_sfdc_reports() Generator[SfdcReport, None, None] [source]
Creates
SfdcReport
objects.- Returns:
generator with
SfdcReport
objects- Return type:
Generator[
SfdcReport
, None, None]- Yield:
SfdcReport
instance based on parsed report parameters- Return type:
- create_reports() list[components.containers.ReportProtocol] [source]
Orchestrating method to handle report objects crafting
- Returns:
collection of
ReportProtocol
objects- Return type:
list[
ReportProtocol
]
- create_summary_report() None [source]
Creates summary report which consist of all important details regarding
ReportProtocol
objects. Summary is generated once all theReportProtocol
are completed.
- print_summary_table() None [source]
Prints summary table which consist of all important details regarding
ReportProtocol
objects. Table is generated once all theReportProtocol
are completed.
Handlers
- class components.handlers.WorkerFactoryProtocol(*args, **kwargs)[source]
Protocol class for worker factory objects.
- Parameters:
queue (
Queue
) – shared, thread-safe queue objectthreads (int) – number of threads, equal to number of
WorkerProtocol
to be deployed
- create_workers() None [source]
Creates
WorkerProtocol
on independent threads
- static active_workers() int [source]
Counts active
WorkerProtocol
in current time.- Returns:
number of active
WorkerProtocol
objects- Return type:
int
- class components.handlers.WorkerProtocol(*args, **kwargs)[source]
Protocol class for worker objects.
- Parameters:
queue (
Queue
) – shared, thread-safe queue object
- _read_stream(report: ReportProtocol) None [source]
Reads the stream of data kept in
ReportProtocol
object via Pandas read method. Deletes response content from the object.- Parameters:
report (
ReportProtocol
) – instance of theReportProtocol
object
- _save_to_csv(report: ReportProtocol) None [source]
Saves readed data to
csv
file using Pandas save method.- Parameters:
report (
ReportProtocol
) – instance of theReportProtocol
object
- _erase_report(report: ReportProtocol) None [source]
Erases the report data.
- Parameters:
report (
ReportProtocol
) – instance of theReportProtocol
object
- report_processing(report: ReportProtocol) None [source]
Orchiestrates the
ReportProtocol
processing.- Parameters:
report (
ReportProtocol
) – instance of theReportProtocol
object
- class components.handlers.WorkerFactory(*, queue: Queue, threads: int = 1)[source]
Concrete class representing WorkerFactory object.
- Parameters:
queue (
Queue
) – shared, thread-safe queue objectthreads (int, optional) – number of threads, equal to number of
WorkerProtocol
to be deployed, defaults to 1
- create_workers() None [source]
Deploys given number of
WorkerProtocol
.
- static active_workers() int [source]
Returns number of currently active
WorkerProtocol
.- Returns:
number of
WorkerProtocol
- Return type:
int
- class components.handlers.Worker(queue: Queue)[source]
Concrete class representing Worker object.
- _read_stream(report: ReportProtocol) None [source]
Reads
ReportProtocol
response
and save it ascontent
atribute. Erases savedresponse
.- Parameters:
report (
ReportProtocol
) – instance of theReportProtocol
object
- _parse_save_path(report: ReportProtocol) PathLike [source]
Parses path to save location.
- Parameters:
report (
ReportProtocol
) – instance of theReportProtocol
object.- Returns:
Path to save location
- Return type:
os.PathLike
- _save_to_csv(report: ReportProtocol) None [source]
Saves
ReportProtocol
content
tocsv
file. Sets object flags.- Parameters:
report (
ReportProtocol
) – instance of theReportProtocol
object
- _erase_report(report: ReportProtocol) None [source]
Deletes
ReportProtocol
objectcontent
.- Parameters:
report (
ReportProtocol
) – instance of theReportProtocol
object
- process_report(report: ReportProtocol) None [source]
Orchiestrates entire process of downloading the report.
- Parameters:
report (
ReportProtocol
) – instance of theReportProtocol
object