dalio.base package¶
Submodules¶
dalio.base.builder module¶
Define extra utility classes used throughout the package
These classes implement certain interfaces used in specific cases and are not constrained an object’s parent class.
dalio.base.constants module¶
Define constant terms
In order to maintain name integrity throughout graphs, constants are used instead of any string name for variables that were created or will be usued in any _Transformer instance before or after the current one. These are often column names for pandas DataFrames, though can be anything that is or will be used to identify data throughout the graph.
dalio.base.datadef module¶
Defines DataDef base class
DataDef instances describe data inputs throughout the graph and ensure the integrity of data continuously. These are composed of various validators that serve both to describe approved data and check for whether data passes a test.
dalio.base.memory module¶
Defines memory transformers
-
class
dalio.base.memory.
LazyRunner
(mem_type, args=None, kwargs=None, buff=1, update=False)¶ Bases:
dalio.base.transformer._Transformer
Memory manager created to set memory input of an object after executing a transformer with given kwargs.
This is useful when you want to store data sourced from a transformer but doesn’t know which kwarg requests will be used. This object waits for a run request to source data from a transformer and set the source of one or more memory objects (in order) when data does arrive, and reutilize it if data is requested with the same kwargs.
For every new and valid evaluation, a new Memory instance is created and saved as a value in the _memory dictionary. Failed memory storage will result in no new Memory instance being created. This is done instead of simply setting inputs to Memory instances created upon initialization in order to reduce the memory usage of LazyRunner instances.
KEEP IN MIND that this does not check if the actual input is the same to relay the data, only the kwargs (for speed’s sake). This creates the risk that inputs or transformer attributes are changed (keeping kwarg requests the same) and the old data is retrieved. Use the pop() or clear() methods to solve this.
KEEP IN MIND there is a risk of having very different inputs being retrieved (from different external sources or date filters, for example) only based on kwarg requests. This is only relevant if _buff > 1.
-
_source
¶ transformer to source data from. No data definitions are used as this should be performed by the memory type uppon setting an input.
- Type
_Transformer
-
_mem_type
¶ type object for generating new memory instances.
- Type
type
-
_args
¶ tuple of arguments for new _mem_type instance initialization
- Type
tuple
-
_kwargs
¶ dict of keyword arguments for new _mem_type instance initialization
- Type
dict
-
_memory
¶ deque containing one Memory instance for every unique kwargs ran with a (kwarg, Memory) tuple structure.
- Type
deque
-
_buff
¶ Maximum number of Memory instances to be stored at any point. Positive numbers will be this limit, -1 represent no buffer limits. This option should be used with caution, as it can be highly memory-inneficient. Subclasses can create new methods of managing this limit.
- Type
int, -1 or >0
-
_update
¶ Whether _memory dict should be updated if a new set of kwargs is ran after reaching maximum capacity (as defined by the _buff attribute). If set to True, the last element of the _memory dict will be substituted.
- Type
bool
-
clear
()¶ Clear memory
-
copy
(*args, **kwargs)¶ Return a copy of this instance with a shallow memory dict copy
-
run
(**kwargs)¶ Compare kwargs with existing keys, update or set _memory in accordance to _update and _buff attributes.
- Raises
BufferError – if new kwargs, buffer is full and update set to False
-
set_buff
(buff)¶ Set the _buff attribute
-
set_input
(new_input)¶ Set the input data source.
- Parameters
new_input (_Transformer) – new transformer to be set as input.
- Raises
TypeError – if new_input is not an instance of _Transformer.
-
set_update
(update)¶ Set the _update attribute
-
with_input
(new_input)¶ Return copy of this transformer with the new input connection.
- Returns
Copy of self with new input.
-
-
class
dalio.base.memory.
LocalMemory
¶ Bases:
dalio.base.memory.Memory
Stores memory in the local session
-
clear
()¶ Clear memory
-
run
(**kwargs)¶ Return data stored in source variable
If data can be coppied, it will. This might not be memory efficient, but it makes behaviour from the Memory._source attribute more consistent with external memory sources.
-
set_input
(new_input)¶ Store input data into source variable
-
-
class
dalio.base.memory.
Memory
¶ Bases:
dalio.base.transformer._Transformer
Implement mechanics to store and retrieve input data.
This is a pseudo-transformer, as it is supposed to behave like on on the surface (implementing all needed methods) but not actually performing any actual transformation.
This is used in pipes that heavily reutilize the same external data source using the same kwarg requests. Implementations store and retrieive data through different methods and locations, and might implement certain requirements that must be met by input data in order for it to be stored.
-
_def
¶ Connection-less data definition that checks for required characteristics of of input data.
- Type
_DataDef
-
_source
¶ Memory source. Implementations will often have additional attributes to manage this source.
- Type
any
-
clear
()¶ Clear memory
-
copy
(*args, **kwargs)¶ Create new instance and memory source
-
run
(**kwargs)¶ Check if location is set and return stored data accordingly
-
set_input
(new_input)¶ Store input data
-
with_input
(new_input)¶ Return copy of this transformer with the new input connection.
- Returns
Copy of self with new input.
-
dalio.base.node module¶
Defines Node abstract class
Nodes are the key building blocks of your model as they represent any data that passes thorugh it. These are usued in subsequent classes to describe and manage data.
dalio.base.transformer module¶
Define Transformer class
Transformers are a base class that represents any kind of data modification. These interact with DataOrigin instances as they are key to their input and output integrity. A set_source() method sets the source of the input, the .run() method cannot be executed if the input”s source is not set.
Module contents¶
import classes