marv¶
Access control lists¶
Creating datasets¶
Datasets are created based on information provided by scanners. A scanner is responsible to group files into named datasets:
def scan(dirpath, dirnames, filenames):
return [DatasetInfo(os.path.basename(x), [x])
for x in filenames
if x.endswith('.csv')]
Scanners are called for every directory within the configured
scanroots, while files and directories starting with a .
and
directories containing an (empty) .marvignore
file are ignored and
will not be traversed into.
Further, traversal into subdirectories can be controlled by
altering the dirnames
list in-place. To block further
traversal, e.g. for a directory-based dataset type, set it to an
empty list – os.walk()
is used behind the scenes:
dirnames[:] = []
Declaring nodes¶
-
marv.
node
(schema=None, header=None, group=None, version=None)[source]¶ Turn function into node.
Parameters: - schema¶ – capnproto schema describing the output messages format
- header¶ – This parameter is currently not supported and only for internal usage.
- group¶ (bool) – A boolean indicating whether the default stream
of the node is a group, meaning it will be used to
published handles for streams or further groups. In case
of
marv.input.foreach
specifications this flag will default to True. This parameter is currently only for internal usage. - version¶ (int) – This parameter currently has no effect.
Returns: A
Node
instance according to the given arguments andinput()
decorators.
-
marv.
input
(name, default=None, foreach=None)[source]¶ Decorator to declare input for a node.
Plain inputs, that is plain python objects, are directly passed to the node. Whereas streams generated by other nodes are requested and once the handles of all input streams are available the node is instantiated.
Parameters: Returns: The original function decorated with this input specification. A function is turned into a node by the
node()
decorator.
Interacting with marv¶
-
marv.
create_stream
(name, **header)[source]¶ Create a stream for publishing messages.
All keyword arguments will be used to form the header.
-
marv.
pull
(handle, enumerate=False)[source]¶ Pulls next message for handle.
Parameters: Returns: A
Pull
task to be yielded. Marv will send the corresponding message as soon as it is available. For groups this message will be a handle to a member of the group. Members of groups are either streams or groups.Examples
Pulling (enumerated) message from stream:
msg = yield marv.pull(stream) idx, msg = yield marv.pull(stream, enumerate=True)
Pulling stream from group and message from stream:
stream = yield marv.pull(group) # a group of streams msg = yield marv.pull(stream)