Class and Function Documentation

A source that monitors an email inbox and yields new unread emails.

This source periodically checks for new unread emails, marks them as read,
and yields their content and metadata. It connects using IMAP and can be
configured to poll at specific intervals.

Args:
    poll_interval_minutes (int, optional): Minutes between email checks. Defaults to 10.
    folder (str, optional): Mailbox folder to check. Defaults to 'INBOX'.
    mark_as_read (bool, optional): Whether to mark emails as read. Defaults to True.
    limit (int, optional): Maximum number of emails to fetch per check. Defaults to 100. 
        if -1, fetch all.
    imap_server (str, optional): IMAP server address. If None, uses config.
    email_address (str, optional): Email address. If None, uses config.
    password (str, optional): Password. If None, uses config.
    
Yields:
    dict: Email metadata and content including:
        - message_id: Unique message ID
        - subject: Email subject
        - from: Sender address
        - to: Recipient address(es)
        - cc: CC address(es)
        - date: Datetime object of when email was sent
        - date_str: Date string from email header
        - plain_text: Plain text content if available
        - html_content: HTML content if available
        - headers: Dictionary of all email headers
        - raw_email: Full raw email content

Send emails for each item in the input iterable using SMTP.

This function processes a list of items and sends an email for each one, using the specified
fields for subject and body content. It supports both HTML and plain text email formats.

Args:
    subject_field (str): Field name in the item to use as email subject
    body_fields (list[str]): List of field names to include in email body
    sender_email (str, optional): Sender's email address. If None, uses config value
    recipient_email (str, optional): Recipient's email address. If None, uses config value
    smtp_server (str, optional): SMTP server address. Defaults to 'smtp.gmail.com'
    port (int, optional): SMTP server port. Defaults to 587

Yields:
    item: Returns each processed item after sending its corresponding email

Raises:
    AssertionError: If subject_field or body_fields are None
    ValueError: If required fields are missing in items

Example:
    >>> items = [{'title': 'Hello', 'content': 'World'}]
    >>> for item in sendEmail(items, 'title', ['content'], 'sender@email.com', 'recipient@email.com'):
    ...     print(f"Processed {item}")

Notes:
    - Requires valid SMTP credentials in config
    - Supports HTML formatting in email body
    - Uses TLS encryption for email transmission

Read and extract text from Microsoft Word (.docx) files.

This function takes an iterable of file paths to .docx documents and yields the
extracted text content from each document, with paragraphs joined by spaces.

Yields:
    str: The full text content of each document with paragraphs joined by spaces

Raises:
    Exception: If there are issues reading the .docx files

Example:
    >>> paths = ['doc1.docx', 'doc2.docx']
    >>> for text in readdocx(paths):
    ...     print(text)

Reads text files from given file paths and yields their contents.

Args:
    file_paths (Iterable[str]): An iterable containing paths to text files to be read.

Yields:
    str: The contents of each text file.

Raises:
    FileNotFoundError: If a file path does not exist.
    IOError: If there is an error reading any of the files.

Example:
    >>> files = ['file1.txt', 'file2.txt']
    >>> for content in readtxt(files):
    ...     print(content)

Download a URL segment and return its content.

This function is a wrapper around downloadURL that specifically handles URL segments.
It attempts to download content from the specified URL with configurable error handling
and timeout settings.

Args:
    fail_on_error (bool, optional): If True, raises exceptions on download errors.
        If False, returns None on errors. Defaults to True.
    timeout (int, optional): The timeout in seconds for the download request. 
        Defaults to 10 seconds.

Returns:
    bytes|None: The downloaded content as bytes if successful, None if fail_on_error
        is False and an error occurs.

Raises:
    Various exceptions from downloadURL function when fail_on_error is True and
    an error occurs during download.

Converts HTML content to text segment.

This function takes HTML content and converts it to plain text format.
If cleanText is enabled, the resulting text will also be cleaned so it 
tries to retain only the main body content.

Args:
    raw (str): The raw HTML content to be converted
    cleanText (bool, optional): Whether to clean and normalize the output text. Defaults to True.
    field (str): The field name to be used for the segment. If None, assuming the incoming item is html.
    append_as (str): The name of the field to append the text to.  If None, just pass on the cleaned text.

Returns:
    str: The extracted text content from the HTML

See Also:
    htmlToText: The underlying function used for HTML to text conversion

Generator function that monitors and yields new entries from an RSS feed.

This function continuously monitors an RSS feed at the specified URL and yields new entries
as they become available. It uses a SQLite database to keep track of previously seen entries
to avoid duplicates.

Args:
    url (str): The URL of the RSS feed to monitor.  If None, the URL is read from the config using
        the key "RSS_URL"
    db_path (str, optional): Path to the SQLite database file for storing entry history.
        Defaults to ':memory:' for an in-memory database.
    poll_interval_minutes (int, optional): Number of minutes to wait between polling
        the RSS feed for updates. Defaults to 10 minutes.

Yields:
    dict: New entries from the RSS feed, containing feed item data.

Example:
    >>> for entry in rss_source("http://example.com/feed.xml"):
    ...     print(entry["title"])

Filter items using a Bloom Filter to yield only distinct elements based on specified fields.

A Bloom Filter is a space-efficient probabilistic data structure used to test whether 
an element is a member of a set. False positive matches are possible, but false 
negatives are not.

Args:
    items (iterable): Input items to filter.
    capacity (int): Expected number of items to be added to the Bloom Filter.
    error_rate (float): Acceptable false positive probability (between 0 and 1).
    field_list (str, optional): Dot-separated string of nested fields to use for 
        distinctness check. Defaults to "_" which uses the entire item.

Yields:
    item: Items that have not been seen before according to the Bloom Filter.

Example:
    >>> items = [{"id": 1, "name": "John"}, {"id": 2, "name": "John"}]
    >>> list(distinctBloomFilter(items, 1000, 0.01, "name"))
    [{'id': 1, 'name': 'John'}]  # Only first item with name "John" is yielded

Note:
    Due to the probabilistic nature of Bloom Filters, there is a small chance
    of false positives (items incorrectly identified as duplicates) based on
    the specified error_rate.

Links the input stream to a threaded queue system.

This segment takes an input stream and links it to a threaded queue system.
It starts the queue system and then starts yielding from the queue.  That way
the upstream units don't have to wait for the downstream segments to draw 
from them.

Fills null (None) values in a sequence of dictionaries with specified defaults.

This generator function processes dictionaries by replacing None values with either
a general default value or specific values for named fields.

Args:
    items: An iterable of dictionaries to process.
    default (str, optional): The default value to use for any None values not 
        specified in kwargs. Defaults to ''.
    **kwargs: Field-specific default values. Each keyword argument specifies a
        field name and the default value to use for that field.

Yields:
    dict: The processed dictionary with None values replaced by defaults.

Raises:
    AssertionError: If any item in the input is not a dictionary.
    TypeError: If any item doesn't support item assignment using square brackets.

Examples:
    >>> data = [{'a': None, 'b': 1}, {'a': 2, 'b': None}]
    >>> list(fill_null(data, default='N/A'))
    [{'a': 'N/A', 'b': 1}, {'a': 2, 'b': 'N/A'}]
    
    >>> list(fill_null(data, b='EMPTY'))
    [{'a': None, 'b': 1}, {'a': 2, 'b': 'EMPTY'}]

Transform items by applying regex pattern replacement.

This segment transforms items by applying a regex pattern replacement to either
the entire item (if field="_") or a specific field of the item.

Args:
    items (Iterable): Input items to transform.
    pattern (str): Regular expression pattern to match.
    replacement (str): Replacement string for matched patterns.
    field (str, optional): Field to apply transformation to. Use "_" for entire item. Defaults to "_".

Yields:
    Union[str, dict]: Transformed items. Returns string if field="_", otherwise returns modified item dict.

Raises:
    TypeError: If extracted value is not a string or if item is not subscriptable when field != "_".

Examples:
    >>> list(regex_replace(["hello world"], r"world", "everyone"))
    ['hello everyone']
    
    >>> list(regex_replace([{"text": "hello world"}], r"world", "everyone", field="text"))
    [{'text': 'hello everyone'}]

Appends the specified fields to the input item.

Equivalent to toDict except that that item is modified with the new key/value pairs 
rather than a new dictionary returned.

Assumes that the input item can has items assigned using bracket notation ([]).

Concatenates specified fields from each item with a delimiter.

    Args:
        items: Iterable of input items to process
        fields: String specifying fields to extract and concatenate
        delimiter (str, optional): String to insert between concatenated fields. Defaults to "

"
        append_as (str, optional): If specified, adds concatenated result as new field with this name. 
                                Defaults to None.

    Yields:
        If append_as is specified, yields the original item with concatenated result added as new field.
        Otherwise, yields just the concatenated string.
    

Yields every nth item from the input stream.

Args:
    items: Iterable of items to process
    n: Number of items to skip between each yield

Yields:
    Every nth item from the input stream.

Execute a command and yields each line passed to stdout as an item.

Fill a template string with values from the input item.

Args:
    item: The input item containing values to fill the template
    template (str): The template string with placeholders for values

Returns:
    str: The filled template string

Yields the first n items from the input stream.
Args:
    n (int): The number of items to yield.
Yields:
    The first n items from the input stream.

Flattens a nested list of items.

Args:
    items: Iterable of items to flatten

Yields:
    Flattened list of items

Filters items based on whether a field contains a specified value.

Args:
    items: Iterable of items to filter
    field: Field name to check for value
    value: Value to check for in the field

Yields:
    Items where the specified field contains the specified value.

Filters items based on whether a field does not contain a specified value.

Args:
    field: Field name to check for value
    value: Value to check for in the field

Yields:
    Items where the specified field does not contain the specified value.

Finds the longest string among specified fields in the input item.  If 
a field is not present or is not a string, it is ignored.  If two or more
fields have the same length, the first one encountered is returned.  If
none of the specified fields are present, and emptry string is yielded.
Args:
    items: The input items
    field_list (str): Comma-separated list of fields to check for longest string
Yields:
    The longest string found in the specified fields of the input items.

Prints a tick marks to help visualize progress.

Prints a tick mark for each tick_count items processed. If eol_count is specified, it will print a new line after every eol_count tick marks.
If print_count is True, it will print the total count of items processed at the end of each line and at the end.

Args:
    items (Iterable): An iterable of items to process.
    tick (str): The character to print as a tick mark. Defaults to '.'.
    tick_count (int): The number of items to process before printing a tick mark. Defaults to 10.
    eol_count (Optional[int]): The number of tick marks to print before starting a new line. If None, no new line is printed. Defaults to 10.
    print_count (bool): If True, prints the count of items processed at the end of each line and at the end.
Yields:
    The original items from the input iterable.

Sleep for a specified number of seconds.

Args:
    items (Iterable): An iterable of items to process.
    seconds (int): The number of seconds to sleep.

Yields:
    None: This segment does not yield any items; it simply sleeps.

Slices a sequence using start and end indices.

This function takes a sequence and a range string in the format "start:end" to slice the sequence.
Both start and end indices are optional.

Args:
    item: Any sequence that supports slicing (e.g., list, string, tuple)
    range (str, optional): String in format "start:end" where both start and end are optional.
        For example: "2:5", ":3", "4:", ":" are all valid. Defaults to None.

Returns:
    The sliced sequence containing elements from start to end index.
    If range is None, returns a full copy of the sequence.

Examples:
    >>> slice([1,2,3,4,5], "1:3")
    [2, 3]
    >>> slice("hello", ":3")
    "hel"
    >>> slice([1,2,3,4,5], "2:")
    [3, 4, 5]

Drains the input stream and dumps each item as a jsonl string.
    

A source that generates input from a string.

This source will generate input from a string, splitting it on a delimiter.

Reads each item from the input stream, interpreting it as a jsonl string. 
    
    

Reads each item from the input stream as a path to a jsonl file. Loads each line of
each file as a json object and yields each individually.

Writes each item into a pickle file. If first_only is True, only the first item is written.
In any event, all items are yielded.

Args:
    fname (str): The name of the file to write.
    first_only (bool): If True, only the first item in the input stream is written.

Writes each item into a files after casting it to a string.

Args:
    fname (str): The name of the file to write.
    new_line (bool): If True, a new line will be written after each item.
    first_only (bool): If True, the segment will write only the first item in the input stream.
        In any event, all items will be yielded.

Generate a range of integers between lower (inclusive) and upper (exclusive)

This segment wraps the built-in range function, allowing you to specify
the lower and upper bounds of the range. The range is inclusive of the
lower bound and exclusive of the upper bound.

Args:
    lower (int): Lower bound of the range (inclusive)
    upper (int): Upper bound of the range (exclusive)

Generate n random integers between lower and upper.

Scale each item in the input stream by the multiplier.

Segment to add a vector to the SimpleVectorDB.

Args:
    item: The item containing the vector data.
    vector_field: The field containing the vector data.
    vector_id: Optional custom ID for the vector.
    metadata_field_list: Optional metadata field list.
    dimension: Expected dimension of the vector (optional).

Returns:
    The ID of the added vector.

Segment to search for similar vectors in the SimpleVectorDB.
Args:
    vector_field: The field containing the vector data.
    top_k: Number of top results to return.
    search_metric: Similarity metric ("cosine" or "euclidean").
    search_method: Search method ("brute-force", "brute-force-heap", or "k-means").
    path: Optional path to a saved vector database.
Yields:
    List of SearchResult objects.

Index documents using Whoosh full-text indexing.

Args:
    items: Iterator of items to index
    index_path (str): Path to the Whoosh index directory.
    field_list (list[str]): List of fields to index.
    yield_doc (bool): If True, yield each indexed document. Otherwise yield the original item.
    continue_on_error (bool): If True, continue processing other documents when one fails.
    overwrite (bool): If True, clear existing index before indexing.
    commit_seconds (int): If > 0, commit changes if it has been this many seconds since the last commit.

Search documents using Whoosh full-text indexing.

Args:
    queries: Iterator of query strings
    index_path (str): Path to the Whoosh index directory.
    limit (int): Maximum number of results to return for each query. Defaults to 100.
    all_results_at_once (bool): If True, yield all results at once. Otherwise, yield one result at a time.
    continue_on_error (bool): If True, continue with next query when one fails.
    reload_seconds (int): If > 0, reload the index if the last search was at least this many seconds ago.