Convenient syncing

Syncing blockchain data at first sight sounds simple but is actually a very complex task. A network client operates in a constantly changing environment where other nodes join and leave the network, are in different geographical locations, have a variety of hardware resources, have varying loads depending on the number of clients and their requests, have varying available bandwidth and possibly even face adversiary actors. In such an environment you want a process that can handle and recover from unexpected behaviour (be it invalid requests, no responses etc). The two classes discussed next do exactly that and allow the user to focus on post processing the Blocks instead of retrieval.

Introducing NodeManager and SyncManager.

Node manager

The node manager is responsible for establishing and maintaining and pool of active connections to healthy NEO nodes.

When started it begins by connecting to a seedlist obtained from the global settings object. It runs three services on an interval.

  1. a connection pool monitor to ensure it meets the configured mininum and maximum client settings and attempt to fill any open spots .

  2. an address list filler which asks for new addresses from connected nodes to ensure it always has a new node to connect to.

  3. a connection pool monitor to ensure that the remote nodes local blockchain height keeps advancing. If they are stuck they’ll be replaced by a new node and tagged for poor performance.

In the unlikely event that all nodes fail and there are no more new addresses to connect to, the node manager will recycle all addresses that it historically was able to connect to. This means addresses that it failed to connect to are not retried. Be aware that estbalishing a connection can take time and so recovery is not instantaneous. The recovery times can be tweaked by adjusting the MAX_NODE_POOL_ERROR and POOL_CHECK_INTERVAL class attributes.

A minimal usage example for individual use is

import asyncio
from neo3 import settings
from neo3.network import convenience
from neo3.core import msgrouter


def connection_done(node_client, failure):
    if failure:
        print(f"Failed to connect to {failure[0]} reason: {failure[1]}.")
    else:
        print(f"Connected to node {node_client.version.user_agent} @ {node_client.address}")

async def main():
    # set network magic to NEO MainNet
    settings.network.magic = 5195086

    # add a local node to the seedlist for the first connection
    settings.network.seedlist = ['127.0.0.1:40333']

    # listen to the connection event broad casted by the node manager
    msgrouter.on_client_connect_done += connection_done

    node_mgr = convenience.NodeManager()
    node_mgr.start()

    # keep alive
    while True:
        await asyncio.sleep(1)

if __name__ == "__main__":
    asyncio.run(main())

If you want the full syncing experience take a look at the example below.

class neo3.network.convenience.nodemanager.NodeManager(*args, **kwds)

Bases: neo3.network.convenience.singleton._Singleton

This class is a convenience class that helps establish and maintain a pool of active connections to NEO nodes.

Attention

This class is singleton.

async shutdown()

Gracefully shutdown the node manager.

Disconnects all active nodes and stops service tasks.

Note

This dependents on asyncio’s Task canceling logic. It waits for all tasks to be cancelled and/or stopped before returning.

Return type

None

start()

Start the node manager services. This does 2 things

  1. Connect to the seed list addresses provided in the configuration

  2. Try to maintain a pool of connected nodes according to the min/max clients configuration settings and monitor that they don’t get stuck.

Return type

None

ADDR_QUERY_INTERVAL = 15

Time interval in seconds for asking nodes for their address list.

MAX_NODE_ERROR_COUNT = 5

Maximum number of times adding a block or header may fail before the node is disconnected.

MAX_NODE_POOL_ERROR = 2

Maximum number of time the minimum connected client setting may be violated by the open connect checker.

MAX_NODE_TIMEOUT_COUNT = 15

Maximum number of times a timeout threshold may be exceeded before the node is disconnected. Requires calling increase_node_timeout_count() which is done automatically by the SyncManager if used.

MONITOR_HEIGHT_INTERVAL = 30

Time interval in seconds for calling the height monitoring check.

PING_INTERVAL = 2

Time interval in seconds for pinging remote nodes. Ping (and the Pong) response informs each other about chain heights

POOL_CHECK_INTERVAL = 10

Time interval in seconds to check the connection pool for open spots.

Sync manager

The sync manager is responsible for bringing the local blockchain in sync with the global blockchain and keeping it in sync.

The sync manager depends on NodeManager for providing it with healty nodes to request data from. This means it requires the node manager to be started ahead of the sync manager.

A full usage example is

import asyncio
from neo3 import settings
from neo3.network import convenience, payloads
from neo3.core import msgrouter

def connection_done(node_client, failure):
    if failure:
        print(f"Failed to connect to {failure[0]} reason: {failure[1]}.")
    else:
        print(f"Connected to node {node_client.version.user_agent} @ {node_client.address}")

def block_received(from_nodeid: int, block: payloads.Block):
    print(f"Received block with height {block.index}")

async def main():
    # set network magic to NEO MainNet
    settings.network.magic = 195086

    # add a local node to test against
    settings.network.seedlist = ['127.0.0.1:40333']

    # listen to the connection events broadcasted by the node manager
    msgrouter.on_client_connect_done += connection_done

    # listen to block received events
    msgrouter.on_block += block_received

    node_mgr = convenience.NodeManager()
    node_mgr.start()

    sync_mgr = convenience.SyncManager()
    await sync_mgr.start()

    # keep alive
    while True:
        await asyncio.sleep(1)

if __name__ == "__main__":
    asyncio.run(main())

Enable logging if you’re interested to see the internals in action.

import logging

stdio_handler = logging.StreamHandler()
stdio_handler.setLevel(logging.DEBUG)
stdio_handler.setFormatter(logging.Formatter("%(asctime)s %(levelname)s - %(module)s:%(lineno)s %(message)s"))

network_logger = logging.getLogger('neo3.network')
network_logger.addHandler(stdio_handler)
network_logger.setLevel(logging.DEBUG)
class neo3.network.convenience.syncmanager.SyncManager(*args, **kwds)

Bases: neo3.network.convenience.singleton._Singleton

async shutdown()

Gracefully shutdown the sync manager.

Stops block persisting and all service tasks.

Note

This dependents on asyncio’s Task canceling logic. It waits for all tasks to be cancelled and/or stopped before returning.

Return type

None

async start(timeout=5)

Start the block syncing service. Requires a started node manager.

Parameters

timeout – time in seconds to wait for finding a started node manager.

Raises

Exception – if no started Nodemanager is found within timeout seconds.

Return type

None

BLOCK_MAX_CACHE_SIZE = 500

Maximum number of Blocks to cache in memory. Block persisting empties the cache allowing new blocks to be requested.

BLOCK_NETWORK_REQ_LIMIT = 500

Maximum number of blocks to ask per request. Cannot exceed MAX_BLOCKS_COUNT.

BLOCK_REQUEST_TIMEOUT = 5

Maximum time in seconds that a node may take to respond to a data request before it is tagged and eventually replaced.