queue Package

queue Package

queue_adapter Module

This module contains contracts for defining adapters to various queueing systems, e.g. PBS/SLURM/SGE.

class fireworks.queue.queue_adapter.QScriptTemplate(template)

Bases: string.Template

delimiter = '$$'
pattern = <_sre.SRE_Pattern object at 0x7fdcd6b0c190>
class fireworks.queue.queue_adapter.QueueAdapterBase

Bases: dict, fireworks.utilities.fw_serializers.FWSerializable

The QueueAdapter is responsible for all interactions with a specific queue management system. This includes handling all details of queue script format as well as queue submission and management.

A user should extend this class with implementations that work on specific queue systems.

defaults = {}
classmethod from_dict(m_dict)
get_njobs_in_queue(username=None)

returns the number of jobs in the queue, probably using subprocess or shutil to call a command like ‘qstat’. returns None when the number of jobs cannot be determined.

Parameters:username – (str) the username of the jobs to count (default is to autodetect)
get_qlogger(name)
get_script_str(launch_dir)

returns a (multi-line) String representing the queue script, e.g. PBS script. Uses the template_file along with internal parameters to create the script.

Parameters:launch_dir – (str) The directory the job will be launched in
submit_to_queue(script_file)

submits the job to the queue, probably using subprocess or shutil

Parameters:script_file – (str) name of the script file to use (String)
template_file = 'OVERRIDE_ME'
to_dict(*args, **kwargs)

queue_launcher Module

This module is used to submit jobs to a queue on a cluster. It can submit a single job, or if used in “rapid-fire” mode, can submit multiple jobs within a directory structure. The details of job submission and queue communication are handled using Queueadapter, which specifies a QueueAdapter as well as desired properties of the submit script.

fireworks.queue.queue_launcher.launch_rocket_to_queue(launchpad, fworker, qadapter, launcher_dir='.', reserve=False, strm_lvl='INFO')

Submit a single job to the queue.

Parameters:
  • launchpad – (LaunchPad)
  • fworker – (FWorker)
  • qadapter – (QueueAdapterBase)
  • launcher_dir – (str) The directory where to submit the job
  • reserve – (bool) Whether to queue in reservation mode
  • strm_lvl – (str) level at which to stream log messages
fireworks.queue.queue_launcher.rapidfire(launchpad, fworker, qadapter, launch_dir='.', nlaunches=0, njobs_queue=10, njobs_block=500, sleep_time=None, reserve=False, strm_lvl='INFO')

Submit many jobs to the queue.

Parameters:
  • launchpad – (LaunchPad)
  • fworker – (FWorker)
  • qadapter – (QueueAdapterBase)
  • launch_dir – directory where we want to write the blocks
  • nlaunches – total number of launches desired; “infinite” for loop, 0 for one round
  • njobs_queue – stops submitting jobs when njobs_queue jobs are in the queue
  • njobs_block – automatically write a new block when njobs_block jobs are in a single block
  • sleep_time – (int) secs to sleep between rapidfire loop iterations
  • reserve – (bool) Whether to queue in reservation mode
  • strm_lvl – (str) level at which to stream log messages

Table Of Contents

This Page