DAGrunner is an abstraction layer for executing Directed Acyclic Graphs (DAGs), providing portability and standardization across different schedulers. Rather than building your graph execution logic around a specific scheduler (such as Dask or Ray), DAGrunner enables you to define your graph once in NetworkX format and execute it with any supported scheduler without modifying your underlying graph definition. It decouples the scientific configuration and graph recipe from execution concerns, ensuring that your graph remains independent of scheduling technology choices. Additionally, DAGrunner provides a unified tooling layer that includes scheduler-agnostic graph visualization, scheduler-independent logging and monitoring, and other operational utilities, allowing you to maintain consistent workflows regardless of which back-end scheduler you choose.
DAGrunner takes advantage of the native markdown rendering support provided by github. To that end, all documentation of DAGrunner resides in markdown files.
See DAGrunner API
(C) Crown Copyright, Met Office. All rights reserved.
This file is part of 'DAGrunner' and is released under the BSD 3-Clause license. See LICENSE in the root of the repository for full licensing details.
The package is pip installable.
pip install .
(uninstall: pip uninstall dagrunner)
This will also make an executable script available to the PATH: dagrunner-execute-graph
usage: dagrunner-execute-graph [-h] [--scheduler SCHEDULER] [--num-workers NUM_WORKERS] [--profiler-filepath PROFILER_FILEPATH] [--dry-run] [--verbose] networkx-graph
see dagrunner-execute-graph --help for more information.
See docs/demo.ipynb
DAGrunner concerns itself with graph execution and does not strictly require processing modules (plugins) to take any particular form. That is, you may or may not choose to use or subclass the plugins provided by DAGrunner. However, for convenience, DAGrunner does define some plugins which fall into two broad categories, some abstract and some for use as they are.
See here for more information.
The dagrunner-execute-graph script exposes a scheduler argument for specifying our preferred scheduler. DAGrunner provides a layer of abstraction for schedulers. This enables a range of schedulers to be selected as per requirement.
These range from dask, ray to our own in-house multiprocessing asynchronous scheduler (built upon the multiprocessing library). See command help for further details.
DAGrunner provides a script dagrunner-logger for running a TCP server. This enables logging to function across the network. Additionally, it will write logs to an sqlite database to aid in realtime monitoring from external tools.
See logger for more information.
Colour:
B&W:
