More examples for workflow packages in our field

[jan-janssen]

More or less domain specific

• https://docs.signac.io/en/latest/
• https://parsl.readthedocs.io/
• https://nomad-lab.eu/prod/v1/docs/howto/customization/workflows.html
• https://asr.readthedocs.io/en/latest/
• https://github.com/AgnostiqHQ/covalent
• https://www.prefect.io

General overview page https://workflows.community/systems

Other interesting articles:

• https://www.computer.org/csdl/magazine/co/2024/08/10632600/1ZgY6uYjw6A

[jan-janssen]

• https://github.com/meirwah/awesome-workflow-engines/
• https://github.com/pditommaso/awesome-pipeline

[jan-janssen]

Last year PerQueue was released. I took a look at their examples and it seems to be primarily file based rather than Python function based:

from pathlib import Path

from perqueue import PersistentQueue, Task, Workflow

here = Path(__file__).parent

# Create 3 distinct workflow tasks
t1 = Task(here / "step_1.py", {"N_samples": 1_000_000_000}, "local:10m")
t2a = Task(here / "step_2a.py", {"savefig_filename": "pi-fig.png"}, "local:1m")
t2b = Task(here / "step_2b.py", None, "local:1m")

# Define dependencies - t2a and t2b depend on t1
wf = Workflow({t1: [], t2a: [t1], t2b: [t1]})

# Submit the worflow through PerQueue
with PersistentQueue() as pq:
    pq.submit(wf)