3 ms·
The two main things you want out of a workflow language are re-entrancy and a DAG of job dependencies. Re-entrancy is the ability for the workflow to pick up wh
by danielecook 7y ago
The two main things you want out of a workflow language are re-entrancy and a DAG of job dependencies. Re-entrancy is the ability for the workflow to pick up where it left if something crashes (basically via caching or detecting the presence of expected output files). The DAG is a directed acyclic graph of job dependencies: First do A, then B, then C. The DAG is worked out by the workflow manager, and jobs can be managed accordingly.
A good workflow manager builds on these ideas further by managing environments, job submission, parallelization, cloud/cluster submission, and other options that make processing large amounts of data a lot easier and more efficient.