4 ms·
What is a workflow language exactly? What benefits do they bring vs using say Python?
by vector_spaces 7y ago
What is a workflow language exactly? What benefits do they bring vs using say Python?
- jkh1 7y agoWorkflow languages are meant to formally describe computational data processing pipelines involving many steps and tools. They are often associated with an engine that allows setting up, running and monitoring of a workflow.
- danielecook 7y agoThe two main things you want out of a workflow language are re-entrancy and a DAG of job dependencies. Re-entrancy is the ability for the workflow to pick up where it left if something crashes (basically via caching or detecting the presence of expected output files). The DAG is a directed acyclic graph of job dependencies: First do A, then B, then C. The DAG is worked out by the workflow manager, and jobs can be managed accordingly. A good workflow manager builds on these ideas further by managing environments, job submission, parallelization, cloud/cluster submission, and other options that make processing large amounts of data a lot easier and more efficient.