4 ms·
(Toil contributor here) Workflow systems are generally used to orchestrate parallel batch jobs on a cluster, which may have hundreds or thousands of individual
by tetron 10y ago
(Toil contributor here)
Workflow systems are generally used to orchestrate parallel batch jobs on a cluster, which may have hundreds or thousands of individual tasks. So the emphasis is on defining what units of work are independent that can run on separate nodes without needing to communicate with other tasks, and how outputs of one task feed into the inputs of another.
- SEJeff 10y agoCould Toil be extended to work on something like say Mesos maybe using the Marathon scheduler, or would you be better off to write a toil framework instead?
- braincode 10y agoAFAIK, there's some support in there already: """ Develop and test on your laptop then deploy on any of the following: Commercial clouds: - Amazon Web Services (including the spot market) - Microsoft Azure Private clouds: - OpenStack High Performance Computing Environments: - GridEngine - Apache Mesos - Parasol - Individual multi-core machines """ http://toil.readthedocs.io/en/latest/ http://toil.readthedocs.io/en/latest/