3 ms·
This looks nice. Taking this one step further, is there a solution in Python where I can describe a set of tasks and their inter-dependencies and the "co-ordina
by noisy_boy 9y ago
This looks nice. Taking this one step further, is there a solution in Python where I can describe a set of tasks and their inter-dependencies and the "co-ordinator" can execute tasks as per the dependencies (including taking defined outputs of one task and feed it to the next child task). This sort of "orchestration" would be very useful e.g. in scenarios like gateway apis which need to pull data from multiple sources and combine the results as a single response.
- rekwah 9y agoMaybe something like Common Workflow Language[0]? [0] http://www.commonwl.org/v1.0/UserGuide.html http://www.commonwl.org/v1.0/UserGuide.html
- stenl 9y agoLuigi from Spotify is very nice. We use it for computational pipelines in our neuroscience research. https://github.com/spotify/luigi/blob/master/README.rst https://github.com/spotify/luigi/blob/master/README.rst
- batbomb 9y agoairflow, pegasus/htcondor, luigi, pinball.
- IanCal 9y agoTo add another option, you might also want to check out dask. The various options all have pros and cons, and fit different problems better or worse. It's worth trying a few different ones out to see what's best for you. http://dask.pydata.org/en/latest/ http://dask.pydata.org/en/latest/