3 ms·
I'm also a developer of a workflow processing system, though not open-source, and fairly specific to our company. A few more things that are desirable if you ha
by kisielk 14y ago
I'm also a developer of a workflow processing system, though not open-source, and fairly specific to our company. A few more things that are desirable if you have a lot of data or need to do processing that takes a lot of time is the ability to run stages in parallel, and also to distribute the computation over a cluster of machines.
- aboytsov 14y agoGreat points. Drake supports the ability to run stages in parallel (at least in theory) - it's been speced out (https://docs.google.com/document/d/1bF-OKNLIG10v_lMes_m4yyaJtAaJKtdK0Jizvi_MNsg/edit#heading=h.2r0uhxc https://docs.google.com/document/d/1bF-OKNLIG10v_lMes_m4yyaJ...), just not implemented yet. But of course, once you have the entire dependency graph, it's easy to know what can be run in parallel and what cannot. As for distributing computations, our approach is that it lies outside of Drake's scope. Drake doesn't know what's going on in steps. But you can always implement a step that would use distributed computation, for example, by submitting a Hadoop job, or in any other way. The only requirement Drake has is for the step to be synchronous, i.e. do not return before all the computation is complete. But even that can be changed for some cases.