Since you asked... If you can model your data flow as pipelines that run when their inputs update, you might interested in Pachyderm (https://www.pachyderm.io/

Since you asked... If you can model your data flow as pipelines that run when their inputs update, you might interested in Pachyderm (https://www.pachyderm.io/ https://www.pachyderm.io/). This uses Kubernetes and Docker images to run multi-stage computations. It's still a bit rough around the edges. (I wouldn't recommend the current version if you work with multi-gigabyte binary input files, though they're currently working on fixing an issue affecting that use case.) They do have some examples (http://pachyderm.readthedocs.io/en/latest/examples/readme.html http://pachyderm.readthedocs.io/en/latest/examples/readme.ht...), including one using TensorFlow. We're experimenting with Pachyderm at work, and the design feels "right" for our use case.