We've built Luigi at Spotify to solve a lot of similar problems: https://github.com/spotify/luigi https://github.com/spotify/luigi Might be worth checking out

We've built Luigi at Spotify to solve a lot of similar problems: https://github.com/spotify/luigi https://github.com/spotify/luigi Might be worth checking out if you are building large data flows. We probably run 10k Luigi "tasks" every day, of which the majority is Hadoop jobs. They are all organized in a large dependency graph expressed within Python, and you also get visualization, exception handling, atomic file operation, etc.