Shameless plug for a similar but totally different system that I've used to pretty good effect lately: https://github.com/gaffo/jorb https://github.com/gaffo/jo

Shameless plug for a similar but totally different system that I've used to pretty good effect lately: https://github.com/gaffo/jorb https://github.com/gaffo/jorb meant for just trying to run batches of items with many steps in a simple way. A major benefit of my tool is that it ends up being pretty easy to re-wire the flows of tasks through the state tree. I tried for a ipynb type workflow as much as possible where I'm storing state on every state change, this lets me kill and restart the processing easily and often only losing the work that was in process on the in flight state changes. I find myself wiring up the first few steps of the state diagram and making the ones I haven't figured out yet temporariliy terminal. Then my computer is crunching away on pre-work and I let one item through at a time in the debugger, figure out that state, and move to the next state, and restart the flow. New items now start processing through the newly non-terminal state. I set breakpoints at all of the exceptional states and let it run working on the next step in the flow. I handle the exceptions and then open the flood gates with parallelism on the state, while also coding up the new state transitions. This lets my computer do a lot of work against things that have concurrency limits or rate limits while working, lowering the totally wall clock time of the large batch job. What do I use it for for reals? I've been using it to check out and modify large swaths of code bases and build them against a build fleet where I'm rate limited on the build submissions and the ability to pull the code down and modify it. Or where I need to do walking of a large number of expensive apis over time to do complex calculations on small (1-5k sets of data). I've always thought of building a dag renderer for the graph but I'm changing the graph during execution a lot, what I actually want is to render the state changes of a given task