This area of research is fascinating - my favorite part of research in this area last 6 months is finding DAGs everywhere. More or less Dataflow models work by

This area of research is fascinating - my favorite part of research in this area last 6 months is finding DAGs everywhere. More or less Dataflow models work by lifting the computational dependencies into data and breaking each compute step into a small enough piece so that they're composable and can replicate across machines arbitrarily. The dependencies between each step is usually modeled as some kind of DAG in a simple case the output of one step feeds into another. In more complex cases you have logical (if this then that) dependencies for error handling and branching cases. I think the Dataflow paper: https://www.vldb.org/pvldb/vol8/p1792-Akidau.pdf https://www.vldb.org/pvldb/vol8/p1792-Akidau.pdf and the MillWheel paper in particular are good reads about the problem: https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/41378.pdf https://static.googleusercontent.com/media/research.google.c... What's especially interesting to me is that the same approach is used for rule/workflow engines. Although the use-cases are somewhat different and the graph structure isn't usually the same - they're still modeling compute steps in a graph just with more conditional logic.