4 ms·
Any time streaming data comes up, I want to point people towards some of the existing research and conceptual tooling under the name of "dataflow processing", w
by falcolas 7y ago
Any time streaming data comes up, I want to point people towards some of the existing research and conceptual tooling under the name of "dataflow processing", which is functionally equivalent to stream processing.
https://en.wikipedia.org/wiki/Dataflow https://en.wikipedia.org/wiki/Dataflow
There's a lot of interesting ideas which we can use to this day, albeit with some changes to work with modern programming languages and data encoding frameworks.
- mrdoops 7y agoThis area of research is fascinating - my favorite part of research in this area last 6 months is finding DAGs everywhere. More or less Dataflow models work by lifting the computational dependencies into data and breaking each compute step into a small enough piece so that they're composable and can replicate across machines arbitrarily. The dependencies between each step is usually modeled as some kind of DAG in a simple case the output of one step feeds into another. In more complex cases you have logical (if this then that) dependencies for error handling and branching cases. I think the Dataflow paper: https://www.vldb.org/pvldb/vol8/p1792-Akidau.pdf https://www.vldb.org/pvldb/vol8/p1792-Akidau.pdf and the MillWheel paper in particular are good reads about the problem: https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/41378.pdf https://static.googleusercontent.com/media/research.google.c... What's especially interesting to me is that the same approach is used for rule/workflow engines. Although the use-cases are somewhat different and the graph structure isn't usually the same - they're still modeling compute steps in a graph just with more conditional logic.
- dmux 7y agoIs DAG in this context "Directed acyclic graph"?
- mpfundstein 7y agoIndeed