5 ms·
Indirectly answering the question - I've skimmed through the git README, the abstract and all the pictures in the academic paper that it references. I have no
by roenxi 6y ago
Indirectly answering the question - I've skimmed through the git README, the abstract and all the pictures in the academic paper that it references.
I have no idea what this thing does. Can someone explain in simple terms what it does?
My organisation is currently investigating on installing Spark on the theory that it connects to databases and we need analytics. As far as I can tell it breaks analytics work into parallel workloads.
[0] https://github.com/TimelyDataflow/differential-dataflow/ https://github.com/TimelyDataflow/differential-dataflow/
- ilaksh 6y agoI just found this video: https://youtu.be/yyhMI9r0A9E https://youtu.be/yyhMI9r0A9E and also this: http://muratbuffalo.blogspot.com/2017/11/on-dataflow-systems-naiad-and-tensorflow.html?m=1 http://muratbuffalo.blogspot.com/2017/11/on-dataflow-systems... Timely dataflow was inspired by Naiad.
- DougBTX 6y agoLets say you have three numbers, a, b, c, and you want to add them together to get the total. Then later, "c" changes, and you'd like to re-compute the total. One option would be to re-run the full sum, a + b + c, which would be fine. However, that repeats the "a + b" calculation. Would it be possible to improve the efficiency of the total calculation by re-using the pre-computed "a + b" if only "c" changes? Differential dataflow is one way to do that, but only really applies if you have lots of data with complex calculations. For analytics, maybe the "a + b" calculation would cover your last 5 years of operations, and then when a new day's worth of data comes in, you just compute the changes to the totals, rather than re-computing the analytics for all those years, all without manually having to write distinct "total" and "update" code.
- ethbr0 6y ago(Someone correct me if I'm wrong!) I think about differential dataflow as the solution to "I can't batch data operations, because I don't know when my various inputs will land." If everything exists at 7am, and/or you don't need the freshest computed values, this is not the solution you need. If data A is ready between 2-4am, data B at noon, and data C sometime between 8am-6pm, this allows you to abstract that uncertainty into code, then let the system solve it on a daily basis. This is not a problem everyone has. But it is a problem most people working with inventory or events have! And it's usually a problem people feeding things to ML have.
- simongray 6y agoSounds like basic memoization and topological sort gets you all the way there? If that's really all this is about, I'm sure there a lots of adhoc implementations of it in many codebases. It doesn't necessarily seem like something you'd need to bring in a Rust framework to do. Edit: lots of downvotes, yet no replies? Can explain someone why my comment is apparently so terrible...?
- brsg 6y agoYou're asking this in good faith, so you don't deserve to get downvoted. I think people on hn are a bit sensitive to comments they perceive as "reductionist" that may oversimplify complex problems, even if they're honest questions. But you're right, at it's core this kind of problem will use techniques like memoization, and most projects that need something like this will have adhoc approaches to solve this problem. The advantage of differential data flow is that it's a generalized approach to this problem. The business logic behind these workflows, between tracking dependencies and updates can get pretty damn complicated and difficult to maintain. Having a generalized approach would make building these dataflows much simpler. The paper it's based on is pretty skimmable, so I recommend taking a look at it. https://raw.githubusercontent.com/TimelyDataflow/differential-dataflow/master/differentialdataflow.pdf https://raw.githubusercontent.com/TimelyDataflow/differentia...
- simongray 6y agoThanks for linking the paper. I think I maybe a bit environmentally damaged from mainly using Clojure. Algorithms similar to this are fairly common in the Clojure ecosystem. Memoization is part of the standard library too. The claim (in the article) that no one cares about Differential Dataflow seems to be only true when talking about this specific library. The general concept surely translates to some combination of simple concepts like memoization, topological sorting, partial application, etc. so it's obvious to me that many adhoc implementations would exist tailored to more specific needs in a different programming languages with different feature sets. Sometimes buying into a framework is a lot more work than rolling your own, especially if it means having to switch to a different programming language.
- asavinov 6y agoAssume you need to execute the following query: SELECT SUM(A) FROM MyTable For large tables it will take some time to compute the result. Now assume we append a new record and want to get the new result. The traditional approach is execute this query again. A better approach is to process this new record only by adding its value in A to the result of the previous query. It is important in (stateful) stream processing. Something similar is implemented in these libraries which however rely on a different data processing conception (alternative to map-reduce): https://github.com/asavinov/prosto https://github.com/asavinov/prosto - Functions matter! No join-groupby, No map-reduce. https://github.com/asavinov/lambdo https://github.com/asavinov/lambdo - Feature engineering and machine learning: together at last!
- worldsayshi 6y agoI think there are a lot of similarly interesting paradigms that goes mostly unnoticed because of a lack of "developer UX". My personal favorite is "Functional hybrid modelling" - https://github.com/giorgidze/Hydra https://github.com/giorgidze/Hydra
- rkhaitan 6y ago(disclaimer: I work at Materialize and I work with Differential regularly) Differential dataflow lets you write code such that the resulting programs are incremental e.g. if you were computing the most retweeted tweet in all of twitter or something like that and 5 minutes later 1000 new tweets showed up it would only take work proportional to the 1000 new tweets to update the results. It wouldn't need to redo the computation across all tweets. Unlike every other similar framework I know of, Differential can also do this for programs with loops / recursion which makes it more possible to write algorithms. Beyond that, as you've noted it parallelizes work nicely. I wrote a blog post that was meant to explain "what does Differential do" and "when it is or isn't useful" and give some concrete examples that might be helpful. https://materialize.com/life-in-differential-dataflow/ https://materialize.com/life-in-differential-dataflow/
- dominotw 6y ago> if you were computing the most retweeted tweet in all of twitter are all the retweets counts of every tweet stored in the memory ? Where are previous counts stored that are merged with new stream of tweets.