6 ms·

The Dataflow Model: Balancing Correctness, Latency, and Cost in Data Processing [pdf]

56 points by timclark 11y ago

chrisseaton 11y ago
Why is there no related work section in this paper? I'm not sure simply calling it 'The' dataflow model is very friendly either to all the other previous dataflow models for parallelism that have been developed over the last four decades or so. Why can this implementation be the definitive one so much that it doesn't even need a qualified name and why aren't any of the others even worth a mention?
- demian 11y ago
  IMHO the same thing happened with the "object" concept in the '90s.
- scott_s 11y ago
  The introduction is essentially the related works section.
dang 11y ago
Url changed from http://blog.acolyer.org/2015/08/18/the-dataflow-model-a-practical-approach-to-balancing-correctness-latency-and-cost-in-massive-scale-unbounded-out-of-order-data-processing/ http://blog.acolyer.org/2015/08/18/the-dataflow-model-a-prac..., which points to this.
- zweiterlinde 11y ago
  This is unfortunate---Colyer's summaries are well worth reading.
  - dang 11y ago
    They're good summaries, but after thinking about this a fair bit, I don't think they're enough to trump HN's preference for original sources. Anyone who wants to can read both, since when we change a URL we include the previous one in the comments, as above.
obulpathi 11y ago
Wow ... this is awesome! Quoting from the paper, "live and breathe under the assumption that we will never know if or when we have seen all of our data, only that new data will arrive, old data may be retracted, and the only way to make this problem tractable". That's another amazing mindshift!
- eternalban 11y ago
  Fundamentally still operating in an 'anticipatory' [1] model of computing. A trending shade of pink lipstick for the old pig. [1]: the margin of this post is too small to contain an elaboration on this ;)
wslh 11y ago
Eric Schmidt is in the list of authors, I wonder if he is doing CS research again as an executive.
- trequartista 11y ago
  And his work email is cloude@google.com? That is quirky
- chubot 11y ago
  There are multiple Eric Schmidt's at Google. That one is not the Chairman/former CEO :)
  - nfoz 11y ago
    Wow. Just imagine all the opportunity for social engineering, with a name like that :)

56 points by timclark 11y ago

Why is there no related work section in this paper? I'm not sure simply calling it 'The' dataflow model is very friendly either to all the other previous dataflow models for parallelism that have been developed over the last four decades or so. Why can this implementation be the definitive one so much that it doesn't even need a qualified name and why aren't any of the others even worth a mention?

demian 11y ago
IMHO the same thing happened with the "object" concept in the '90s.
scott_s 11y ago
The introduction is essentially the related works section.

dang 11y ago

Url changed from http://blog.acolyer.org/2015/08/18/the-dataflow-model-a-practical-approach-to-balancing-correctness-latency-and-cost-in-massive-scale-unbounded-out-of-order-data-processing/ http://blog.acolyer.org/2015/08/18/the-dataflow-model-a-prac..., which points to this.

zweiterlinde 11y ago
This is unfortunate---Colyer's summaries are well worth reading.
- dang 11y ago
  They're good summaries, but after thinking about this a fair bit, I don't think they're enough to trump HN's preference for original sources. Anyone who wants to can read both, since when we change a URL we include the previous one in the comments, as above.

obulpathi 11y ago

Wow ... this is awesome! Quoting from the paper, "live and breathe under the assumption that we will never know if or when we have seen all of our data, only that new data will arrive, old data may be retracted, and the only way to make this problem tractable". That's another amazing mindshift!

eternalban 11y ago
Fundamentally still operating in an 'anticipatory' [1] model of computing. A trending shade of pink lipstick for the old pig. [1]: the margin of this post is too small to contain an elaboration on this ;)

wslh 11y ago

Eric Schmidt is in the list of authors, I wonder if he is doing CS research again as an executive.

trequartista 11y ago
And his work email is cloude@google.com? That is quirky
chubot 11y ago
There are multiple Eric Schmidt's at Google. That one is not the Chairman/former CEO :)
- nfoz 11y ago
  Wow. Just imagine all the opportunity for social engineering, with a name like that :)