3 ms·
Largest data flow job ever? I’m sure Google would beg to differ. At Quantcast we process 50PB every day, and that’s nothing compared to real scale like Google.
by paulsutter 6y ago
Largest data flow job ever? I’m sure Google would beg to differ. At Quantcast we process 50PB every day, and that’s nothing compared to real scale like Google.
And merge joins from sorted data? Joins have been done that way since the punched card days on mainframes (and by any scaled data system)
- sbarre 6y agoSurely you read the article before posting, right? From literally the first sentence: > from our largest Dataflow job
- ascar 6y agoSurely you have read the guidlines before posting, right? > Please don't comment on whether someone read an article. "Did you even read the article? It mentions that" can be shortened to "The article mentions that." https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html The hackernews title and the article title say "the". Critizing this clickbait is more than warranted.
- sbarre 6y agoFair point, I could have worded it better. But I think it's a reach to call this "clickbait". Article titles are shortened all the time, and you can't expect them to have all the context in the title. However, one should reasonably expect people participating in a discussion about an article (particularly when posting criticism) to have actually read it. Complaining about something that is provably false in the first sentence of the article is the bigger sin here, is it not?
- davweb 6y agoIt's largest Dataflow[1] job ever, with a capital D, not "data flow". [1]: https://cloud.google.com/dataflow https://cloud.google.com/dataflow
- enigmo 6y agoMisleading headline, the first sentence is "how Spotify optimized and sped up elements from *our largest Dataflow job*". Surely it's not the largest ever run, even on Dataflow.