5 ms·
On another note - these kinds of "let's go back and check what we can do with a single node today" approaches are valuable to recalibrate our thinking once in a
by sriku 8y ago
On another note - these kinds of "let's go back and check what we can do with a single node today" approaches are valuable to recalibrate our thinking once in a while. I didn't interpret this post as a case against distributed processing - there are many things going for it - but just to avoid falling into hype.
A couple of great posts by Frank McSherry along the same lines a few years ago, doing page rank for a 128bil edge graph on a single laptop (in the second post below) -
The first post - "scalability! But at what cost"
http://www.frankmcsherry.org/graph/scalability/cost/2015/01/15/COST.html http://www.frankmcsherry.org/graph/scalability/cost/2015/01/...
The second post - "bigger data, same laptop"
http://www.frankmcsherry.org/graph/scalability/cost/2015/02/04/COST2.html http://www.frankmcsherry.org/graph/scalability/cost/2015/02/...
- Radim 8y agoThis is spot on. There are "highly scalable environments" which are flexible and generic, and then there are local focused tools, by their nature optimized in a way generic frameworks can never be. Think ancient tools like grep versus Hadoop or Spark. Or our word2vec in Gensim vs word2vec in Spark or Tensorflow. The occasional recalibration between "OMG-big-O-toward-infinity!-need-scale" versus "our-data-is-actually-finite-what's-a-good-operational-compromise" is a worthwhile exercise.
- tomkwong 8y agoThese posts are more than 3 years old though. The latest spark/graphx should perform much better. I wonder if there's an updated study is available...
- frankmcsherry 8y agoGraphX had an even-poorer showing in 2016 in the BigDatalog paper: http://yellowstone.cs.ucla.edu/~yang/paper/sigmod2016-p958.pdf http://yellowstone.cs.ucla.edu/~yang/paper/sigmod2016-p958.p... I wouldn't hold your breath on it getting faster. Edit: and if you are interested in state-of-the-art stuff, here is a thread to start you out on the papers currently being presented at SIGMOD2018: https://twitter.com/frankmcsherry/status/1002812204075364353 https://twitter.com/frankmcsherry/status/1002812204075364353