3 ms·
I saw a presentation from a company named parallel machines, they claim x800 faster than Spark performance on pagerank, and you know what, working with GraphX
by nl 11y ago
I saw a presentation from a company named parallel machines, they claim x800 faster than Spark performance on pagerank, and you know what, working with GraphX
800 times faster or 800%? 800% (ie, 8 times) is believable, but I'd need a lot of convincing if they are claiming 800 times.
Frank McSherry's work on graph computation is pretty good, and he get's 16 times speed up over Spark[1]. Maybe if they are claiming 800 times less resource usage or something?
[1] http://www.frankmcsherry.org/pagerank/distributed/performance/2015/07/08/pagerank.html http://www.frankmcsherry.org/pagerank/distributed/performanc...
- eranation 11y agoI'll double check, you are probably right. In any case they are coming from HPC background and squeezing hardware to the limit. Also they don't use the JVM they compile a popular data science language to native code (I didn't sign an NDA but not sure if they want more info to be revealed yet) Running native vs JVM, using hardware capabilities and utilizing every piece of performance they can think of (one of the co founders I think was a CPU designer at Intel or something) lets them get to the theoretical limit of linearity. Spark might get there eventually... But it will take time