13 ms·

I work on something similar. Spark is still fairly unoptimized, and there is more room for performance improvements.

by jkot 11y ago

eranation 11y ago
I saw a presentation from a company named parallel machines, they claim x800 faster than Spark performance on pagerank, and you know what, working with GraphX, I am not utterly disbelieving them. Spark has a long way to go on terms of performance. Having that said, this article is completely misleading and incorrect technically. Tachyon is documented in Spark docs as a off heap storage solution since Spark SQL was called Shark. Nothing new and nothing "competing" with spark. It's still catalyst that runs the spark SQL queries. Just a better memory store. This article is simply wrong.
- nl 11y ago
  I saw a presentation from a company named parallel machines, they claim x800 faster than Spark performance on pagerank, and you know what, working with GraphX 800 times faster or 800%? 800% (ie, 8 times) is believable, but I'd need a lot of convincing if they are claiming 800 times. Frank McSherry's work on graph computation is pretty good, and he get's 16 times speed up over Spark[1]. Maybe if they are claiming 800 times less resource usage or something? [1] http://www.frankmcsherry.org/pagerank/distributed/performance/2015/07/08/pagerank.html http://www.frankmcsherry.org/pagerank/distributed/performanc...
  - eranation 11y ago
    I'll double check, you are probably right. In any case they are coming from HPC background and squeezing hardware to the limit. Also they don't use the JVM they compile a popular data science language to native code (I didn't sign an NDA but not sure if they want more info to be revealed yet) Running native vs JVM, using hardware capabilities and utilizing every piece of performance they can think of (one of the co founders I think was a CPU designer at Intel or something) lets them get to the theoretical limit of linearity. Spark might get there eventually... But it will take time

I work on something similar. Spark is still fairly unoptimized, and there is more room for performance improvements.

by jkot 11y ago

I work on something similar. Spark is still fairly unoptimized, and there is more room for performance improvements.

I saw a presentation from a company named parallel machines, they claim x800 faster than Spark performance on pagerank, and you know what, working with GraphX, I am not utterly disbelieving them. Spark has a long way to go on terms of performance. Having that said, this article is completely misleading and incorrect technically. Tachyon is documented in Spark docs as a off heap storage solution since Spark SQL was called Shark. Nothing new and nothing "competing" with spark. It's still catalyst that runs the spark SQL queries. Just a better memory store. This article is simply wrong.

nl 11y ago
I saw a presentation from a company named parallel machines, they claim x800 faster than Spark performance on pagerank, and you know what, working with GraphX 800 times faster or 800%? 800% (ie, 8 times) is believable, but I'd need a lot of convincing if they are claiming 800 times. Frank McSherry's work on graph computation is pretty good, and he get's 16 times speed up over Spark[1]. Maybe if they are claiming 800 times less resource usage or something? [1] http://www.frankmcsherry.org/pagerank/distributed/performance/2015/07/08/pagerank.html http://www.frankmcsherry.org/pagerank/distributed/performanc...
- eranation 11y ago
  I'll double check, you are probably right. In any case they are coming from HPC background and squeezing hardware to the limit. Also they don't use the JVM they compile a popular data science language to native code (I didn't sign an NDA but not sure if they want more info to be revealed yet) Running native vs JVM, using hardware capabilities and utilizing every piece of performance they can think of (one of the co founders I think was a CPU designer at Intel or something) lets them get to the theoretical limit of linearity. Spark might get there eventually... But it will take time