I'll double check, you are probably right. In any case they are coming from HPC background and squeezing hardware to the limit. Also they don't use the JVM they

I'll double check, you are probably right. In any case they are coming from HPC background and squeezing hardware to the limit. Also they don't use the JVM they compile a popular data science language to native code (I didn't sign an NDA but not sure if they want more info to be revealed yet) Running native vs JVM, using hardware capabilities and utilizing every piece of performance they can think of (one of the co founders I think was a CPU designer at Intel or something) lets them get to the theoretical limit of linearity. Spark might get there eventually... But it will take time