To be fair, Java programmers sweat annotation bullets to cajole the JIT to make Java code perform to within 50% of equivalent C code so getting this kind of per

To be fair, Java programmers sweat annotation bullets to cajole the JIT to make Java code perform to within 50% of equivalent C code so getting this kind of performance from the get-go seems like a pretty good deal to me. See also maxas versus CUDA in Nervana's Neon library on github. In this case, naively, it's used to make SGEMM operations that are 20-30% faster, but across the entire universe of SGEMM inputs, it can be 3-6x faster. Whether this is because of the use of assembler or because Scott Gray is a better dense matrix programmer than the current cuBLAS team is an open question because cuBLAS is closed source. Finally, you've ignored the 800 lb gorilla in the room: Intel is making processors that look more and more like GPUs in terms of cores, vector units, and SIMD width whilst investing in FPGAs through the purchase of Altera. But what they're not making is a consistent paradigm for programming them. They could embrace an open standard like OpenCL. They could give NVIDIA a flaming middle finger and build a CUDA compiler like AMD did (Rhythm). Or they could make one of their proprietary languages like Intel TBB (https://www.threadingbuildingblocks.org/ https://www.threadingbuildingblocks.org/) or ISPC (https://ispc.github.io/ https://ispc.github.io/) a first-class citizen and throw their massive support engine behind it. But instead, they seem to throw these APIs over the fence and to continue to rely on "recompile and run" in the hopes that optimizing compilers can magically parallelize otherwise serial code. Which is why I continue to stick with CUDA/NVIDIA...