2 ms·
I wonder if similar performance can be achieved with Spark accelerator like https://github.com/apache/datafusion-comet https://github.com/apache/datafusion-come
by whoevercares 2y ago
I wonder if similar performance can be achieved with Spark accelerator like https://github.com/apache/datafusion-comet https://github.com/apache/datafusion-comet. Of course it didn’t exist 4 years ago, but would it cheaper to build?
- spott 2y agoIt sounds like a lot of the value here was in a more appropriate level of abstraction, not in the speed of the compute itself. Ray allowed them to optimize elements that spark didn’t, and that was what improved performance, not that spark itself was slow.