3 ms·
Spark SQL, with their new Catalyst optimization, is real time compiling their queries into code (assembling Scala quasiquoted fragments into an AST, compiling t
by rektide 11y ago
Spark SQL, with their new Catalyst optimization, is real time compiling their queries into code (assembling Scala quasiquoted fragments into an AST, compiling that). No ridiculously awesome GPU backend here, but some of the same general kinds of tech- compiled queries.
https://databricks.com/blog/2015/04/13/deep-dive-into-spark-sqls-catalyst-optimizer.html https://databricks.com/blog/2015/04/13/deep-dive-into-spark-... http://people.csail.mit.edu/matei/papers/2015/sigmod_spark_sql.pdf http://people.csail.mit.edu/matei/papers/2015/sigmod_spark_s...
- jandrewrogers 11y agoReal-time SQL compilation has been common in analytical database platforms for many years. Open source has really been the exception to this for some reason, so it is nice to see that people are starting to close that gap a bit. Compiling queries should really be viewed as table stakes for a modern data platform.
- threeseed 11y agoI use SparkSQL every single day and it is not anywhere close to this. You could combine SparkSQL with Tachyon but it is still pretty far from what MapD seems to be accomplishing. Would love to see parts of Spark skip the JVM and call out to Rust or LLVM for some aspects.