3 ms·
To put this into context I would recommend reading 'MonetDB/X100: Hyper-Pipelining Query Execution' [0]. Vectorized execution has been sort of an open secret in
by rusanu 10y ago
To put this into context I would recommend reading 'MonetDB/X100: Hyper-Pipelining Query Execution' [0]. Vectorized execution has been sort of an open secret in database industry for quite some time now.
For me, is particularly interesting reading the Spark achievements. I was part of the similar Hive effort (the Stinger initiative [1]) and I contributed some parts of the Hive vectorized execution [2]. I see the same solution that applied to Hive now applies to Spark:
- move to a columnar, highly compressed storage format (Parquet, for Hive it was ORC)
- implement a vectorized execution engine
- code generation instead of plan interpretation. This is particularly interesting for me because for Hive this was discussed then and actually not adopted (ORC and vectorized execution had, justifiably, bigger priority).
Looking at the numbers presented in OP, it looks very nice. Aggregates, Filters, Sort, Scan ('decoding') show big improvement (I would expected these, is exactly what vectorized execution is best at). I like that Hash-Join also shows significant improvement, is obvious their implementation is better than the HIVE-4850 I did, of which I'm not too proud. The SM/SMB join is not affected, no surprise there.
I would like to see a separation of how much of the improvement comes from vectorization vs. how much from code generation. I get the feeling that the way they did it these cannot be separated. I think there is no vectorized plan/operators to compare against the code generation, they implemented both simultaneously. I'm speculating, but I guess the new whole-stage code generation it generates vectorized code, so there is no vectorized execution w/o code generation.
All in all, congrats to the DataBricks team. This will have a big impact.
[0] http://oai.cwi.nl/oai/asset/16497/16497B.pdf http://oai.cwi.nl/oai/asset/16497/16497B.pdf
[1] http://hortonworks.com/blog/100x-faster-hive/ http://hortonworks.com/blog/100x-faster-hive/
[2] https://issues.apache.org/jira/browse/HIVE-4160 https://issues.apache.org/jira/browse/HIVE-4160
- sitkack 10y agoHow much of this work gets the working set off of the JVM heap? They are generating JVM bytecode? Compact heap layout is the next big win.
- rusanu 10y agoOP points to SPARK-12795 [0] and is all open source. They generate Java source code. You can read more at the prototype pull request: https://github.com/apache/spark/pull/10735 https://github.com/apache/spark/pull/10735 (I couldn't find a spec doc). If I understand correctly they insert a `WholeStageCodegen`[1] operator into the plan: /** * WholeStageCodegen compile a subtree of plans that support codegen together into single Java * function. * * Here is the call graph of to generate Java source (plan A support codegen, but plan B does not): * * WholeStageCodegen Plan A FakeInput Plan B * ========================================================================= * * -> execute() * | * doExecute() ---------> inputRDDs() -------> inputRDDs() ------> execute() * | * +-----------------> produce() * | * doProduce() -------> produce() * | * doProduce() * | * doConsume() <--------- consume() * | * doConsume() <-------- consume() * * SparkPlan A should override doProduce() and doConsume(). * * doCodeGen() will create a CodeGenContext, which will hold a list of variables for input, * used to generated code for BoundReference. */ [0] https://issues.apache.org/jira/browse/SPARK-12795 https://issues.apache.org/jira/browse/SPARK-12795 [1] https://github.com/apache/spark/blob/0e70fd61b4bc92bd744fc44dd3cbe91443207c72/sql/core/src/main/scala/org/apache/spark/sql/execution/WholeStageCodegenExec.scala https://github.com/apache/spark/blob/0e70fd61b4bc92bd744fc44...
- Everlag 10y agoFor anyone looking for info about column oriented DBs, I can recommend [0] as a great place to start. It covers the basic optimizations and compares some older solutions at a fairly high level. [0]http://db.csail.mit.edu/pubs/abadi-column-stores.pdf http://db.csail.mit.edu/pubs/abadi-column-stores.pdf
- daviesliu 10y agoRight now, vectorization is ONLY used in Parquet reader and in-memory cache, other operators still work in one-row-at-a-time mode, but compiled into single (or a few) Java function(s), the intermediate results of these operators could be hold in CPU register or in stack. It's easy to be high performant and support complicated data types and expressions comparing to vectorized execution, so we (Databricks team) picked whole-stage code generation other than vectorized execution. disclaimer: I worked on whole-stage codegen and other performance related stuff.