3 ms·
How much of this work gets the working set off of the JVM heap? They are generating JVM bytecode? Compact heap layout is the next big win.
by sitkack 10y ago
How much of this work gets the working set off of the JVM heap? They are generating JVM bytecode? Compact heap layout is the next big win.
- rusanu 10y agoOP points to SPARK-12795 [0] and is all open source. They generate Java source code. You can read more at the prototype pull request: https://github.com/apache/spark/pull/10735 https://github.com/apache/spark/pull/10735 (I couldn't find a spec doc). If I understand correctly they insert a `WholeStageCodegen`[1] operator into the plan: /** * WholeStageCodegen compile a subtree of plans that support codegen together into single Java * function. * * Here is the call graph of to generate Java source (plan A support codegen, but plan B does not): * * WholeStageCodegen Plan A FakeInput Plan B * ========================================================================= * * -> execute() * | * doExecute() ---------> inputRDDs() -------> inputRDDs() ------> execute() * | * +-----------------> produce() * | * doProduce() -------> produce() * | * doProduce() * | * doConsume() <--------- consume() * | * doConsume() <-------- consume() * * SparkPlan A should override doProduce() and doConsume(). * * doCodeGen() will create a CodeGenContext, which will hold a list of variables for input, * used to generated code for BoundReference. */ [0] https://issues.apache.org/jira/browse/SPARK-12795 https://issues.apache.org/jira/browse/SPARK-12795 [1] https://github.com/apache/spark/blob/0e70fd61b4bc92bd744fc44dd3cbe91443207c72/sql/core/src/main/scala/org/apache/spark/sql/execution/WholeStageCodegenExec.scala https://github.com/apache/spark/blob/0e70fd61b4bc92bd744fc44...