2 ms·
It entails fusing "narrow" operations on data frames into a single method of generated code to avoid virtual method invocations, help the JIT, and improve hardw
by haimez 11y ago
It entails fusing "narrow" operations on data frames into a single method of generated code to avoid virtual method invocations, help the JIT, and improve hardware branch prediction among other things.
Basically you take the general, composable API functions and generate specific equivalent code that avoids the overhead of dealing with abstract interfaces like Iterators.
- vvanders 11y agoNifty. I've always been a big fan of code generation. How does Spark guarantee data memory contiguity(or do they at all)? Do they use misc.sun.unsafe or some form of memory management?
- haimez 11y agoYeah, since spark 1.6 (some optimizations in spark 1.5) the data being operated on is managed "off heap" which means it doesn't add to garbage collection times and is stored in a more contiguous and cache friendly layout. It's definitely using Unsafe but can't speak to the exact implementation.
- azth 11y ago> Basically you take the general, composable API functions and generate specific equivalent code that avoids the overhead of dealing with abstract interfaces like Iterators. That's pretty cool. Similar to what Rust already does when the compiler elides allocations for chained iterators. I wonder if value type support in the JVM would avoid the need to do this type of code gen.