3 ms·
Nifty. I've always been a big fan of code generation. How does Spark guarantee data memory contiguity(or do they at all)? Do they use misc.sun.unsafe or some f
by vvanders 11y ago
Nifty. I've always been a big fan of code generation.
How does Spark guarantee data memory contiguity(or do they at all)? Do they use misc.sun.unsafe or some form of memory management?
- haimez 11y agoYeah, since spark 1.6 (some optimizations in spark 1.5) the data being operated on is managed "off heap" which means it doesn't add to garbage collection times and is stored in a more contiguous and cache friendly layout. It's definitely using Unsafe but can't speak to the exact implementation.