> that's of the main points of columnar data formats such as Parquet and ORC, I'm not sure that I understand the main point of in-memory columnar data formats.

> that's of the main points of columnar data formats such as Parquet and ORC, I'm not sure that I understand the main point of in-memory columnar data formats. ORC has had its own in-memory columnar data-format for the last couple of years - VectorizedRowBatch. This doesn't use any Unsafe access, but uses pure JVM arrays for layout since it allows the Java JIT to unroll a lot of loops internally. Here's an example of a patch from Intel for unrolling the 64 bit operations into 256 bit operations and allowing the Java code to be auto-vectorized to use ymm<n> registers https://issues.apache.org/jira/browse/HIVE-10180 https://issues.apache.org/jira/browse/HIVE-10180 Secondly, we maintain isRepeating=true independently for each column, which means that operations for 1024 rows can sometimes be as fast as operations for 1 row. While in a row-based model, it would have no way of maintaining duplicate information on a column directly. I have looked at the ValueVectors impl in Drill and Tungsten in Spark - which are more cache efficient than Hive's internal columnar structure. However erasing type information into a byte[] structure prevents the JIT from auto-vectorizing inner loops.