3 ms·

Nice to see a new columnar data format alternative. Just a quick question though. The existing columnar data formats such as Parquet and ORC aim to be space-ef

by buremba 11y ago

Nice to see a new columnar data format alternative. Just a quick question though. The existing columnar data formats such as Parquet and ORC aim to be space-efficient since the data is stored in disk and IO operations are usually the bottleneck. The columnar data formats shine in big-data area so the amount of data will be huge. Given that columnar data formats can be compressed efficiently and that's of the main points of columnar data formats such as Parquet and ORC, I'm not sure that I understand the main point of in-memory columnar data formats. Once the data is in-memory and we can access any column of a row in constant-time what's the difference between a row-oriented data format and columnar data format?

khc 11y ago
Random memory access isn't really constant time when you factor in hardware prefetch and cache lines. See "What Every Programmer Should Know About Memory"
- jaltekruse 11y ago
  Exactly this, the most basic example is an operation on a single column. If the data for a single column is mixed in with other data in a row-oriented layout, you are going to have to bring a new chunk of memory into the CPU cache sooner to process a given amount of data. If all of the data is packed together tightly you can read many more values out of the cache before you need to go get more from main memory.
- TallGuyShort 11y ago
  That, and regardless of row- or column-orientation, any common in-memory format that was actually as well-established as this looks like it will be in Big Data projects would be nice.
mamcx 11y ago
I have a similar question. I'm building a relational engine with columnar storage, but is only in-memory. All the info I have read is about improve IO, but wonder about a in-memory storage that is performant (currently: I just have a array per-column). My understanding is that insert will suffer badly for things like compressed columns. If my relational engine is used alike-Datasets where is common to iterate by rows AND do joins, unions, projections and filters, so how balance the thing?
gopalv 11y ago
> that's of the main points of columnar data formats such as Parquet and ORC, I'm not sure that I understand the main point of in-memory columnar data formats. ORC has had its own in-memory columnar data-format for the last couple of years - VectorizedRowBatch. This doesn't use any Unsafe access, but uses pure JVM arrays for layout since it allows the Java JIT to unroll a lot of loops internally. Here's an example of a patch from Intel for unrolling the 64 bit operations into 256 bit operations and allowing the Java code to be auto-vectorized to use ymm<n> registers https://issues.apache.org/jira/browse/HIVE-10180 https://issues.apache.org/jira/browse/HIVE-10180 Secondly, we maintain isRepeating=true independently for each column, which means that operations for 1024 rows can sometimes be as fast as operations for 1 row. While in a row-based model, it would have no way of maintaining duplicate information on a column directly. I have looked at the ValueVectors impl in Drill and Tungsten in Spark - which are more cache efficient than Hive's internal columnar structure. However erasing type information into a byte[] structure prevents the JIT from auto-vectorizing inner loops.

by buremba 11y ago

khc 11y ago

Random memory access isn't really constant time when you factor in hardware prefetch and cache lines. See "What Every Programmer Should Know About Memory"

jaltekruse 11y ago
Exactly this, the most basic example is an operation on a single column. If the data for a single column is mixed in with other data in a row-oriented layout, you are going to have to bring a new chunk of memory into the CPU cache sooner to process a given amount of data. If all of the data is packed together tightly you can read many more values out of the cache before you need to go get more from main memory.
TallGuyShort 11y ago
That, and regardless of row- or column-orientation, any common in-memory format that was actually as well-established as this looks like it will be in Big Data projects would be nice.

mamcx 11y ago

I have a similar question. I'm building a relational engine with columnar storage, but is only in-memory. All the info I have read is about improve IO, but wonder about a in-memory storage that is performant (currently: I just have a array per-column). My understanding is that insert will suffer badly for things like compressed columns. If my relational engine is used alike-Datasets where is common to iterate by rows AND do joins, unions, projections and filters, so how balance the thing?

gopalv 11y ago

> that's of the main points of columnar data formats such as Parquet and ORC, I'm not sure that I understand the main point of in-memory columnar data formats. ORC has had its own in-memory columnar data-format for the last couple of years - VectorizedRowBatch. This doesn't use any Unsafe access, but uses pure JVM arrays for layout since it allows the Java JIT to unroll a lot of loops internally. Here's an example of a patch from Intel for unrolling the 64 bit operations into 256 bit operations and allowing the Java code to be auto-vectorized to use ymm<n> registers https://issues.apache.org/jira/browse/HIVE-10180 https://issues.apache.org/jira/browse/HIVE-10180 Secondly, we maintain isRepeating=true independently for each column, which means that operations for 1024 rows can sometimes be as fast as operations for 1 row. While in a row-based model, it would have no way of maintaining duplicate information on a column directly. I have looked at the ValueVectors impl in Drill and Tungsten in Spark - which are more cache efficient than Hive's internal columnar structure. However erasing type information into a byte[] structure prevents the JIT from auto-vectorizing inner loops.