3 ms·
There's already been some interesting column format optimization work at Meta, as their Velox execution engine team worked with Apache Arrow to align their colu
by jauntywundrkind 3y ago
There's already been some interesting column format optimization work at Meta, as their Velox execution engine team worked with Apache Arrow to align their columnar formats. This talk is actually happening at VeloxCon, so there's got to be some awareness! https://engineering.fb.com/2024/02/20/developer-tools/velox-apache-arrow-15-composable-data-management/ https://engineering.fb.com/2024/02/20/developer-tools/velox-... https://news.ycombinator.com/item?id=39454763 https://news.ycombinator.com/item?id=39454763
I wonder how much if any overlap there is here, and whether it was intentional or accidentally similar. Ah, "return efficient Velox vectors" is on the list, but still seems likely to be some overlap in encoding strategies etc.
The four main points seem to be: a) encoding metadata as part of stream rather than fixed metadata, b) nls are just another encoding, c) no stripe footer/only stream locations is in footer, d) FlatBuffers! Shout out to FlatBuffers, wasn't expecting to see them making a comeback!
I do wish there were a lot more diagrams/slides. There's four bullet points, and Yoav Helfman talks to them, but there's not a ton of showing what he's talking about.
- hiyer 3y agoWhat's the future for datafusion (https://arrow.apache.org/datafusion/ https://arrow.apache.org/datafusion/) if Arrow is moving towards Velox?
- theLiminator 3y agoWhat do you mean by Arrow is moving towards Velox? Arrow is a standard for in-memory columnar data? At most arrow might adopt innovations made in Velox into its spec that datafusion will then adopt?
- hiyer 3y agoI meant the Arrow ecosystem. Datafusion is the query processing project therein at present, so I was curious to know what's the future for that. As you said, it's possible Datafition will adopt some stuff from Velox.
- yencabulator 3y agoThe StringView/ListView and REE optimizations are already part of Arrow, and should be usable with Datafusion (Datafusion just uses Arrow datatypes everywhere). https://docs.rs/datafusion/latest/datafusion/common/arrow/array/type.StringViewBuilder.html https://docs.rs/datafusion/latest/datafusion/common/arrow/ar... https://docs.rs/datafusion/latest/datafusion/common/arrow/datatypes/enum.DataType.html https://docs.rs/datafusion/latest/datafusion/common/arrow/da...
- westurner 3y agoHow do Parquet, Lance, and Nimble compare? lancedb/lance: https://github.com/lancedb/lance https://github.com/lancedb/lance : > Modern columnar data format for ML. Convert from Parquet in 2-lines of code for 100x faster random access, a vector index, data versioning, and more. Compatible with pandas, DuckDB, Polars, and pyarrow Can the optimizations in lance be ported to other formats without significant redesign?