4 ms·
What's the future for datafusion (https://arrow.apache.org/datafusion/ https://arrow.apache.org/datafusion/) if Arrow is moving towards Velox?
by hiyer 3y ago
What's the future for datafusion (https://arrow.apache.org/datafusion/ https://arrow.apache.org/datafusion/) if Arrow is moving towards Velox?
- theLiminator 3y agoWhat do you mean by Arrow is moving towards Velox? Arrow is a standard for in-memory columnar data? At most arrow might adopt innovations made in Velox into its spec that datafusion will then adopt?
- hiyer 3y agoI meant the Arrow ecosystem. Datafusion is the query processing project therein at present, so I was curious to know what's the future for that. As you said, it's possible Datafition will adopt some stuff from Velox.
- yencabulator 3y agoThe StringView/ListView and REE optimizations are already part of Arrow, and should be usable with Datafusion (Datafusion just uses Arrow datatypes everywhere). https://docs.rs/datafusion/latest/datafusion/common/arrow/array/type.StringViewBuilder.html https://docs.rs/datafusion/latest/datafusion/common/arrow/ar... https://docs.rs/datafusion/latest/datafusion/common/arrow/datatypes/enum.DataType.html https://docs.rs/datafusion/latest/datafusion/common/arrow/da...