3 ms·
I ran into exactly this while profiling a Parquet-based vector index on DataFusion. The vectors all have a fixed dimension, but LIST decoding and rebuilding the
by petrizhang 2mo ago
I ran into exactly this while profiling a Parquet-based vector index on DataFusion. The vectors all have a fixed dimension, but LIST decoding and rebuilding the Arrow arrays still showed up as a significant CPU cost. Have you looked at whether a similar fast path could be added to parquet-rs?
- gunnarmorling 2mo agoI haven't looked at parquet-rs specifically, but I can't think of any reason why it couldn't implement this kind of optimization.
- petrizhang 2mo agoThanks! I’ll take a closer look at parquet-rs and see if I can prototype a fast path for fixed-size, non-null lists.