4 ms·
A question I've never found a good answer to for columnar query engines is how to handle indexing. For queries with very high selectivity it seems like any ga
by xKingfisher 7y ago
A question I've never found a good answer to for columnar query engines is how to handle indexing.
For queries with very high selectivity it seems like any gains from increased cache-friendliness or SIMD would be erased by still needing to make random forward strides through the data.
The only solution I can really think of is to make the index point to a block of N elements and then take advantage of vector processing on the matching blocks. Have you run into this issue?
I haven't been able to find any examples of column SQL engine discussions that don't require whole-table scans.
- atombender 7y agoOne interesting solution is in Lucene — which isn't perhaps classically columnar, but something of a hybrid — which uses skip lists to allow random skipping through compressed index ("posting list") data.