2 ms·
Lucene among other things implements an inverted index - basically tracks frequency of a word across different documents. In my opinion, it is already in a colu
by innagadadavida 5y ago
Lucene among other things implements an inverted index - basically tracks frequency of a word across different documents. In my opinion, it is already in a columnar like format and highly optimized for the use case and won't see any benefit from changing on disk formats.
- yazaddaruvala 5y ago> it is already in a columnar like format and highly optimized for the use case I mean it has more complex data structures (FST) than just columnar, but yeah for doc values and such I agree, that is exactly why I'm curious if Arrow is targeting that usecase, and will be competitively "highly optimized". > and won't see any benefit from changing on disk formats. I'm less interested in Lucene actually migrating to Arrow (although if it reduces tech-debt they should look into it), I'm most interested in if Arrow will help future Lucene-like libraries get implemented with competitive performance. Also since Arrow version=X is cross-language compatible, it would be amazing to be able to create "Lucene" indexes (or segments) in Java (perhaps for legacy reasons), then use Rust or Go to query the data.