4 ms·
Is anyone familiar enough to know if Arrow is also targeting usage by libraries like Lucene?
by yazaddaruvala 5y ago
Is anyone familiar enough to know if Arrow is also targeting usage by libraries like Lucene?
- innagadadavida 5y agoLucene among other things implements an inverted index - basically tracks frequency of a word across different documents. In my opinion, it is already in a columnar like format and highly optimized for the use case and won't see any benefit from changing on disk formats.
- yazaddaruvala 5y ago> it is already in a columnar like format and highly optimized for the use case I mean it has more complex data structures (FST) than just columnar, but yeah for doc values and such I agree, that is exactly why I'm curious if Arrow is targeting that usecase, and will be competitively "highly optimized". > and won't see any benefit from changing on disk formats. I'm less interested in Lucene actually migrating to Arrow (although if it reduces tech-debt they should look into it), I'm most interested in if Arrow will help future Lucene-like libraries get implemented with competitive performance. Also since Arrow version=X is cross-language compatible, it would be amazing to be able to create "Lucene" indexes (or segments) in Java (perhaps for legacy reasons), then use Rust or Go to query the data.