3 ms·
Since Lucene already supports ANN vector search, I am curious how this project is different than what's in core.
by 100k 3y ago
Since Lucene already supports ANN vector search, I am curious how this project is different than what's in core.
- juniusluo 3y agoLucene has 2 limits: 1. Only supports 1 vector per doc. The doc content will have many vectors. You will have to store the chunks of one doc as different docs in Lucene. If you want to leverage Lucene's fields to store doc's metadata, such as title, tags, etc, and fetch them in search, need to work around. Also if you still want the traditional keyword search ability over doc content, better to keep as one doc. 2. Max vector dimension is 1024. OpenAI embedding is 1536. Lucene has discussion on increasing it, but there are concerns.