3 ms·
I don't think you're missing anything. Elastic (I'm not very familiar with Solr) is built on assumptions from TFIDF etc., and a big one is that you reduce the s
by polm23 9y ago
I don't think you're missing anything. Elastic (I'm not very familiar with Solr) is built on assumptions from TFIDF etc., and a big one is that you reduce the size of your potential result set by looking up words in an index. Obviously that doesn't hold if your documents are represented as word vectors.
You can use something like annoy for searching vector space. It's not made specifically for text search, so you'll have to roll your own normalization and so on, but that's usually the less complicated part anyway.
https://github.com/spotify/annoy https://github.com/spotify/annoy
- nl 9y agoYeah, Annoy (and the other reply which pointed to FAISS) are interesting. I read https://erikbern.com/2018/02/15/new-benchmarks-for-approximate-nearest-neighbors.html https://erikbern.com/2018/02/15/new-benchmarks-for-approxima... (by the author of Annoy) with great interest. He points to HNSW which I've know nothing about, but seems very fast!