3 ms·
Elasticsearch, and probably Solr too, has some support for vector searches. https://www.elastic.co/blog/text-similarity-search-with-vectors-in-elasticsearch htt
by jillesvangurp 5y ago
Elasticsearch, and probably Solr too, has some support for vector searches. https://www.elastic.co/blog/text-similarity-search-with-vectors-in-elasticsearch https://www.elastic.co/blog/text-similarity-search-with-vect...
I looked into using that last year. In the end we went with a simple more like this query, which proved good enough relative to the model our machine learning people came up with (not so great because real world data is just hard). The nice thing with this is that you can combine vector search clauses with additional clauses and other features such as aggregations, highlighting, etc.
Another way to use machine learning with Elasticsearch would be the more like this plugin which re-ranks results matching a particular query. It's a simple linear regression (usually) based model that you train using features in the form of elasticsearch query clauses. Basically it learns the boost factors for each of the features. E.g. Soundcloud uses this in their search.
Either way, the hard part of stuff like this is testing your system. Going from a "this looks ballpark right" to "our users click on more results by 0.5% after change X", is a long journey. Not a lot of companies are able to do this and I've seen a few teams fall into the trap of having a not so great machine learned model with a flawed tests suite proving not much at all about its alleged properties. A well crafted query can do a lot of the same things. A benefit is that you can fix things easily whereas most machine learning is a bit more like a black box (either works or does not).
- gk1 5y agoVector search is where the next leap in search and recommender system improvements will come from. * Is already coming from. Google, Bing, Spotify, Pinterest, Facebook, etc, already use it in production. All flavors of ES have some kind of vector search available, albeit with some limitations. There are also stand-alone open-source and managed solutions popping up left and right (I work for one, Pinecone.io). From what I’m seeing, the hard parts people encounter are: - Finding or developing good embedding models. I consider KPI testing a part of this process, since it tells you whether or not your models are actually “good.” - Finding and tuning vector search libraries that can meet speed and accuracy requirements. (Easy if you use ES although you’ll run into latency issues with >10M docs. Even easier with a managed vector search solution.) - Scaling the above to handle millions (or billions!) of items and high query throughput. Again, less of an issue if you’re using managed ES on AWS or Elastic.co, or a managed vector search solution like Pinecone, or you have a team of distributed systems engineers sitting idle.