5 ms·

ES has support for vector search now too. Really you want both in use cases where the user expects the the top results to contain the search keywords, but also

ES has support for vector search now too. Really you want both in use cases where the user expects the the top results to contain the search keywords, but also wants results that are synonyms or conceptually similar. TF/IDF and BM25 help with first part and vectors help with the second. Theoretically only vectors should be needed, but that isn't my experience in practice.

gk1 4y ago
Which is why Pinecone supports hybrid search, which has shown to provide better results for out-of-domain use cases than either semantic search or keyword search alone: https://www.pinecone.io/learn/hybrid-search-intro/ https://www.pinecone.io/learn/hybrid-search-intro/
- ChocoluvH 4y ago
  IMO vector databases should not mess with ElasticSearch. The real focus should be to improve the recall of vector search. Pity that nobody is doing real AI research here. Money wasted in marketing and branding.
ChocoluvH 4y ago
Totally agree. The thing is that ElasticSearch does not meet our requirements in vector searching. I am currently running with Milvus + ElasticSearch, works perfect. The latest Milvus version is super fast and scalable (>50M vectors). Haven't tried Zilliz Cloud. Have to find out what the cost is. I am old school. IMO ElasticSearch is only good for keyword search and these so called "vector databases" products are only good for vector search.
- jeadie 4y ago
  Didn't even realise Milvus was so lacking. https://github.com/marqo-ai/marqo https://github.com/marqo-ai/marqo also has a hybrid approach. It's just a more complete/end-to-end platform than pinecone, so it really just depends on what you're building
  - ChocoluvH 4y ago
    I personally like Milvus very much. My point is I only trust stuff that focuses their own business. Especially for small startups.
- hcentelles 4y ago
  Could you please elaborate on how you utilize both of them together, and for which specific use case? I'm attempting to gain a better understanding of the hybrid approach.
  - ChocoluvH 4y ago
    Certainly! The thing is to make ElasticSearch scores "comparable" to Milvus scores. Lots of ways to do this, but there's no single good solution. For example you could calculate BM25 score offline, or use TF-IDF score to do some kind of filtering. Again there's no single perfect answer. You'd have to do a lot of experiment according to your own use case and your own data to get the best results. Also a lot of tuning needs to be done during all phases: 1) query pre-processing 2) query tokenizing 3) retrieval 4) ranking and reranking I personally would not trust any universal "hybird-search" solutions. All toy demos. It usually takes 5-10 good engineers to build a decent search engine/system for any real use case. It also requires a lot of turning, tricks, hand-written rules to make things work.
- billythemaniam 4y ago
  If ES doesn't work for you, I recommend Vespa. https://github.com/vespa-engine/vespa https://github.com/vespa-engine/vespa Others have made other suggestions, but Vespa has two unique features. First it is battle tested at a large scale, second it supports combining the keyword and vector scores in several ways. The latter is something that other hybrid systems don't do very well in my experience.

gk1 4y ago

Which is why Pinecone supports hybrid search, which has shown to provide better results for out-of-domain use cases than either semantic search or keyword search alone: https://www.pinecone.io/learn/hybrid-search-intro/ https://www.pinecone.io/learn/hybrid-search-intro/

ChocoluvH 4y ago
IMO vector databases should not mess with ElasticSearch. The real focus should be to improve the recall of vector search. Pity that nobody is doing real AI research here. Money wasted in marketing and branding.

ChocoluvH 4y ago

Totally agree. The thing is that ElasticSearch does not meet our requirements in vector searching. I am currently running with Milvus + ElasticSearch, works perfect. The latest Milvus version is super fast and scalable (>50M vectors). Haven't tried Zilliz Cloud. Have to find out what the cost is. I am old school. IMO ElasticSearch is only good for keyword search and these so called "vector databases" products are only good for vector search.

jeadie 4y ago
Didn't even realise Milvus was so lacking. https://github.com/marqo-ai/marqo https://github.com/marqo-ai/marqo also has a hybrid approach. It's just a more complete/end-to-end platform than pinecone, so it really just depends on what you're building
- ChocoluvH 4y ago
  I personally like Milvus very much. My point is I only trust stuff that focuses their own business. Especially for small startups.
hcentelles 4y ago
Could you please elaborate on how you utilize both of them together, and for which specific use case? I'm attempting to gain a better understanding of the hybrid approach.
- ChocoluvH 4y ago
  Certainly! The thing is to make ElasticSearch scores "comparable" to Milvus scores. Lots of ways to do this, but there's no single good solution. For example you could calculate BM25 score offline, or use TF-IDF score to do some kind of filtering. Again there's no single perfect answer. You'd have to do a lot of experiment according to your own use case and your own data to get the best results. Also a lot of tuning needs to be done during all phases: 1) query pre-processing 2) query tokenizing 3) retrieval 4) ranking and reranking I personally would not trust any universal "hybird-search" solutions. All toy demos. It usually takes 5-10 good engineers to build a decent search engine/system for any real use case. It also requires a lot of turning, tricks, hand-written rules to make things work.
billythemaniam 4y ago
If ES doesn't work for you, I recommend Vespa. https://github.com/vespa-engine/vespa https://github.com/vespa-engine/vespa Others have made other suggestions, but Vespa has two unique features. First it is battle tested at a large scale, second it supports combining the keyword and vector scores in several ways. The latter is something that other hybrid systems don't do very well in my experience.