5 ms·

I only heard about vector databases along with the recent advents of AI. Assuming they've been around for a while, what were the benefits of using them over "no

I only heard about vector databases along with the recent advents of AI. Assuming they've been around for a while, what were the benefits of using them over "normal" search engines (e.g. ElasticSearch)?

fzliu 4y ago
Traditional search such as ES and Lucene rely primarily on bag-of-words retrieval and keyword matching e.g. BM25, TF-IDF, etc. Vector databases such as Milvus allow for _semantic_ search. Here's a highly simplified example: if my query was "fields related to computer science", a semantic engine could return "statistics" and "electrical engineering" while avoiding things such as "social science" and "political science".
billythemaniam 4y ago
ES has support for vector search now too. Really you want both in use cases where the user expects the the top results to contain the search keywords, but also wants results that are synonyms or conceptually similar. TF/IDF and BM25 help with first part and vectors help with the second. Theoretically only vectors should be needed, but that isn't my experience in practice.
- gk1 4y ago
  Which is why Pinecone supports hybrid search, which has shown to provide better results for out-of-domain use cases than either semantic search or keyword search alone: https://www.pinecone.io/learn/hybrid-search-intro/ https://www.pinecone.io/learn/hybrid-search-intro/
  - ChocoluvH 4y ago
    IMO vector databases should not mess with ElasticSearch. The real focus should be to improve the recall of vector search. Pity that nobody is doing real AI research here. Money wasted in marketing and branding.
- ChocoluvH 4y ago
  Totally agree. The thing is that ElasticSearch does not meet our requirements in vector searching. I am currently running with Milvus + ElasticSearch, works perfect. The latest Milvus version is super fast and scalable (>50M vectors). Haven't tried Zilliz Cloud. Have to find out what the cost is. I am old school. IMO ElasticSearch is only good for keyword search and these so called "vector databases" products are only good for vector search.
  - jeadie 4y ago
    Didn't even realise Milvus was so lacking. https://github.com/marqo-ai/marqo https://github.com/marqo-ai/marqo also has a hybrid approach. It's just a more complete/end-to-end platform than pinecone, so it really just depends on what you're building
    - ChocoluvH 4y ago
      I personally like Milvus very much. My point is I only trust stuff that focuses their own business. Especially for small startups.
  - hcentelles 4y ago
    Could you please elaborate on how you utilize both of them together, and for which specific use case? I'm attempting to gain a better understanding of the hybrid approach.
    - ChocoluvH 4y ago
      Certainly! The thing is to make ElasticSearch scores "comparable" to Milvus scores. Lots of ways to do this, but there's no single good solution. For example you could calculate BM25 score offline, or use TF-IDF score to do some kind of filtering. Again there's no single perfect answer. You'd have to do a lot of experiment according to your own use case and your own data to get the best results. Also a lot of tuning needs to be done during all phases: 1) query pre-processing 2) query tokenizing 3) retrieval 4) ranking and reranking I personally would not trust any universal "hybird-search" solutions. All toy demos. It usually takes 5-10 good engineers to build a decent search engine/system for any real use case. It also requires a lot of turning, tricks, hand-written rules to make things work.
jamesblonde 4y ago
OpenSearch (elasticsearch open-source fork by AWS) has supported similarity search for embeddings for a couple of years now with its k-NN plugin. It supports 2 engines - FAISS and HNSW - and has post-result filtering support and replication support. A good option, imo.
snowstormsun 4y ago
This is a good explanation by Niels Reimers (who created SBERT): https://www.youtube.com/watch?v=ukIYZw3uRX0 https://www.youtube.com/watch?v=ukIYZw3uRX0

fzliu 4y ago

Traditional search such as ES and Lucene rely primarily on bag-of-words retrieval and keyword matching e.g. BM25, TF-IDF, etc. Vector databases such as Milvus allow for _semantic_ search. Here's a highly simplified example: if my query was "fields related to computer science", a semantic engine could return "statistics" and "electrical engineering" while avoiding things such as "social science" and "political science".

billythemaniam 4y ago

ES has support for vector search now too. Really you want both in use cases where the user expects the the top results to contain the search keywords, but also wants results that are synonyms or conceptually similar. TF/IDF and BM25 help with first part and vectors help with the second. Theoretically only vectors should be needed, but that isn't my experience in practice.

gk1 4y ago
Which is why Pinecone supports hybrid search, which has shown to provide better results for out-of-domain use cases than either semantic search or keyword search alone: https://www.pinecone.io/learn/hybrid-search-intro/ https://www.pinecone.io/learn/hybrid-search-intro/
- ChocoluvH 4y ago
  IMO vector databases should not mess with ElasticSearch. The real focus should be to improve the recall of vector search. Pity that nobody is doing real AI research here. Money wasted in marketing and branding.
ChocoluvH 4y ago
Totally agree. The thing is that ElasticSearch does not meet our requirements in vector searching. I am currently running with Milvus + ElasticSearch, works perfect. The latest Milvus version is super fast and scalable (>50M vectors). Haven't tried Zilliz Cloud. Have to find out what the cost is. I am old school. IMO ElasticSearch is only good for keyword search and these so called "vector databases" products are only good for vector search.
- jeadie 4y ago
  Didn't even realise Milvus was so lacking. https://github.com/marqo-ai/marqo https://github.com/marqo-ai/marqo also has a hybrid approach. It's just a more complete/end-to-end platform than pinecone, so it really just depends on what you're building
  - ChocoluvH 4y ago
    I personally like Milvus very much. My point is I only trust stuff that focuses their own business. Especially for small startups.
- hcentelles 4y ago
  Could you please elaborate on how you utilize both of them together, and for which specific use case? I'm attempting to gain a better understanding of the hybrid approach.
  - ChocoluvH 4y ago
    Certainly! The thing is to make ElasticSearch scores "comparable" to Milvus scores. Lots of ways to do this, but there's no single good solution. For example you could calculate BM25 score offline, or use TF-IDF score to do some kind of filtering. Again there's no single perfect answer. You'd have to do a lot of experiment according to your own use case and your own data to get the best results. Also a lot of tuning needs to be done during all phases: 1) query pre-processing 2) query tokenizing 3) retrieval 4) ranking and reranking I personally would not trust any universal "hybird-search" solutions. All toy demos. It usually takes 5-10 good engineers to build a decent search engine/system for any real use case. It also requires a lot of turning, tricks, hand-written rules to make things work.

jamesblonde 4y ago

OpenSearch (elasticsearch open-source fork by AWS) has supported similarity search for embeddings for a couple of years now with its k-NN plugin. It supports 2 engines - FAISS and HNSW - and has post-result filtering support and replication support. A good option, imo.

snowstormsun 4y ago

This is a good explanation by Niels Reimers (who created SBERT): https://www.youtube.com/watch?v=ukIYZw3uRX0 https://www.youtube.com/watch?v=ukIYZw3uRX0