4 ms·

> In 2023, there has been a shift towards acknowledging keyword search also has value and that a combination of vector + keyword search (aka hybrid search) oper

> In 2023, there has been a shift towards acknowledging keyword search also has value and that a combination of vector + keyword search (aka hybrid search) operates in the sweet spot. Do you have evidence that this hybrid approach is better than a vector search? That would be surprising.

dmezzetti 3y ago
In relation to txtai: https://neuml.hashnode.dev/benefits-of-hybrid-search https://neuml.hashnode.dev/benefits-of-hybrid-search If you run a web search for "vector hybrid search" you'll come across plenty of articles for a number of vector databases discussing the topic.
softwaredoug 3y ago
I'd actually want evidence for the opposite. Why vector retrieval for search? What's the evidence for that? BM25 is often a baseline for improving upon. But I see improvements using lots of techniques, not just vector retrieval.
- dmezzetti 3y ago
  It would be good to have a BM25 baseline entry on this leaderboard: https://huggingface.co/spaces/mteb/leaderboard https://huggingface.co/spaces/mteb/leaderboard And I'm sure you're aware of the BEIR paper: https://arxiv.org/abs/2306.07471 https://arxiv.org/abs/2306.07471. Elastic references that in this blogpost: https://www.elastic.co/blog/improving-information-retrieval-elastic-stack-hybrid https://www.elastic.co/blog/improving-information-retrieval-... I agree that BM25 retrieval + vector re-ranking can work. But vector search does bring results to the table that vanilla BM25 can't, even with a large retrieval window. So I do think there is a place for both with the usual "it depends on your data/requirements" caveat.
  - softwaredoug 3y ago
    The opposite is also true. BM25 prefers lexical matches and brings these candidates back that vector search often doesn’t. I am not disagreeing vectors are useful, but I think benchmark based evidence is not the same as deploying a solution that must scale, be constantly updated, serve many use cases like filtering, search syntax, etc customers want. And plus I think there’s a real danger Of herding to vector retrieval (even then one view of it) which cuts off exploration of diverse solutions.
    - dmezzetti 3y ago
      Hybrid retrieval can possibly be the best of both worlds. I invested a lot of time with the last txtai release adding a minimal dependency Python-based BM25 component (https://neuml.hashnode.dev/building-an-efficient-sparse-keyword-index-in-python https://neuml.hashnode.dev/building-an-efficient-sparse-keyw...). And keyword-only indexes are supported if one desires. I'm with you 100% on not herding to any one way for any problem. I still remember the pre-2023 world where you weren't pressured to work LLMs into everything.
deleted 3y ago
[deleted]
lmeyerov 3y ago
Imagine "day 2" search tasks like the user wanting to filter results for a specific time range You could force the data team to go through hoops like multimodal embeddings, multiple indexes, post-filtering... or figure it out Near-term, it makes sense to do one thing well, so many new vector DBs are really mutable vector indexes with lackluster search. But eventually, if you are not a narrowly-scoped kernel (faiss) but the main search platform needing $$$-level marketshare for growing payrolls, you're incentivized to figure search out beyond ANN, and your competitors are advertising that they are doing the same. So you add post-filtering, and then eventually, maybe even push-down/dynamic. Source: We have been happily using on-the-fly GPU knn embeddings for years in pygraphistry, but when we added billion-scale vector indexing to power louie.ai RAG search for real-time news & database querying, we outgrow pure ann on our first user, and I'm thankful we did diligence on our DB selection
- sroussey 3y ago
  And what did that due diligence reveal?
  - lmeyerov 3y ago
    Most prominent for a surprising amount of the vendors we looked at last year (https://gradientflow.com/the-vector-database-index/ https://gradientflow.com/the-vector-database-index/): - We would have paid more for storing billion+ rows in-memory than our gov & enterprise customers were paying for a full product. Since then, ivfpq etc have gotten more popular to do responsive searches for bigger-than-memory datasets. If we needed small in-memory workloads, we could have done faiss+pandas+fastapi and moved on. - Vendors pitched themselves as managed databases yet their users complain about them going down & losing data - Many did not support basic search operators: time filtering, ... - Few/none had real modern security, e.g., row-level multi-tenant ABAC I haven't been tracking on a per-vendor basis as each publicizes making piecemeal progress on these. We found one that did #1-3 without failing us yet, and we're biting the bullet to implement #4. A scalable OSS answer to #4 might be the one that causes us to switch.
gk1 3y ago
Yes, we published benchmarks here: https://www.pinecone.io/blog/hybrid-search/ https://www.pinecone.io/blog/hybrid-search/

dmezzetti 3y ago

In relation to txtai: https://neuml.hashnode.dev/benefits-of-hybrid-search https://neuml.hashnode.dev/benefits-of-hybrid-search If you run a web search for "vector hybrid search" you'll come across plenty of articles for a number of vector databases discussing the topic.

softwaredoug 3y ago

I'd actually want evidence for the opposite. Why vector retrieval for search? What's the evidence for that? BM25 is often a baseline for improving upon. But I see improvements using lots of techniques, not just vector retrieval.

dmezzetti 3y ago
It would be good to have a BM25 baseline entry on this leaderboard: https://huggingface.co/spaces/mteb/leaderboard https://huggingface.co/spaces/mteb/leaderboard And I'm sure you're aware of the BEIR paper: https://arxiv.org/abs/2306.07471 https://arxiv.org/abs/2306.07471. Elastic references that in this blogpost: https://www.elastic.co/blog/improving-information-retrieval-elastic-stack-hybrid https://www.elastic.co/blog/improving-information-retrieval-... I agree that BM25 retrieval + vector re-ranking can work. But vector search does bring results to the table that vanilla BM25 can't, even with a large retrieval window. So I do think there is a place for both with the usual "it depends on your data/requirements" caveat.
- softwaredoug 3y ago
  The opposite is also true. BM25 prefers lexical matches and brings these candidates back that vector search often doesn’t. I am not disagreeing vectors are useful, but I think benchmark based evidence is not the same as deploying a solution that must scale, be constantly updated, serve many use cases like filtering, search syntax, etc customers want. And plus I think there’s a real danger Of herding to vector retrieval (even then one view of it) which cuts off exploration of diverse solutions.
  - dmezzetti 3y ago
    Hybrid retrieval can possibly be the best of both worlds. I invested a lot of time with the last txtai release adding a minimal dependency Python-based BM25 component (https://neuml.hashnode.dev/building-an-efficient-sparse-keyword-index-in-python https://neuml.hashnode.dev/building-an-efficient-sparse-keyw...). And keyword-only indexes are supported if one desires. I'm with you 100% on not herding to any one way for any problem. I still remember the pre-2023 world where you weren't pressured to work LLMs into everything.

deleted 3y ago

[deleted]

lmeyerov 3y ago

Imagine "day 2" search tasks like the user wanting to filter results for a specific time range You could force the data team to go through hoops like multimodal embeddings, multiple indexes, post-filtering... or figure it out Near-term, it makes sense to do one thing well, so many new vector DBs are really mutable vector indexes with lackluster search. But eventually, if you are not a narrowly-scoped kernel (faiss) but the main search platform needing $$$-level marketshare for growing payrolls, you're incentivized to figure search out beyond ANN, and your competitors are advertising that they are doing the same. So you add post-filtering, and then eventually, maybe even push-down/dynamic. Source: We have been happily using on-the-fly GPU knn embeddings for years in pygraphistry, but when we added billion-scale vector indexing to power louie.ai RAG search for real-time news & database querying, we outgrow pure ann on our first user, and I'm thankful we did diligence on our DB selection

sroussey 3y ago
And what did that due diligence reveal?
- lmeyerov 3y ago
  Most prominent for a surprising amount of the vendors we looked at last year (https://gradientflow.com/the-vector-database-index/ https://gradientflow.com/the-vector-database-index/): - We would have paid more for storing billion+ rows in-memory than our gov & enterprise customers were paying for a full product. Since then, ivfpq etc have gotten more popular to do responsive searches for bigger-than-memory datasets. If we needed small in-memory workloads, we could have done faiss+pandas+fastapi and moved on. - Vendors pitched themselves as managed databases yet their users complain about them going down & losing data - Many did not support basic search operators: time filtering, ... - Few/none had real modern security, e.g., row-level multi-tenant ABAC I haven't been tracking on a per-vendor basis as each publicizes making piecemeal progress on these. We found one that did #1-3 without failing us yet, and we're biting the bullet to implement #4. A scalable OSS answer to #4 might be the one that causes us to switch.

gk1 3y ago

Yes, we published benchmarks here: https://www.pinecone.io/blog/hybrid-search/ https://www.pinecone.io/blog/hybrid-search/