6 ms·

I always wondered why a RAG index has to be a vector DB. If the model understands text/code and can generate text/code it should be able to talk to OpenSearch

by k__ 2y ago

I always wondered why a RAG index has to be a vector DB. If the model understands text/code and can generate text/code it should be able to talk to OpenSearch no problem.

te_chris 2y ago
Honestly you clocked the secret: it doesn’t. It makes sense for the hype, though. As we got LLM’s we also got wayyyy better embedding models, but they’re not dependencies.
simonw 2y ago
It doesn't have to be a vector DB - and in fact I'm seeing increasing skepticism that embedding vector DBs are the best way to implement RAG. A full-text search index using BM25 or similar may actually work a lot better for many RAG applications. I wrote up some notes on building FTS-based RAG here: https://simonwillison.net/2024/Jun/21/search-based-rag/ https://simonwillison.net/2024/Jun/21/search-based-rag/
- rcarmo 2y ago
  I've been using SQLite FTS (which is essentially BM25) and it works so well I haven't really bothered with vector databases, or Postgres, or anything else yet. Maybe when my corpus exceeds 2GB...
- ianbutler 2y ago
  In 2019 I was using vector search to narrow the search space within 100s of millions of documents and then do full text search on the top 10k or so docs. That seems like a better stacking of the technologies even now
  - ramoz 2y ago
    Interesting. Why did you need to “narrow” the search space using vector space? Did you build custom embeddings and feel confident about retrieval segments? I did similar in 2019 but typically in reverse, FTS, and a dual tower model to rerank. Vector search was an additional capability but never augmented the FTS.
    - ianbutler 2y ago
      It was in consideration of how slow our FTS at the time was over large amount of documents and the window we wanted to keep response times in and you're correct, we had custom embeddings and we had a reasonably high confidence. So vector search would reduce the space to like 10k documents and then we'd take the document ids and FTS acted as the final authority on the ranking.
- niam 2y ago
  What are the arguments for embedded vector DBs being suboptimal in RAG, out of curiosity?
  - simonw 2y ago
    The biggest one is that it's hard to get "zero matches" from an embeddings database. You get back all results ordered by distance from the user's query, but it will really scrape the bottom of the barrel if there aren't any great matches - which can lead to bugs like this one: https://simonwillison.net/2024/Jun/6/accidental-prompt-injection/#embeddings-risk https://simonwillison.net/2024/Jun/6/accidental-prompt-injec... The other problem is that embeddings search can miss things that a direct keyword match would have caught. If you have key terms that are specific to your corpus - product names for example - there's a risk that a vector match might not score those as highly as BM25 would have so you may miss the most relevant documents. Finally, embeddings are much more black box and hard to debug and reason about. We have decades of experience tweaking and debugging and improving BM25-style FTS search - the whole field of "Information Retrieval". Throwing that all away in favour of weird new embedding vectors is suboptimal.
    - kgeist 2y ago
      >but because embeddings search orders by similarity score it will ALWAYS return results, really scraping the bottom of the barrel if it has to Why not have a similarity threshold? Say, if the distance is below 0.7, do not accept the search result.
      - simonw 2y ago
        It turns out picking that threshold is extremely difficult - I've tried! The value seems to differ for different searches, so picking eg 0.7 as a fixed value isn't actually as useful as you would expect.
        zmccormick7 2y ago
        Agreed that thresholds don't work when applied to the cosine similarity of embeddings. But I have found that the similarity score returned by high-quality rerankers, especially Cohere, are consistent and meaningful enough that using a threshold works well there.
verdverm 2y ago
You can view RAG as a bigger word2vec. The canonical example being "king - man + woman = queen". Words, or now chunks, have geometric distribution, cluster, and relationships... on semantic levels What is happening is that text is being embedded into a different space, and that format is an array of floats (a point in the embedding space). When we do retrieval, we embed the query and then find other points close to that query. The reason for Vector DB is (1) to optimize for this use-case, we have many specialized data stores / indexes (redis, elastic, dolt, RDBMS) (2) often to be memory based for faster retrieval. PgVector will be interesting to watch. I personally use Qdrant Full-text search will never be able to do some of the things that are possible in the embedding space. The most capable systems will use both techniques
- k__ 2y ago
  "When we do retrieval, we embed the query and then find other points close to that query." To me that just sounds like OpenSearch with extra steps. How is this different/better than a search engine?
benreesman 2y ago
Inner product similarity in an embedding space is often a very valuable feature in a ranker, and the effort/wow ratio at the prototype phase is good, but the idea that it’s the only pillar of an IR stack is SaaS marketing copy. Vector DBs are cool, you want one handy (particularly for recommender tasks). I recommend FAISS as a solid baseline all these years later. If you’re on modern x86_64 then SVS is pretty shit hot. A search engine that only uses a vector DB is a PoC. For folks who want to go deeper on the topic, Lars basically invented the modern “news feed”, which looks a lot like a production RAG system would [1]. 1. https://youtu.be/BuE3DIJGWOw https://youtu.be/BuE3DIJGWOw

by k__ 2y ago

I always wondered why a RAG index has to be a vector DB. If the model understands text/code and can generate text/code it should be able to talk to OpenSearch no problem.

te_chris 2y ago

Honestly you clocked the secret: it doesn’t. It makes sense for the hype, though. As we got LLM’s we also got wayyyy better embedding models, but they’re not dependencies.

simonw 2y ago

It doesn't have to be a vector DB - and in fact I'm seeing increasing skepticism that embedding vector DBs are the best way to implement RAG. A full-text search index using BM25 or similar may actually work a lot better for many RAG applications. I wrote up some notes on building FTS-based RAG here: https://simonwillison.net/2024/Jun/21/search-based-rag/ https://simonwillison.net/2024/Jun/21/search-based-rag/

rcarmo 2y ago
I've been using SQLite FTS (which is essentially BM25) and it works so well I haven't really bothered with vector databases, or Postgres, or anything else yet. Maybe when my corpus exceeds 2GB...
ianbutler 2y ago
In 2019 I was using vector search to narrow the search space within 100s of millions of documents and then do full text search on the top 10k or so docs. That seems like a better stacking of the technologies even now
- ramoz 2y ago
  Interesting. Why did you need to “narrow” the search space using vector space? Did you build custom embeddings and feel confident about retrieval segments? I did similar in 2019 but typically in reverse, FTS, and a dual tower model to rerank. Vector search was an additional capability but never augmented the FTS.
  - ianbutler 2y ago
    It was in consideration of how slow our FTS at the time was over large amount of documents and the window we wanted to keep response times in and you're correct, we had custom embeddings and we had a reasonably high confidence. So vector search would reduce the space to like 10k documents and then we'd take the document ids and FTS acted as the final authority on the ranking.
niam 2y ago
What are the arguments for embedded vector DBs being suboptimal in RAG, out of curiosity?
- simonw 2y ago
  The biggest one is that it's hard to get "zero matches" from an embeddings database. You get back all results ordered by distance from the user's query, but it will really scrape the bottom of the barrel if there aren't any great matches - which can lead to bugs like this one: https://simonwillison.net/2024/Jun/6/accidental-prompt-injection/#embeddings-risk https://simonwillison.net/2024/Jun/6/accidental-prompt-injec... The other problem is that embeddings search can miss things that a direct keyword match would have caught. If you have key terms that are specific to your corpus - product names for example - there's a risk that a vector match might not score those as highly as BM25 would have so you may miss the most relevant documents. Finally, embeddings are much more black box and hard to debug and reason about. We have decades of experience tweaking and debugging and improving BM25-style FTS search - the whole field of "Information Retrieval". Throwing that all away in favour of weird new embedding vectors is suboptimal.
  - kgeist 2y ago
    >but because embeddings search orders by similarity score it will ALWAYS return results, really scraping the bottom of the barrel if it has to Why not have a similarity threshold? Say, if the distance is below 0.7, do not accept the search result.
    - simonw 2y ago
      It turns out picking that threshold is extremely difficult - I've tried! The value seems to differ for different searches, so picking eg 0.7 as a fixed value isn't actually as useful as you would expect.
      - zmccormick7 2y ago
        Agreed that thresholds don't work when applied to the cosine similarity of embeddings. But I have found that the similarity score returned by high-quality rerankers, especially Cohere, are consistent and meaningful enough that using a threshold works well there.

verdverm 2y ago

You can view RAG as a bigger word2vec. The canonical example being "king - man + woman = queen". Words, or now chunks, have geometric distribution, cluster, and relationships... on semantic levels What is happening is that text is being embedded into a different space, and that format is an array of floats (a point in the embedding space). When we do retrieval, we embed the query and then find other points close to that query. The reason for Vector DB is (1) to optimize for this use-case, we have many specialized data stores / indexes (redis, elastic, dolt, RDBMS) (2) often to be memory based for faster retrieval. PgVector will be interesting to watch. I personally use Qdrant Full-text search will never be able to do some of the things that are possible in the embedding space. The most capable systems will use both techniques

k__ 2y ago
"When we do retrieval, we embed the query and then find other points close to that query." To me that just sounds like OpenSearch with extra steps. How is this different/better than a search engine?

benreesman 2y ago

Inner product similarity in an embedding space is often a very valuable feature in a ranker, and the effort/wow ratio at the prototype phase is good, but the idea that it’s the only pillar of an IR stack is SaaS marketing copy. Vector DBs are cool, you want one handy (particularly for recommender tasks). I recommend FAISS as a solid baseline all these years later. If you’re on modern x86_64 then SVS is pretty shit hot. A search engine that only uses a vector DB is a PoC. For folks who want to go deeper on the topic, Lars basically invented the modern “news feed”, which looks a lot like a production RAG system would [1]. 1. https://youtu.be/BuE3DIJGWOw https://youtu.be/BuE3DIJGWOw