3 ms·

I also found pure RAG with vector search to not work. I was creating a bot that could find answers to questions about things by looking at Slack discussions. A

I also found pure RAG with vector search to not work. I was creating a bot that could find answers to questions about things by looking at Slack discussions. At first, I downloaded entire channels, loaded them into a vector DB, and did RAG. The results sucked. Vector searches don't understand things very well, and in this world, specific keywords and error messages are very searchable. Instead, I take the user's query, ask an LLM (Claude / Bedrock) to find keywords, then search Slack using the API, get results, and use an LLM to filter for discussions that are relevant, then summarize them all in a response. This is slow, of course, so it's very multi-threaded. A typical response will be within 30 seconds.

siquick 2y ago
When you’re creating your embedding you can store keywords from the content (using an LLM) in the metadata of each chunk which would positively increase the relevancy of results turned from the retrieval. LlamaIndex does this out of the box.
- thefourthchime 2y ago
  That's interesting! I didn't know that
benreesman 2y ago
Zero shot key phrase extraction is a reasonably well-studied field. I don’t know what the current SOTA is, but the one that was pretty hot shit last time I needed one was kbir-inspec which is on HuggingFace and you can test it right on the page. Might be worth a shot if performance is a tricky spot in your setup.
qeternity 2y ago
Are you doing this for a product or for internal usage?
s-macke 2y ago
I find these discussions funny. For decades we had search engines based on the query terms (keywords). Then there were lots of discussions and some implementations to put a semantic search on top of it to improve the keyword search. A hybrid search. Google Search did exactly that already in 2015 [1]. Now we start from pure semantic search and put keyword search on top of it to improve the semantic search and call it hybrid search. In both approaches, the overall search performance is exactly identical - to the last digit. I am glad, that so far, no one has called this an innovation. But you could certainly write a lot of blog articles about it. [1] https://searchengineland.com/semantic-search-entity-based-search-388221 https://searchengineland.com/semantic-search-entity-based-se...
- spencerchubb 2y ago
  Except now the semantic capabilities are so much stronger. The transformer allows the model to get meaning from words that are far apart from each other
  - s-macke 2y ago
    You are talking about English, right? And only for searches without any special technical terms or abbreviations? Also my use case includes more than 20 languages. To find usable embeddings for all languages is next to impossible. However, there are keyword plugins for most languages in Solr or ElasticSearch. Btw. In my benchmarks the result look something like this in English (MAP=mean average precision): BM25(keyword search) -> MAP=45% Embedding (Ada-002) -> MAP=49% Hybrid (BM25 + Embedding) -> MAP=57% Hybrid (Embedding + BM25) -> MAP=57% And that's before you use synonym dictionaries for keyword searches.
    - spencerchubb 2y ago
      I'm curious, in your benchmark, what's the difference between BM25+Embedding and Embedding+BM25? And what do you use to make the embedding If you make the embedding with an LLM, it should work for any language the LLM is trained on.

siquick 2y ago

When you’re creating your embedding you can store keywords from the content (using an LLM) in the metadata of each chunk which would positively increase the relevancy of results turned from the retrieval. LlamaIndex does this out of the box.

thefourthchime 2y ago
That's interesting! I didn't know that

benreesman 2y ago

Zero shot key phrase extraction is a reasonably well-studied field. I don’t know what the current SOTA is, but the one that was pretty hot shit last time I needed one was kbir-inspec which is on HuggingFace and you can test it right on the page. Might be worth a shot if performance is a tricky spot in your setup.

qeternity 2y ago

Are you doing this for a product or for internal usage?

s-macke 2y ago

I find these discussions funny. For decades we had search engines based on the query terms (keywords). Then there were lots of discussions and some implementations to put a semantic search on top of it to improve the keyword search. A hybrid search. Google Search did exactly that already in 2015 [1]. Now we start from pure semantic search and put keyword search on top of it to improve the semantic search and call it hybrid search. In both approaches, the overall search performance is exactly identical - to the last digit. I am glad, that so far, no one has called this an innovation. But you could certainly write a lot of blog articles about it. [1] https://searchengineland.com/semantic-search-entity-based-search-388221 https://searchengineland.com/semantic-search-entity-based-se...

spencerchubb 2y ago
Except now the semantic capabilities are so much stronger. The transformer allows the model to get meaning from words that are far apart from each other
- s-macke 2y ago
  You are talking about English, right? And only for searches without any special technical terms or abbreviations? Also my use case includes more than 20 languages. To find usable embeddings for all languages is next to impossible. However, there are keyword plugins for most languages in Solr or ElasticSearch. Btw. In my benchmarks the result look something like this in English (MAP=mean average precision): BM25(keyword search) -> MAP=45% Embedding (Ada-002) -> MAP=49% Hybrid (BM25 + Embedding) -> MAP=57% Hybrid (Embedding + BM25) -> MAP=57% And that's before you use synonym dictionaries for keyword searches.
  - spencerchubb 2y ago
    I'm curious, in your benchmark, what's the difference between BM25+Embedding and Embedding+BM25? And what do you use to make the embedding If you make the embedding with an LLM, it should work for any language the LLM is trained on.