5 ms·

The point about synthetic query generation is good. We found users had very poor queries, so we initially had the LLM generate synthetic queries. But then we fo

The point about synthetic query generation is good. We found users had very poor queries, so we initially had the LLM generate synthetic queries. But then we found that the results could vary widely based on the specific synthetic query it generated, so we had it create three variants (all in one LLM call, so that you can prompt it to generate a wide variety, instead of getting three very similar ones back), do parallel search, and then use reciprocal rank fusion to combine the list into a set of broadly strong performers. For the searches we use hybrid dense + sparse bm25, since dense doesn't work well for technical words. This, combined with a subsequent reranker, basically eliminated any of our issues on search.

avereveard 1y ago
final tip is to also feed the interpretation of the user search to the user on the other side, so he can check if the llm understanding was correct.
deepsquirrelnet 1y ago
> For the searches we use hybrid dense + sparse bm25, since dense doesn't work well for technical words. One thing I’m always curious about is if you could simplify this and get good/better results using SPLADE. The v3 models look really good and seem to provide a good balance of semantic and lexical retrieval.
siva7 1y ago
Boy, that should not be the concern of the end user (developer) but those implementing RAG solutions as a service at Amazon, Microsoft, Openai and so on.
- pamelafox 1y ago
  At Microsoft, that's all baked into Azure AI Search - hybrid search does BM25, vector search, and re-ranking, just with setting booleans to true. It also has a new Agentic retrieval feature that does the query rewriting and parallel search execution. Disclosure: I work at MS and help maintain our most popular open-source RAG template, so I follow the best practices closely: https://github.com/Azure-Samples/azure-search-openai-demo/ https://github.com/Azure-Samples/azure-search-openai-demo/ So few developers realize that you need more than just vector search, so I still spend many of my talks emphasizing the FULL retrieval stack for RAG. It's also possible to do it on top of other DBs like Postgres, but takes more effort.
  - catmanjan 1y ago
    I'd love to work with Azure search but because copilot with external items has been made so cheap it's hard to justify...
    - pamelafox 1y ago
      Do you mean that you're using the Copilot indexer for Sharepoint docs? https://learn.microsoft.com/en-us/microsoftsearch/semantic-index-for-copilot https://learn.microsoft.com/en-us/microsoftsearch/semantic-i... AI Search team's been working with the Sharepoint team to offer more options, so that devs can get best of both worlds. Might have some stuff ready for Ignite (mid November).
      - catmanjan 1y ago
        No we have a Microsoft graph connector which inserts externalitems into graph, copilot is able to surface these, probably via the same semantic search database The capability was there for years, but it was expensive - something like $0.60 per 1000 items indexed, then sometimes after copilot was added it became free for up to 50 million items, and now it's free for unlimited items - you just can't beat that for price... https://techcommunity.microsoft.com/blog/microsoft365copilotblog/microsoft-graph-connectors-update-expand-copilot%E2%80%99s-knowledge-with-50-million-ite/4243648 https://techcommunity.microsoft.com/blog/microsoft365copilot...
  - alansaber 1y ago
    That is concerning given that pure vector search is terrible outside of abstractions
    - pamelafox 1y ago
      I know :( But I think vector DBs and vector search got so hyped that people thought you could switch entirely over to them. Lots of APIs and frameworks also used "vector store" as the shorthand for "retrieval data source", which didn't help. That's why I write blog posts like https://blog.pamelafox.org/2024/06/vector-search-is-not-enough.html https://blog.pamelafox.org/2024/06/vector-search-is-not-enou...
alansaber 1y ago
Yep- that's all best practice. I want to know if we could push performance further- routing the query to different embedding models or scoring strategies, or using multiple re-rankers- still feels like the process is missing something.
- tifa2up 1y ago
  OP. The way you improve it is move away from single shot semantic/keyword search and have an agentic system that can evaluate results and do follow-up queries.

avereveard 1y ago

final tip is to also feed the interpretation of the user search to the user on the other side, so he can check if the llm understanding was correct.

deepsquirrelnet 1y ago

> For the searches we use hybrid dense + sparse bm25, since dense doesn't work well for technical words. One thing I’m always curious about is if you could simplify this and get good/better results using SPLADE. The v3 models look really good and seem to provide a good balance of semantic and lexical retrieval.

siva7 1y ago

Boy, that should not be the concern of the end user (developer) but those implementing RAG solutions as a service at Amazon, Microsoft, Openai and so on.

pamelafox 1y ago
At Microsoft, that's all baked into Azure AI Search - hybrid search does BM25, vector search, and re-ranking, just with setting booleans to true. It also has a new Agentic retrieval feature that does the query rewriting and parallel search execution. Disclosure: I work at MS and help maintain our most popular open-source RAG template, so I follow the best practices closely: https://github.com/Azure-Samples/azure-search-openai-demo/ https://github.com/Azure-Samples/azure-search-openai-demo/ So few developers realize that you need more than just vector search, so I still spend many of my talks emphasizing the FULL retrieval stack for RAG. It's also possible to do it on top of other DBs like Postgres, but takes more effort.
- catmanjan 1y ago
  I'd love to work with Azure search but because copilot with external items has been made so cheap it's hard to justify...
  - pamelafox 1y ago
    Do you mean that you're using the Copilot indexer for Sharepoint docs? https://learn.microsoft.com/en-us/microsoftsearch/semantic-index-for-copilot https://learn.microsoft.com/en-us/microsoftsearch/semantic-i... AI Search team's been working with the Sharepoint team to offer more options, so that devs can get best of both worlds. Might have some stuff ready for Ignite (mid November).
    - catmanjan 1y ago
      No we have a Microsoft graph connector which inserts externalitems into graph, copilot is able to surface these, probably via the same semantic search database The capability was there for years, but it was expensive - something like $0.60 per 1000 items indexed, then sometimes after copilot was added it became free for up to 50 million items, and now it's free for unlimited items - you just can't beat that for price... https://techcommunity.microsoft.com/blog/microsoft365copilotblog/microsoft-graph-connectors-update-expand-copilot%E2%80%99s-knowledge-with-50-million-ite/4243648 https://techcommunity.microsoft.com/blog/microsoft365copilot...
- alansaber 1y ago
  That is concerning given that pure vector search is terrible outside of abstractions
  - pamelafox 1y ago
    I know :( But I think vector DBs and vector search got so hyped that people thought you could switch entirely over to them. Lots of APIs and frameworks also used "vector store" as the shorthand for "retrieval data source", which didn't help. That's why I write blog posts like https://blog.pamelafox.org/2024/06/vector-search-is-not-enough.html https://blog.pamelafox.org/2024/06/vector-search-is-not-enou...

alansaber 1y ago

Yep- that's all best practice. I want to know if we could push performance further- routing the query to different embedding models or scoring strategies, or using multiple re-rankers- still feels like the process is missing something.

tifa2up 1y ago
OP. The way you improve it is move away from single shot semantic/keyword search and have an agentic system that can evaluate results and do follow-up queries.