6 ms·

Has anyone tried skipping the embedding path in favor of combining a proper FTS engine with an LLM yet? A tool like Lucene seems far more competent at the task

Has anyone tried skipping the embedding path in favor of combining a proper FTS engine with an LLM yet? A tool like Lucene seems far more competent at the task of "find most relevant text fragment" compared to what is realized in a typical vector search application today. I'd also argue that you get more inspectability and control this way. You could even manage the preferred size of the fragments on a per-document basis using an entirely separate heuristic at indexing time. The semantic capabilities of vector search seem nice, but could you not achieve a similar outcome by using the LLM to project synonymous OR clauses into the FTS query based upon static background material or prior search iteration(s)?

lmeyerov 2y ago
Generally you do both and rank the combined results This is also why generally you don't get a vectordb and instead just add a vector index to the DB you are already using Re:semantic vs synonym, that's really domain dependent. As the number of hits go up, and queries get more interesting, the more vectors get interesting. At the same time, vectors are heavy, so there's also the question of pushing the semantic aspect to the ranker vs the index, but I don't see that discussed much (search & storage vs compute & latency)
- bob1029 2y ago
  > There's also the question of pushing the semantic aspect to the ranker vs the index Could it make sense to perform dynamic vector lookup over the FTS result set best fragments? This could save a lot of money if you have a massive corpus to index because you'd only be paying to embed things that are being searched for at runtime. Focusing on just the best fragments could also improve the SnR going into the final vector search phase, especially if the fragment length is managed appropriately for each kind of document. If we are dealing with a method from a codebase, then we might prefer to have an unlimited fragment length. For a 20 megabyte PDF, it could be closer to the size of a tweet.
woodson 2y ago
Query expansion has been done since forever, long before even word2vec and other semantic embedding models, e.g. using WordNet (not a DNN, despite its name), see https://lucene.apache.org/core/3_3_0/api/contrib-wordnet/org/apache/lucene/wordnet/package-summary.html https://lucene.apache.org/core/3_3_0/api/contrib-wordnet/org....
- social_quotient 2y ago
  With query expansion do you find it needed for ux purposes to tell the user what you have done to their original query? or do you just run with it and let the result interactions speak for themselves?
maxlamb 2y ago
What’s FTS?
- shallmn 2y ago
  In this context “Full Text Search”. In the context of rage-quitting, something entirely different.
- DandyDev 2y ago
  Thank you! I really hate it when people throw around acronyms instead of just hitting a few extra keys on their keyboard for clarity
prng2021 2y ago
You can use both search methods by using hybrid search. Here’s an implementation: https://learn.microsoft.com/en-us/azure/search/hybrid-search-overview https://learn.microsoft.com/en-us/azure/search/hybrid-search...
- manishsharan 2y ago
  I wonder how I could implement something similar using Lucene and a vercor db ?
  - bob1029 2y ago
    Build a service that talks to both at the same time, or maybe some project like this: https://github.com/JuniusLuo/VecLucene https://github.com/JuniusLuo/VecLucene
  - esafak 2y ago
    All the vector stores are going hybrid; it's becoming a table stakes feature.
zby 2y ago
Seems like the idea of Agentic Rag (https://zzbbyy.substack.com/p/agentic-rag https://zzbbyy.substack.com/p/agentic-rag). I have created a very bare bones implementation as an example at https://github.com/zby/LLMEasyTools/tree/main/examples/agentic_rag https://github.com/zby/LLMEasyTools/tree/main/examples/agent... (it uses whoosh for indexing) and I am working on a more complete one at https://github.com/zby/answerbot https://github.com/zby/answerbot.

lmeyerov 2y ago

Generally you do both and rank the combined results This is also why generally you don't get a vectordb and instead just add a vector index to the DB you are already using Re:semantic vs synonym, that's really domain dependent. As the number of hits go up, and queries get more interesting, the more vectors get interesting. At the same time, vectors are heavy, so there's also the question of pushing the semantic aspect to the ranker vs the index, but I don't see that discussed much (search & storage vs compute & latency)

bob1029 2y ago
> There's also the question of pushing the semantic aspect to the ranker vs the index Could it make sense to perform dynamic vector lookup over the FTS result set best fragments? This could save a lot of money if you have a massive corpus to index because you'd only be paying to embed things that are being searched for at runtime. Focusing on just the best fragments could also improve the SnR going into the final vector search phase, especially if the fragment length is managed appropriately for each kind of document. If we are dealing with a method from a codebase, then we might prefer to have an unlimited fragment length. For a 20 megabyte PDF, it could be closer to the size of a tweet.

woodson 2y ago

Query expansion has been done since forever, long before even word2vec and other semantic embedding models, e.g. using WordNet (not a DNN, despite its name), see https://lucene.apache.org/core/3_3_0/api/contrib-wordnet/org/apache/lucene/wordnet/package-summary.html https://lucene.apache.org/core/3_3_0/api/contrib-wordnet/org....

social_quotient 2y ago
With query expansion do you find it needed for ux purposes to tell the user what you have done to their original query? or do you just run with it and let the result interactions speak for themselves?

maxlamb 2y ago

What’s FTS?

shallmn 2y ago
In this context “Full Text Search”. In the context of rage-quitting, something entirely different.
DandyDev 2y ago
Thank you! I really hate it when people throw around acronyms instead of just hitting a few extra keys on their keyboard for clarity

prng2021 2y ago

You can use both search methods by using hybrid search. Here’s an implementation: https://learn.microsoft.com/en-us/azure/search/hybrid-search-overview https://learn.microsoft.com/en-us/azure/search/hybrid-search...

manishsharan 2y ago
I wonder how I could implement something similar using Lucene and a vercor db ?
- bob1029 2y ago
  Build a service that talks to both at the same time, or maybe some project like this: https://github.com/JuniusLuo/VecLucene https://github.com/JuniusLuo/VecLucene
- esafak 2y ago
  All the vector stores are going hybrid; it's becoming a table stakes feature.

zby 2y ago

Seems like the idea of Agentic Rag (https://zzbbyy.substack.com/p/agentic-rag https://zzbbyy.substack.com/p/agentic-rag). I have created a very bare bones implementation as an example at https://github.com/zby/LLMEasyTools/tree/main/examples/agentic_rag https://github.com/zby/LLMEasyTools/tree/main/examples/agent... (it uses whoosh for indexing) and I am working on a more complete one at https://github.com/zby/answerbot https://github.com/zby/answerbot.