3 ms·
My $0.02: correct chunking can improve accuracy, but it does not change the fact that it is still a single-shot operation. I have commented on this before, so I
by _pdp_ 3y ago
My $0.02: correct chunking can improve accuracy, but it does not change the fact that it is still a single-shot operation. I have commented on this before, so I am repeating myself, but what RAGs are trying to do is the equivalent of looking up some information (let's say via a search engine), and you happen to have the correct answer in the first 5 results - not the links but the actual excerpt from the crawled pages. You don't need many evals to naturally figure out that this will only sometimes work. So, chunking improves the performance as long as the search phrase can discover the correct information, but it does not consider that the search itself could be wrong or require more evaluation. Add to the mix that vectorisation of the records does not work well for non-tokens, made-up words, foreign languages, etc, and you start getting the idea of the complexity involved. This is why more context is better but up to a limit.
IMHO, in most use cases, chunking optimisation strategies will not substantially improve performance. What I think might improve performance is running N search strategies with multiple variations of the search phrase and picking up the best answer. But this is currently expensive and slow.
Having developed a RAG platform over one and a half years ago, I find many of these challenges strikingly familiar.
- serjester 3y agoThere's far more to a RAG pipeline than chunking documents, chunking is just one way to interface with a file. In our case we use query decomposition, document summaries and chunking to achieve strong results. Your right that chunking is just one piece of this. But without quality chunks you're either going to miss context come query time (bad chunks) or use 100X the tokens (full file context).
- register 3y agoCan you describe a little bit more in detail what is your stragegy on query decomposition?
- gardnr 3y agoHere is a description: https://js.langchain.com/docs/use_cases/query_analysis/techniques/decomposition https://js.langchain.com/docs/use_cases/query_analysis/techn... > When a user asks a question there is no guarantee that the relevant results can be returned with a single query. Sometimes to answer a question we need to split it into distinct sub-questions, retrieve results for each sub-question, and then answer using the cumulative context. > For example if a user asks: “How is Web Voyager different from reflection agents”, and we have one document that explains Web Voyager and one that explains reflection agents but no document that compares the two, then we’d likely get better results by retrieving for both “What is Web Voyager” and “What are reflection agents” and combining the retrieved documents than by retrieving based on the user question directly. > This process of splitting an input into multiple distinct sub-queries is what we refer to as query decomposition. It is also sometimes referred to as sub-query generation.
- mistermann 3y ago> What I think might improve performance is running N search strategies with multiple variations of the search phrase and picking up the best answer. But this is currently expensive and slow. Eerily similar to Thinking Fast and Slow, and may help explain (when combined with biological and social evolutionary theory) why people have such a strong aversion to System 2 thinking.
- _pdp_ 3y agoHa, never thought of that. Thank you :)
- mistermann 3y agoIt'd be funny if humanity was permanently stalled at the stage AI is currently at. Well, funny if one could watch it remotely like on The Truman Show instead of being trapped within it.