3 ms·
This doesn't deserve the downvotes it (currently) has. This is a completely accurate statement, simple RAG works well when questions are highly correlated with
by kristjansson 3y ago
This doesn't deserve the downvotes it (currently) has. This is a completely accurate statement, simple RAG works well when questions are highly correlated with specifics chunks of documents. It does not allow an LLM to synthesize an entire corpus to an answer (e.g. a book report).
Other modes of RAG that retrieve multiple times per generation (e.g. every so-many tokens) can improve performance, and intuitively feel closer to the 'sparse attention over the entire corpus' that RAG is approximating.
However, fine-tuning on relevant, high quality, knowledge-rich question/answer pairs seems dominant, when such examples are available or can be generated.
- filterfiber 3y agoI have to agree, RAG is effectively just compensating for data not fitting in the context window, and it has (potentially severe) limitations. Summarizing a book I think is a great example, RAG prevents LLMs from correlating all of the data. It will likely never be obsolete, but a larger context would be far preferable in nearly all scenerios.
- svcrunch 3y agoI think your comment is accurate, but regarding your last point: "However, fine-tuning on relevant, high quality, knowledge-rich question/answer pairs seems dominant, when such examples are available or can be generated." How does one solve the problem of access-controlled data, if not through RAG? Do you imagine a separate version of the LLM for every user, reflecting their unique permissions on the data? Also, in scenarios where the data is being updated regularly, RAG provides much lower latency to the new information. Deletes also present a challenge for a pure-LLM approach.
- willy_k 3y agoI don’t know much about the space, but couldn’t LoRA be used to (moderately) efficiently use a relevant and individual fine-tuned model alongside a general LLM?