3 ms·
It's not just about context length, it's about performance. A 100 million token context that takes an hour to start returning an answer to a prompt isn't very
by simonw 3y ago
It's not just about context length, it's about performance.
A 100 million token context that takes an hour to start returning an answer to a prompt isn't very useful for most things.
As long as there is a relationship between the length of the context and the time it takes to produce an output, there will be a reason to be selective about what goes into that context - aka a reason to use RAG techniques.
- verdverm 3y agoThere's also the focus. A long context means the LLM is more easily distracted. Context "gets the right neurons firing" and why RAG does well. It's also why reranking with RAG improves output, in further increases the quality and/or relevance into a smaller context. Which is all to say really, data quality is probably the most distinguishing factor in LLM/RAG systems... to get back to OP's questions
- spencerchubb 3y agoGood point! If datasets get that large that is an argument in favor of RAG