3 ms·
RAG will not change how many tokens LLM can produce at once. Longer context on the other hand, could put some RAG use cases to sleep, if your instructions are
by karmasimida 3y ago
RAG will not change how many tokens LLM can produce at once.
Longer context on the other hand, could put some RAG use cases to sleep, if your instructions are like, literally a manual long, then there is no need for rag.
- 7734128 3y agoI think RAG could be used that do that. If you have a one time retrieval in the beginning, basically amending the prompt, then I agree with you. But there are projects (classmate doing his masters thesis project as one implementation of this) that retrieves once every few tokens and make the retrieved information available to the generation somehow. That would not take a toll on the context window.