3 ms·

I believe RAG is a temporary hack until we figure out virtually infinite context. I think LLM context is going to be like cache levels. The first level is smal

I believe RAG is a temporary hack until we figure out virtually infinite context. I think LLM context is going to be like cache levels. The first level is small but super fast (like working memory). The next level is larger but slower, and so on. RAG is basically a bad version of attention mechanisms. RAG is used to focus your attention on relevant documents. The problem is that RAG systems are not trained to minimize loss, it is just a similarity score. Obligatory note that I could be wrong and it's just my armchair opinion

mikecaulley 3y ago
This doesn’t consider compute cost; the RAG model is much more efficient compared to infinite context length.
- epistasis 3y ago
  Agreed. I think that RAG implemented via tool-calling, with multiple agents talking to each other, is a much much more likely evolution in the future versus a single unified model. I could very well be wrong! But we wouldn't want LLMs to be performing lots of arithmetic calculations via exploiting hidden parts of themselves that do linear regression or whatever, far better to just give them the calculator and get results faster and cheaper. Similarly, we can give them a search engine (RAG) and let them figure it out more efficiently.
choilive 3y ago
I think that "next level" would essentially be a RAG referential information system that gets data via search engines or databases. Maybe we will have the "google search" equivalent completely intended for LLM clients where all data is stored, searched, and returned via vector embeddings, but it could tap on exabytes of information.
- 0x008 3y ago
  That is actually already what is happening, tools like llama_index are trying to provide all the necessary integrations.
simonw 3y ago
It's not just about context length, it's about performance. A 100 million token context that takes an hour to start returning an answer to a prompt isn't very useful for most things. As long as there is a relationship between the length of the context and the time it takes to produce an output, there will be a reason to be selective about what goes into that context - aka a reason to use RAG techniques.
- verdverm 3y ago
  There's also the focus. A long context means the LLM is more easily distracted. Context "gets the right neurons firing" and why RAG does well. It's also why reranking with RAG improves output, in further increases the quality and/or relevance into a smaller context. Which is all to say really, data quality is probably the most distinguishing factor in LLM/RAG systems... to get back to OP's questions
- spencerchubb 3y ago
  Good point! If datasets get that large that is an argument in favor of RAG
darby_eight 3y ago
> I believe RAG is a temporary hack until we figure out virtually infinite context. I'd assume "large enough" context is the actual goal here, not "virtually infinite".
fallingknife 3y ago
That's going to be quite a while then because LLMs don't even perform well at anything close to the current context limit. They will miss important details when the context gets over a couple thousand tokens.
ttul 3y ago
https://arxiv.org/html/2404.07143v1 https://arxiv.org/html/2404.07143v1 “This work introduces an efficient method to scale Transformer-based Large Language Models (LLMs) to infinitely long inputs with bounded memory and computation”
2099miles 3y ago
As a concept infinite context will not be discovered. We can spin up huge models with large contexts that cost more but even 1M context is 1 expensive and 2 not 3trillion(size of some datasets) Rag is not bad attention, rag is the user not knowing the context to give the LLM

mikecaulley 3y ago

This doesn’t consider compute cost; the RAG model is much more efficient compared to infinite context length.

epistasis 3y ago
Agreed. I think that RAG implemented via tool-calling, with multiple agents talking to each other, is a much much more likely evolution in the future versus a single unified model. I could very well be wrong! But we wouldn't want LLMs to be performing lots of arithmetic calculations via exploiting hidden parts of themselves that do linear regression or whatever, far better to just give them the calculator and get results faster and cheaper. Similarly, we can give them a search engine (RAG) and let them figure it out more efficiently.

choilive 3y ago

I think that "next level" would essentially be a RAG referential information system that gets data via search engines or databases. Maybe we will have the "google search" equivalent completely intended for LLM clients where all data is stored, searched, and returned via vector embeddings, but it could tap on exabytes of information.

0x008 3y ago
That is actually already what is happening, tools like llama_index are trying to provide all the necessary integrations.

simonw 3y ago

It's not just about context length, it's about performance. A 100 million token context that takes an hour to start returning an answer to a prompt isn't very useful for most things. As long as there is a relationship between the length of the context and the time it takes to produce an output, there will be a reason to be selective about what goes into that context - aka a reason to use RAG techniques.

verdverm 3y ago
There's also the focus. A long context means the LLM is more easily distracted. Context "gets the right neurons firing" and why RAG does well. It's also why reranking with RAG improves output, in further increases the quality and/or relevance into a smaller context. Which is all to say really, data quality is probably the most distinguishing factor in LLM/RAG systems... to get back to OP's questions
spencerchubb 3y ago
Good point! If datasets get that large that is an argument in favor of RAG

darby_eight 3y ago

> I believe RAG is a temporary hack until we figure out virtually infinite context. I'd assume "large enough" context is the actual goal here, not "virtually infinite".

fallingknife 3y ago

That's going to be quite a while then because LLMs don't even perform well at anything close to the current context limit. They will miss important details when the context gets over a couple thousand tokens.

ttul 3y ago

https://arxiv.org/html/2404.07143v1 https://arxiv.org/html/2404.07143v1 “This work introduces an efficient method to scale Transformer-based Large Language Models (LLMs) to infinitely long inputs with bounded memory and computation”

2099miles 3y ago

As a concept infinite context will not be discovered. We can spin up huge models with large contexts that cost more but even 1M context is 1 expensive and 2 not 3trillion(size of some datasets) Rag is not bad attention, rag is the user not knowing the context to give the LLM