3 ms·

Is there any real point to further RAG work given extremely large contexts are clearly on the way with 1M token contexts already proven?

Narciss 3y ago
It's still much cheaper to run RAG in production (at least if you are using closed models). I'd love to use the entire context of GPT4, but if I do that in production it'll cost much more than using some RAG-dependent implementation.
- humansareok1 3y ago
  But this is just current state. Token costs continue to go down and contexts will continue to get larger.
neverokay 3y ago
The LLM is always out of date. RAG is here to stay.
eropple 3y ago
I think so. Mostly: - Hosted solutions charge you for tokens. More tokens, more money. Keeping money in your pocket: generally recognized as rad. - 1M tokens wouldn't hold the entire codebase I have open in my other window.
chasd00 3y ago
correct me if i'm wrong but you still have to get the context populated. It's still the RAG pattern you just put more data in the prompt than before.
- humansareok1 3y ago
  Well I guess the assumption is that you can just fit everything into the context and there'd be nothing to retrieve anymore.
cosmojg 3y ago
Memory-constrained environments.
- humansareok1 3y ago
  If you're running a 1M context LLM I assume you are well past memory constraints since the weights alone are 10s of GBs.
istinetz 3y ago
yes - private data, real-time data, curated data, citations with no hallucinations, RAG on tabular data, RAG on video, RAG on hierarchical mixed data, RAG over a graph
whakim 3y ago
I'm still not sold on recall at such large context window sizes. It's easy for an LLM to find a needle in a haystack, but in most RAG use-cases it's like finding a needle in a stack of needles, and the benchmarks don't really reflect that. There's also the speed and cost implications of dumping millions of tokens into a prompt - it's prohibitively slow and expensive right now.

Narciss 3y ago

It's still much cheaper to run RAG in production (at least if you are using closed models). I'd love to use the entire context of GPT4, but if I do that in production it'll cost much more than using some RAG-dependent implementation.

humansareok1 3y ago
But this is just current state. Token costs continue to go down and contexts will continue to get larger.

neverokay 3y ago

The LLM is always out of date. RAG is here to stay.

eropple 3y ago

I think so. Mostly: - Hosted solutions charge you for tokens. More tokens, more money. Keeping money in your pocket: generally recognized as rad. - 1M tokens wouldn't hold the entire codebase I have open in my other window.

chasd00 3y ago

correct me if i'm wrong but you still have to get the context populated. It's still the RAG pattern you just put more data in the prompt than before.

humansareok1 3y ago
Well I guess the assumption is that you can just fit everything into the context and there'd be nothing to retrieve anymore.

cosmojg 3y ago

Memory-constrained environments.

humansareok1 3y ago
If you're running a 1M context LLM I assume you are well past memory constraints since the weights alone are 10s of GBs.

istinetz 3y ago

yes - private data, real-time data, curated data, citations with no hallucinations, RAG on tabular data, RAG on video, RAG on hierarchical mixed data, RAG over a graph

whakim 3y ago

I'm still not sold on recall at such large context window sizes. It's easy for an LLM to find a needle in a haystack, but in most RAG use-cases it's like finding a needle in a stack of needles, and the benchmarks don't really reflect that. There's also the speed and cost implications of dumping millions of tokens into a prompt - it's prohibitively slow and expensive right now.