3 ms·

One point the article argues is that with 1M+ token context windows, we shouldn't use RAG to include chunks from a larger collection -- just include everything!

One point the article argues is that with 1M+ token context windows, we shouldn't use RAG to include chunks from a larger collection -- just include everything! This is a very important open question to me, as I have been building a fairly nuanced RAG pipeline for AI coding. With GPT-4 Turbo, it is definitely NOT a good idea to just throw lots of extraneous code into the 128k content window. That distracts and confuses GPT, makes it code worse and less able to follow complex system prompts. You get much better results if you curate a smaller portion of the code that is relevant to the task at hand. I am really interested to find out if Gemini has these same tendencies. If so, quality RAG is going to remain valuable for curating the context. If not, then Gemini would have a huge advantage over GPT-4. It would be really valuable to be able to naively harness such a large context window.

meandmycode 3y ago
Also, while the cost of tokens is lower, let's argue it's cheap enough not to care. Reading 1m tokens surely isn't realistic for latency?
- hansonw 3y ago
  If sub-quadratic architectures (eg Mamba) become a thing, it will become feasible to precompute most of the work for a fixed prefix (i.e. system prompt) and the latency can be pretty minimal. Even with current transformers, if you have a fixed system prompt, you can save the KV cache and it helps a lot (though the inference time of each incremental token is still linear).
refulgentis 3y ago
Cosign, I feel like the longer context window is mostly a dud, especially above 16K tokens. I don't miss sweating the details on what exactly to fit into 4096, but if I just use GPT-4, I'm left with the distinct impression the current context extension stuff is hacky, and something needs to be done in training to actually make use of it. Google miraculously getting 1M context with perfect recall is more fuel for my "don't tell me, show me" fire when it comes to Google. They're batting roughly 0 for 2 years now
- dontupvoteme 3y ago
  The main thing I'd like large context for is the ability to shove a ton of API/Python/etc documentation into the window for a given problem so it knows better what to do. In terms of the actual output it can be annoying if the convo goes on too long and it hits the window, but for technical things this hardly matters as you can just restart it - more of an RP issue as i understand.
hansonw 3y ago
If you think about it, RAG is a relatively primitive “first pass attention layer” that is binary and semi-heuristic based. I think it’s fairly safe to say that in the long term RAG will be integrated into the model architecture somehow, just a matter of when :)
mediaman 3y ago
There is a lot of emerging evidence that ability to reason, and ability to compare within-context dramatically declines with large context. It is unable to find edits: https://twitter.com/mengdi_en/status/1763338245696274469 https://twitter.com/mengdi_en/status/1763338245696274469 It hallucinates changes badly: https://twitter.com/thomasahle/status/1763408041960231010 https://twitter.com/thomasahle/status/1763408041960231010 To be fair, finding diffs is an odd way to use LLMs, but does show the highly variable abilities of these models over bigger contexts. In other models, reasoning performance declines with more tokens (no data yet here for Gemini 1.5 Pro): https://arxiv.org/html/2402.14848v1 https://arxiv.org/html/2402.14848v1 Note in that last chart that significant declines in reasoning are apparent over just a few thousand tokens.
- panarky 3y ago
  I took the text of Macbeth and changed the dialect of one line: Original: "The rest, that are within the note of expectation, alreadie are i'th' Court" New: "It looks like everyone is here already" Then I gave the entire text to Gemini, ~400000 tokens, and asked it to analyze the play's diction and style, and to identify any lines that didn't match the prevailing dialect. It found that one line I'd changed.
  - interstice 3y ago
    Isn’t it quite likely it already had the whole text from its training data? Which might make it a bit of a different case.
    - panarky 3y ago
      I was responding to the assertion that the "ability to compare within-context dramatically declines with large context". My test demonstrates that even with a large context, Gemini 1.5 finds the needle in the haystack. And it's not even searching for specific terms or meanings, the only clues are subtle differences in style and diction.
- dontupvoteme 3y ago

meandmycode 3y ago

Also, while the cost of tokens is lower, let's argue it's cheap enough not to care. Reading 1m tokens surely isn't realistic for latency?

hansonw 3y ago
If sub-quadratic architectures (eg Mamba) become a thing, it will become feasible to precompute most of the work for a fixed prefix (i.e. system prompt) and the latency can be pretty minimal. Even with current transformers, if you have a fixed system prompt, you can save the KV cache and it helps a lot (though the inference time of each incremental token is still linear).

refulgentis 3y ago

Cosign, I feel like the longer context window is mostly a dud, especially above 16K tokens. I don't miss sweating the details on what exactly to fit into 4096, but if I just use GPT-4, I'm left with the distinct impression the current context extension stuff is hacky, and something needs to be done in training to actually make use of it. Google miraculously getting 1M context with perfect recall is more fuel for my "don't tell me, show me" fire when it comes to Google. They're batting roughly 0 for 2 years now

dontupvoteme 3y ago
The main thing I'd like large context for is the ability to shove a ton of API/Python/etc documentation into the window for a given problem so it knows better what to do. In terms of the actual output it can be annoying if the convo goes on too long and it hits the window, but for technical things this hardly matters as you can just restart it - more of an RP issue as i understand.

hansonw 3y ago

If you think about it, RAG is a relatively primitive “first pass attention layer” that is binary and semi-heuristic based. I think it’s fairly safe to say that in the long term RAG will be integrated into the model architecture somehow, just a matter of when :)

mediaman 3y ago

There is a lot of emerging evidence that ability to reason, and ability to compare within-context dramatically declines with large context. It is unable to find edits: https://twitter.com/mengdi_en/status/1763338245696274469 https://twitter.com/mengdi_en/status/1763338245696274469 It hallucinates changes badly: https://twitter.com/thomasahle/status/1763408041960231010 https://twitter.com/thomasahle/status/1763408041960231010 To be fair, finding diffs is an odd way to use LLMs, but does show the highly variable abilities of these models over bigger contexts. In other models, reasoning performance declines with more tokens (no data yet here for Gemini 1.5 Pro): https://arxiv.org/html/2402.14848v1 https://arxiv.org/html/2402.14848v1 Note in that last chart that significant declines in reasoning are apparent over just a few thousand tokens.

panarky 3y ago
I took the text of Macbeth and changed the dialect of one line: Original: "The rest, that are within the note of expectation, alreadie are i'th' Court" New: "It looks like everyone is here already" Then I gave the entire text to Gemini, ~400000 tokens, and asked it to analyze the play's diction and style, and to identify any lines that didn't match the prevailing dialect. It found that one line I'd changed.
- interstice 3y ago
  Isn’t it quite likely it already had the whole text from its training data? Which might make it a bit of a different case.
  - panarky 3y ago
    I was responding to the assertion that the "ability to compare within-context dramatically declines with large context". My test demonstrates that even with a large context, Gemini 1.5 finds the needle in the haystack. And it's not even searching for specific terms or meanings, the only clues are subtle differences in style and diction.
dontupvoteme 3y ago