6 ms·

Big Post About Big Context

49 points by che_shr_cat 3y ago

vb-8448 3y ago
can someone explain me, maybe with an example, what this part means: > 6) A very important and at the same time difficult class of solutions is model result validation Thanks.
- che_shr_cat 3y ago
  That means the models still confabulate and make other errors, and for many cases it's a problem, so there should be solutions to control model output quality
- che_shr_cat 3y ago
  the close example is Machine translation quality estimation (MTQE)
- che_shr_cat 3y ago
  I updated the post a bit, added a few clarifications
anotherpaulg 3y ago
One point the article argues is that with 1M+ token context windows, we shouldn't use RAG to include chunks from a larger collection -- just include everything! This is a very important open question to me, as I have been building a fairly nuanced RAG pipeline for AI coding. With GPT-4 Turbo, it is definitely NOT a good idea to just throw lots of extraneous code into the 128k content window. That distracts and confuses GPT, makes it code worse and less able to follow complex system prompts. You get much better results if you curate a smaller portion of the code that is relevant to the task at hand. I am really interested to find out if Gemini has these same tendencies. If so, quality RAG is going to remain valuable for curating the context. If not, then Gemini would have a huge advantage over GPT-4. It would be really valuable to be able to naively harness such a large context window.
- meandmycode 3y ago
  Also, while the cost of tokens is lower, let's argue it's cheap enough not to care. Reading 1m tokens surely isn't realistic for latency?
  - hansonw 3y ago
    If sub-quadratic architectures (eg Mamba) become a thing, it will become feasible to precompute most of the work for a fixed prefix (i.e. system prompt) and the latency can be pretty minimal. Even with current transformers, if you have a fixed system prompt, you can save the KV cache and it helps a lot (though the inference time of each incremental token is still linear).
- refulgentis 3y ago
  Cosign, I feel like the longer context window is mostly a dud, especially above 16K tokens. I don't miss sweating the details on what exactly to fit into 4096, but if I just use GPT-4, I'm left with the distinct impression the current context extension stuff is hacky, and something needs to be done in training to actually make use of it. Google miraculously getting 1M context with perfect recall is more fuel for my "don't tell me, show me" fire when it comes to Google. They're batting roughly 0 for 2 years now
  - dontupvoteme 3y ago
    The main thing I'd like large context for is the ability to shove a ton of API/Python/etc documentation into the window for a given problem so it knows better what to do. In terms of the actual output it can be annoying if the convo goes on too long and it hits the window, but for technical things this hardly matters as you can just restart it - more of an RP issue as i understand.
- hansonw 3y ago
  If you think about it, RAG is a relatively primitive “first pass attention layer” that is binary and semi-heuristic based. I think it’s fairly safe to say that in the long term RAG will be integrated into the model architecture somehow, just a matter of when :)
bloaf 3y ago
I think this kind of haystack recall is going to be extremely valuable to large enterprises with massive volumes of standards/practices/technical documents. The status quo is that you have employees ingest and internalize this material over years of employment, then have those experienced employees manage large projects by telling the project if they're in compliance with the organization's standards. A reliable recall system like this isn't just a drop in replacement for new employees, its a drop in replacement for a lot of what makes the expensive employees important: the backwards-and-forwards knowledge of the standards/practices.
- whatshisface 3y ago
  Although our standard template constructors store all of mankind's knowledge and obviate the need for dangerous thinking, only a tech-priest of great experience may adequately consecrate them.
  - cyrillite 3y ago
    Is this a reference or just some solid on the fly sci-fi commentary?
    - whatshisface 3y ago
      It's a reference to WH40k, a fictional universe where faster-than-light travel survived the collapse of civilization in the form of pragmatic tools called "standard template constructors," made to take the burden of science and engineering staff off the shoulders of small planetary colonies, and the formalized rituals of the post-apocalyptic martian cults that remembered in some ways how to use them.

49 points by che_shr_cat 3y ago

can someone explain me, maybe with an example, what this part means: > 6) A very important and at the same time difficult class of solutions is model result validation Thanks.

che_shr_cat 3y ago
That means the models still confabulate and make other errors, and for many cases it's a problem, so there should be solutions to control model output quality
che_shr_cat 3y ago
the close example is Machine translation quality estimation (MTQE)
che_shr_cat 3y ago
I updated the post a bit, added a few clarifications

anotherpaulg 3y ago

One point the article argues is that with 1M+ token context windows, we shouldn't use RAG to include chunks from a larger collection -- just include everything! This is a very important open question to me, as I have been building a fairly nuanced RAG pipeline for AI coding. With GPT-4 Turbo, it is definitely NOT a good idea to just throw lots of extraneous code into the 128k content window. That distracts and confuses GPT, makes it code worse and less able to follow complex system prompts. You get much better results if you curate a smaller portion of the code that is relevant to the task at hand. I am really interested to find out if Gemini has these same tendencies. If so, quality RAG is going to remain valuable for curating the context. If not, then Gemini would have a huge advantage over GPT-4. It would be really valuable to be able to naively harness such a large context window.

meandmycode 3y ago
Also, while the cost of tokens is lower, let's argue it's cheap enough not to care. Reading 1m tokens surely isn't realistic for latency?
- hansonw 3y ago
  If sub-quadratic architectures (eg Mamba) become a thing, it will become feasible to precompute most of the work for a fixed prefix (i.e. system prompt) and the latency can be pretty minimal. Even with current transformers, if you have a fixed system prompt, you can save the KV cache and it helps a lot (though the inference time of each incremental token is still linear).
refulgentis 3y ago
Cosign, I feel like the longer context window is mostly a dud, especially above 16K tokens. I don't miss sweating the details on what exactly to fit into 4096, but if I just use GPT-4, I'm left with the distinct impression the current context extension stuff is hacky, and something needs to be done in training to actually make use of it. Google miraculously getting 1M context with perfect recall is more fuel for my "don't tell me, show me" fire when it comes to Google. They're batting roughly 0 for 2 years now
- dontupvoteme 3y ago
  The main thing I'd like large context for is the ability to shove a ton of API/Python/etc documentation into the window for a given problem so it knows better what to do. In terms of the actual output it can be annoying if the convo goes on too long and it hits the window, but for technical things this hardly matters as you can just restart it - more of an RP issue as i understand.
hansonw 3y ago
If you think about it, RAG is a relatively primitive “first pass attention layer” that is binary and semi-heuristic based. I think it’s fairly safe to say that in the long term RAG will be integrated into the model architecture somehow, just a matter of when :)

bloaf 3y ago

I think this kind of haystack recall is going to be extremely valuable to large enterprises with massive volumes of standards/practices/technical documents. The status quo is that you have employees ingest and internalize this material over years of employment, then have those experienced employees manage large projects by telling the project if they're in compliance with the organization's standards. A reliable recall system like this isn't just a drop in replacement for new employees, its a drop in replacement for a lot of what makes the expensive employees important: the backwards-and-forwards knowledge of the standards/practices.

whatshisface 3y ago
Although our standard template constructors store all of mankind's knowledge and obviate the need for dangerous thinking, only a tech-priest of great experience may adequately consecrate them.
- cyrillite 3y ago
  Is this a reference or just some solid on the fly sci-fi commentary?
  - whatshisface 3y ago
    It's a reference to WH40k, a fictional universe where faster-than-light travel survived the collapse of civilization in the form of pragmatic tools called "standard template constructors," made to take the burden of science and engineering staff off the shoulders of small planetary colonies, and the formalized rituals of the post-apocalyptic martian cults that remembered in some ways how to use them.