4 ms·

This isn't true. Transformer architectures generally take quadratic time wrt sequence length, not exponential. Architectural innovations like flash attention a

• dartos 2y ago
  During inference time, yes, but training time does scale exponentially as backpropagation still has to happen. You can’t use fancy flash attention tricks either.
  • thunderbird120 2y ago
    No, additional context does not cause exponential slowdowns and you absolutely can use FlashAttention tricks during training, I'm doing it right now. Transformers are not RNNs, they are not unrolled across timesteps, the backpropagation path for a 1,000,000 context LLM is not any longer than a 100 context LLM of the same size. The only thing which is larger is the self attention calculation which is quadratic wrt compute and linear wrt memory if you use FlashAttention or similar fused self attention calculations. These calculations can be further parallelized using tricks like ring attention to distribute very large attention calculations over many nodes. This is how google trained their 10M context version of Gemini.
    • dartos 2y ago
      I may be missing something, but I thought that each context token would result in an 3 additional parameters per context token for self attention to build its map, since each attention must calculate a value considering all existing context
    • upghost 2y ago
      So why are the context windows so "small", then? It would seem that if the cost was not so great, then having a larger context window would give an advantage over the competition.
      • thunderbird120 2y ago
        The cost for both training and inference is vaguely quadratic while, for the vast majority of users, the marginal utility of additional context is sharply diminishing. For 99% of ChatGPT users something like 8192 tokens, or about 20 pages of context would be plenty. Companies have to balance the cost of training and serving models. Google did train an uber long context version of Gemini but since Gemini itself fundamentally was not better than GPT-4 or Claude this didn't really matter much, since so few people actually benefited from such a niche advantage it didn't really shift the playing field in their favor.
        • deleted 2y ago
          [deleted]
        • Der_Einzige 2y ago
          Marginal utility only drops because effective context is really bad, i.e. most models still vastly prefer the first things they see and those "needle in a haystack" tests are misleading in that they convince people that LLMs do a good job of handling their whole context when they just don't. If we have the effective context window equal to the claimed context window, well, I'd start worrying a bit about most of the risks that AI doomers talk about...
      • monkmartinez 2y ago
        Memory. I don't know the equation, but its very easy to see when you load a 128k context model at 8K vs 80K. The quant I am running would double VRAM requirements when loading 80K.
        • thomasfromcdnjs 2y ago
          This was my understanding too. Would love more people to chime in on the limits and costs of larger contexts.
    • menaerus 2y ago
      > The only thing which is larger is the self attention calculation which is quadratic wrt compute and linear wrt memory if you use FlashAttention or similar fused self attention calculations. FFWD input is self-attention output. And since the output of self-attention layer is [context, d_model], FFWD layer input will grow as well. Consequently, FFWD layer compute cost will grow as well, no? The cost of FFWD layer according to my calculations is ~(4+2 * true(w3)) * d_model * dff * n_layers * context_size so the FFWD cost grows linearly wrt the context size. So, unless I misunderstood the transformer architecture, larger the context the larger the compute of both self-attention and FFWD is?
      • Kubuxu 2y ago
        FFWD later is independent of context size, each processed token passes thought the same weights.
        • menaerus 2y ago
          So you're saying that if I have a sentence of 10 words, and I want the LLM to predict the 11th word, FFWD compute is going to be independent of the context size? I don't understand how since that very context is what makes the likeliness of output of next prediction worthy, or not? More specifically, FFWD layer is essentially self attention output [context, d_model] matrix matmul'd with W1, W2 and W3 weights?
  • hyperbovine 2y ago
    I’m confused. Backdrop scales linearly w