3 ms·

Btw, why do transformers have a limit input size in the first place? I'm pretty sure the self-attention mechanisms scale (although with bad complexity) to arbit

• MacsHeadroom 3y ago
  >(although with bad complexity) Because of exactly that. Also the attention mechanism is baked in during pretraining. So whatever max context length you want increases the compute cost of training by at least a function of said "bad complexity." Even just 4096 tokens of max context is much more expensive to train than 2048. So if we want models with 8k, 32k, or more context then the training costs get out of hand quickly.
  • rewq4321 3y ago
    > Also the attention mechanism is baked in during pretraining IIUC, this is no longer necessarily true with positional encodings like ALiBi: https://github.com/ofirpress/attention_with_linear_biases https://github.com/ofirpress/attention_with_linear_biases