4 ms·
Shouldn't the memory needed scale quadratically with sequence length rather than the linear scaling they have in their equations?
by letitgo12345 3y ago
Shouldn't the memory needed scale quadratically with sequence length rather than the linear scaling they have in their equations?
- visarga 3y agoNot if they use flash attention which solves the problem in fixed memory by working tile by tile. They never materialise the whole attention matrix at once. But the computation time is still quadratic.
- letitgo12345 3y agoThey present it as an article about transformers in general, not ones using Flash Attention. Anyway maybe they're presenting per token memory requirement instead of the requirement for the entire sequence at once.
- visarga 3y agoThey don't mention it explicitly except in one place: > GPT-NeoX achieves 150 TFLOP/s/A100 with normal attention and 180 FLOP/s/A100 with Flash Attention. This advice implies they are using flash attention.
- akomtu 3y agoHas anyone tried to make it cubic? LLMs have reached the current limit by increasing window size, but who knows what increasing dimensionality will bring.