3 ms·
There are already linear attention advances. Gpt-4-32k is almost certainly using some form of flash attention. Attention isn't really O(n2) anymore.
by famouswaffles 4y ago
There are already linear attention advances. Gpt-4-32k is almost certainly using some form of flash attention.
Attention isn't really O(n2) anymore.
- cs702 4y agoMy understanding is that FlashAttention's memory use is linear, or close to linear in practice, but computation is still O(n²). I'm unaware of anyone being able to apply FlashAttention on, say, a million tokens, because it must execute ~1/2 x 1,000,000^2 x n_head dot-products, each in a subspace with d_head dimensions. That's not exactly computationally cheap!
- famouswaffles 4y agoNo you're right. I mistook you. Compute isn't linear yet.
- lucidrains 4y agoIt is only linear in terms of memory, not compute. Flash attention is a big advance, but not enough for 1 million tokens