3 ms·
> memory during inference vs the O(T^2) speed and O(T) memory (flash memory) of a GPT model O(T²) speed is only for naive attention. There's a ton of possible
by PoignardAzur 4y ago
> memory during inference vs the O(T^2) speed and O(T) memory (flash memory) of a GPT model
O(T²) speed is only for naive attention. There's a ton of possible optimizations for attention that give you something closer to O(T).
- GaggiX 4y agoYou can only approximate the attention layer if you don't want the O(T^2) speed but no "efficient attention" can beat the quadratic transformer bottleneck in terms of performance.