It is fundamentally impossible for linear attention to outperform quadratic attention. What you could do instead is have a few quadratic attention layers in the

It is fundamentally impossible for linear attention to outperform quadratic attention. What you could do instead is have a few quadratic attention layers in the first layers and one in the last and have everything else use linear attention. This would allow you to have a context length of millions of tokens within 24 GB VRAM for a 7b model because you have eliminated the KV cache for the inner layers, while still retaining the ability to perform any token vs any other token attention. The final layer is also somewhat important since it allows the model to see all its reasoning outputs and attend over its previous reasoning steps.