4 ms·
To be honest this makes me less excited about linear transformers. If even heavily optimized, they are still (nearly) no better than normal flash attention up
by thomasahle 3y ago
To be honest this makes me less excited about linear transformers.
If even heavily optimized, they are still (nearly) no better than normal flash attention up to context length 10^4.
And then you haven't even started to account for the degradation in learning.
Maybe if you're doing 100k attention at inference it starts making sense... But then there are other methods you can start using too.
- f_devd 3y agoYou should check out MoE-Mamba (https://arxiv.org/abs/2401.04081 https://arxiv.org/abs/2401.04081), it's faster and more accurate than Transformer-MoE. Of course only time will tell if it's better when scaled up further than the paper goes.