3 ms·

So, it's beaten by Transformer models. Transformers don't all use quadratic attention; there's a whole zoo of alternate attentions. And even the RWKV author re

by gwern 3y ago

So, it's beaten by Transformer models. Transformers don't all use quadratic attention; there's a whole zoo of alternate attentions. And even the RWKV author refers to it as 'transformer mode'.