2 ms·
It's beaten by only two weights available models, it beats LLama 13B and Dolly. What do you mean it's training in a quasi-Transformer manner? Do you mean the la
by rolisz 3y ago
It's beaten by only two weights available models, it beats LLama 13B and Dolly. What do you mean it's training in a quasi-Transformer manner? Do you mean the language pretraining? That's been done before Transformers.
RWKV doesn't have the quadratic attention of Transformers, so I'd say it's pretty different.
- gwern 3y agoSo, it's beaten by Transformer models. Transformers don't all use quadratic attention; there's a whole zoo of alternate attentions. And even the RWKV author refers to it as 'transformer mode'.