3 ms·
That's because the big change was moving from LSTM RNNs to Transformers. If you had tried to train a scaled-up RNN on GPT-3-level compute and data, it would hav
by gwern 3y ago
That's because the big change was moving from LSTM RNNs to Transformers. If you had tried to train a scaled-up RNN on GPT-3-level compute and data, it would have been a much better RNN, yes... and terrible compared to GPT-3, because the RNN scaling law is much worse and they are unable to make good use of context. See Kaplan et al 2020, key graph: https://gwern.net/doc/ai/nn/transformer/gpt/2020-kaplan-figure7-rnnsvstransformers.png https://gwern.net/doc/ai/nn/transformer/gpt/2020-kaplan-figu... (I'm not going to try to calculate it, but by 'terrible' I mean that the bent scaling law would give you, like, GPT-2 performance for GPT-3-level investment. Suffice it to say that the DL conversations would be a trifle different today if that had been what happened.)
All the other stuff has been largely minor polishes. For example, using tokens instead of characters? Nah. ByT5 works fine, as do very large context window approaches like Jukebox. Large BPE vocabs may be somewhat more efficient but the value is nothing like RNN->Transformer. A lot more training data? Doesn't matter if your model scaling law is bent because the model can't make use of more data meaningfully - it's not like it's overfitting to smaller amounts of data either, it's just underfitting everything. Structure? Irrelevant, the point of web scrapes is bulk and diversity. etc
- rolisz 3y agoWell, RNNs are making a comeback: https://lmsys.org/blog/2023-05-10-leaderboard/ https://lmsys.org/blog/2023-05-10-leaderboard/ RWKV is an RNN based model and it's competitive (with the open source models, not with GPT4)
- gwern 3y agoRWKV is interesting but hasn't shown Transformer-beating improvement yet, and its key is that it's training in a quasi-Transformer manner to begin with, which is why it gets the better scaling law that makes it feasible, and so demonstrates my point. (RWKV is more like a Transformer which is easily converted into a RNN for sampling, which is an equivalence we've known for a long time.)
- rolisz 3y agoIt's beaten by only two weights available models, it beats LLama 13B and Dolly. What do you mean it's training in a quasi-Transformer manner? Do you mean the language pretraining? That's been done before Transformers. RWKV doesn't have the quadratic attention of Transformers, so I'd say it's pretty different.
- gwern 3y agoSo, it's beaten by Transformer models. Transformers don't all use quadratic attention; there's a whole zoo of alternate attentions. And even the RWKV author refers to it as 'transformer mode'.