3 ms·
We don't know that. No one has demonstrated it. It's very likely that at larger scales, for a given amount of compute you cannot train a traditional RNN to be a
by p1esk 3y ago
We don't know that. No one has demonstrated it. It's very likely that at larger scales, for a given amount of compute you cannot train a traditional RNN to be as a good as a transformer.
- famouswaffles 3y agoWe are saying the same thing. Transformers are more compute efficient than RNNs. Nobody is denying that but the switch from RNNs didn't precede some performance wall(i.e it's not like we were training bigger RNNs that weren't getting better). We use Transformers today in large part because they got rid of recursion and in effect could massively parallelize compute.