3 ms·
transformers were also just better at the LM task than 2018 RNNs for equal amount of flop training
by whimsicalism 3y ago
transformers were also just better at the LM task than 2018 RNNs for equal amount of flop training
- VHRanger 3y agoYeah, that's just the training stability part to my knowledge
- whimsicalism 3y agothey're also just less capable models. like just adding attention on top of an RNN made them a lot better
- SpaceManNabs 3y agoCalculating self-attention is still quadratic though. So you get the negatives of transformers there too.