22 ms·
The performance advantages of transformer architectures have little to do with "transformers" themselves, but rather other architecture changes that were made a
by riotnrrd 5y ago
The performance advantages of transformer architectures have little to do with "transformers" themselves, but rather other architecture changes that were made along the way. They can be matched or beat in performance by tweaked conventional CNNs: https://arxiv.org/abs/2201.03545 https://arxiv.org/abs/2201.03545
- mountainriver 5y agoYeah LSTMs can outperform transformers on a lot of small tasks. What transformers do exceptionally well is scale.
- robbedpeter 5y agoPart of the scaling benefit comes from the efficiency of training. LSTM and other recurrent architectures take more bits per parameter and the attention mechanism doesn't optimize for context - each lstm neuron uses gating individually, where transformers attention mechanisms can use the context of an entire layer. LSTMs with attention are super interesting, and there are some neat self-growing hopfield networks with attention that could push into transformer style efficacy and efficiency gains.