Transformers are synonymous with scaling as it was the first model that kept improving both by adding more and more layers and throwing more training data at it

Transformers are synonymous with scaling as it was the first model that kept improving both by adding more and more layers and throwing more training data at it. All the previous models like CNNs, ResNet, LSTMs etc. hit some depth/training size where no progress was happening anymore as gradient was unreliably flowing down.