Over the years hundreds of variants of RNN, CNN and Transformer have been proposed, but under the same budget of weights and compute, and with the same dataset,

Over the years hundreds of variants of RNN, CNN and Transformer have been proposed, but under the same budget of weights and compute, and with the same dataset, they are very close in performance. Models don't matter, gradient descent finds a way. The real hero is the dataset. And what is in the training set? Language is our best repository for past experience. We have painstakingly collected our lessons, over thousands of years, and transmitted them through culture and books. To recreate them from scratch would take a similarly long time. Our culture is smarter than us, it is the result of our history. So under these reasons I believe the real secret is in the training set. I don't think the problem was the model, but everything else around it.