When you look at the history of scaling up Transformers, e.g. GPT2, most people in the community were in fact quite surprised that just scaling up those models

When you look at the history of scaling up Transformers, e.g. GPT2, most people in the community were in fact quite surprised that just scaling up those models worked so well. And every time you read a lot of opinions that this is the end and scaling them up further will not give (much) improvements. This criticism is lower now but still there. Scaling self-attention has also worked much further than it was initially believed it could work. Initially it was always believed to be the main bottleneck, but this turned out wrong. The feed-forward layers are much more the bottleneck when looking at where most of the compute is spent. It's still a bit unclear how much further we can take the scaling. We are mostly at the limit of what is financial feasible today but as long as some form of generalized Moore's law continues (e.g. number of transistors per dollar), it becomes financial feasible to scale further. At some point we also hit a limit of available data. But maybe self-play (or variants) might solve this. I guess most researchers agree that just scaling up further is maybe not optimal (w.r.t. reaching AGI), or also will not be enough, but it's a somewhat open question.