> the trend of needing enormous models I would rather say that there isn't a trend of needing enormous models but rather a trend of observing that enormous mod

> the trend of needing enormous models I would rather say that there isn't a trend of needing enormous models but rather a trend of observing that enormous models (and straightforward increases in size) work. We've already seen many other tricks applied for transformer models (essentially, most of "BERTology") that demonstrated that being smarter about taking into account some stuff in training can get a big benefit, but that is all orthogonal to the size issue. That trend would be bucked only if it turned out that for this task 770M is sufficient, i.e. that scaling this 'being smarter' approach from 770M parameters to the 12B parameters of OpenAI Codex does not get a further improvement - however, I am quite convinced that it would be noticeably better than this; and my expectations would be that the eventual outcome of this technique would be that OpenAI Codex v2 (or however that would be called) would beat this by having a model that includes this technique and some other improvements to training process and increases the size beyond 12B parameters.