I mean, it was clearly a better language model than the at-the-time best model (which, if I recall correctly, was Yang et al's mixture-of-softmaxes). But the bi

I mean, it was clearly a better language model than the at-the-time best model (which, if I recall correctly, was Yang et al's mixture-of-softmaxes). But the biggest surprise over time has been how big of a sponge large transformer models are (absorbing huge amounts of data), which was definitely not evident until later. I was also personally surprised at how long you need to train these models (in terms of epochs) but I think some of the work leading to that (around optimal batch size, etc.) was already being done. I think that actually changed quite a lot over the next few years, from really long training (many epochs over the same data) to really "short" training (1-2 epochs on a much larger dataset). The ML community took to it very quickly; multi-headed attention was being used in everything very soon afterwards, and LSTMs were no longer used. But it took a few months. HuggingFace really kicked a lot of this off, then BERT, then DistilBERT.