Ironically, google kina did the same thing with BERT.. The original insight came from Jeremy Howards ulmfit. The first transfer learning model for nlp. Tranform

Ironically, google kina did the same thing with BERT.. The original insight came from Jeremy Howards ulmfit. The first transfer learning model for nlp. Tranformers are great an all, but everything after ulmfit has pretty much just been the result of increased model size