What happened was Google beat itself at its own game, if you will. Google PaLM-1 is beaten so often simply because Google Chinchilla scaling is so much better.

What happened was Google beat itself at its own game, if you will. Google PaLM-1 is beaten so often simply because Google Chinchilla scaling is so much better. (Note that PaLM-2 is not benchmarked in OP.) PaLM-1 is hobbled by the fact that it was probably the largest (because the last) LLM to be trained with the Kaplan scaling laws rather than the Chinchilla. As soon as Chinchilla came out, no one would train like PaLM-1 again, because it was giving up so much performance compared to if one had instead trained a much smaller Chinchilla-optimal model. (This had the interesting consequence that PaLM-1 would thereby remain the largest, by parameter-count, dense LLM trained for probably years to come - because why would you train one inefficiently as that, while a larger-than-PaLM-1 Chinchilla-optimal model would require staggering levels of compute+data.) The PaLM-1 paper came out within days of the Chinchilla paper, and many people noted that this pointed to extraordinary levels of dysfunctionality within Google - that DeepMind would not tell Google Brain that they were wasting literally millions of dollars of compute by training a model in what DeepMind was busy showing was a very suboptimal way.