It was only 'wasted' if the intention was to make a commercial model, not a research model. Google could afford a 100x larger model (than PaLM), a 1000x one wou

It was only 'wasted' if the intention was to make a commercial model, not a research model. Google could afford a 100x larger model (than PaLM), a 1000x one would probably require a consortium including other corporations and possibly governments. Chinchilla used 1.4T tokens - is there even 100x more text data available, in any practical sense - as in potentially available for training? As long as the model size scaling improves performance alone, it makes sense to scale. Only once performance saturates, given same data, it's time to switch attention to training on as much data as possible - and hopefully such model turns out to be superhuman in writing code - improving itself on the algorithmic side, starting the singularity. As an additional point, an AGI in this approach may well need to be trained on images and videos (both containing text in addition to other information) - which requires way more compute power. This would make enormous 'wasteful' text models a way to reach a point where enormous future models and training architectures can be reused for visual and sound data with small changes.