Well, we are already hitting pretty strong limits in terms of corpus size. For reference, GPT-3 has 175 billion parameters. That's a few TB (uncompressed). GP

Well, we are already hitting pretty strong limits in terms of corpus size. For reference, GPT-3 has 175 billion parameters. That's a few TB (uncompressed). GPT-3 was trained on 45 TB of text. Most of it comes from 8 years of web crawling, and a significant chunk of the remainder comes from online book corpora [0]. At some point you just run out of human-written text to train on. You can't just keep making the model bigger (by orders of magnitude I mean) without scaling the input data the same. And at some point we just run out of the latter. [0]: https://in.springboard.com/blog/openai-gpt-3/ https://in.springboard.com/blog/openai-gpt-3/