3 ms·
Yep, the data requirements are especially condemning. We have recently scaled from using Wikipedia as a training corpus to basically the entire Internet. And it
by matusp 4y ago
Yep, the data requirements are especially condemning. We have recently scaled from using Wikipedia as a training corpus to basically the entire Internet. And it helped a lot. But where are we going to scale next? There is simply not enough useful data for the models to learn. People generate a lot of text and language, but most of it is repetitive and not really that useful. The fact that GPT-3 is already two years old and nothing got better than that is telling. If the road to AGI was as simple as scaling, we would have bigger and better models by now. I suspect that even the top labs have found out that GPT-3 is probably near the limits of what they can realistically achieve with deep learning LMs.
- gwern 4y ago> to basically the entire Internet. Please take a few moments to think about how this is not even remotely close to the 'entire Internet'. Let me point out that Common Crawl doesn't even contain Twitter. That's how incredibly incomplete it is! Google Books estimates there's something like >100m books in existence, of which GPT-3 has seen part (it didn't even do 1 epoch, and like 95% of it was purely English anyway, and it omits most of Github which is why Codex has to be so heavily trained, and...). There are millions of books and papers published every year, endless billions of social media posts and comments in thousands of languages not just English, and on and on and on. The world is a very big place. And this is just text. Text is great, don't get me wrong (it may even have some special properties compared to other modalities), but I'm baffled that in a period where things like Imagen are hitting the front page every few days that I need to point out that there are things other than text, like, y'know, images and video and audio.