> to basically the entire Internet. Please take a few moments to think about how this is not even remotely close to the 'entire Internet'. Let me point out tha

> to basically the entire Internet. Please take a few moments to think about how this is not even remotely close to the 'entire Internet'. Let me point out that Common Crawl doesn't even contain Twitter. That's how incredibly incomplete it is! Google Books estimates there's something like >100m books in existence, of which GPT-3 has seen part (it didn't even do 1 epoch, and like 95% of it was purely English anyway, and it omits most of Github which is why Codex has to be so heavily trained, and...). There are millions of books and papers published every year, endless billions of social media posts and comments in thousands of languages not just English, and on and on and on. The world is a very big place. And this is just text. Text is great, don't get me wrong (it may even have some special properties compared to other modalities), but I'm baffled that in a period where things like Imagen are hitting the front page every few days that I need to point out that there are things other than text, like, y'know, images and video and audio.