> As per the creators, the OpenAI GPT-3 model has been trained about 45 TB text data from multiple sources which include Wikipedia and books. It's about 400 B

> As per the creators, the OpenAI GPT-3 model has been trained about 45 TB text data from multiple sources which include Wikipedia and books. It's about 400 B tokens. Library if Congress is about 40M books, let's say 50K tokens per book, or about 2T tokens. Not necessarily unique. I would say it's plausible that it was a decent percent of the indexed text available, and even more of the unique content. GPT2 was 10B tokens. Do we have 20T tokens available for GPT4? Maybe. But the low hanging fruit are definitely plucked.