4 ms·
What % of all digitized and reasonably easy-to-access text data did they use to train GPT-3? I'm wondering whether the current limits on GPT-n are computation o
by hntrader 6y ago
What % of all digitized and reasonably easy-to-access text data did they use to train GPT-3? I'm wondering whether the current limits on GPT-n are computation or data.
- kortex 6y ago> As per the creators, the OpenAI GPT-3 model has been trained about 45 TB text data from multiple sources which include Wikipedia and books. It's about 400 B tokens. Library if Congress is about 40M books, let's say 50K tokens per book, or about 2T tokens. Not necessarily unique. I would say it's plausible that it was a decent percent of the indexed text available, and even more of the unique content. GPT2 was 10B tokens. Do we have 20T tokens available for GPT4? Maybe. But the low hanging fruit are definitely plucked.