3 ms·
> GPT-4’s predecessor, GPT-3, was trained on forty-five terabytes of text data Is this correct? I've seen varying reports of the training set size.
by 2OEH8eoCRo0 4y ago
> GPT-4’s predecessor, GPT-3, was trained on forty-five terabytes of text data
Is this correct? I've seen varying reports of the training set size.
- versteegen 4y agoGood catch, that's way off the mark. GPT3 was actually trained over 300B tokens, but some of those were repeats (which is now considered bad practice). Based on Table 2.2 in the GPT3 paper, I calculate the actual training set was only ~238B tokens (assuming they shuffle as they should, rather than selecting each minibatch with replacement). That's approximately 1TB of text (~4 characters per token). The source of the 45TB number is this quote from the paper: > The CommonCrawl data was downloaded from 41 shards of monthly CommonCrawl covering 2016 to 2019, constituting 45TB of compressed plaintext before filtering and 570GB after filtering, roughly equivalent to 400 billion byte-pair-encoded tokens.
- jimmySixDOF 4y agoDr Alan D. Thompson has the best tracking of this training set data I am aware of also a 1hr YouTube on the same subject. https://lifearchitect.ai/whats-in-my-ai/ https://lifearchitect.ai/whats-in-my-ai/