Good catch, that's way off the mark. GPT3 was actually trained over 300B tokens, but some of those were repeats (which is now considered bad practice). Based on

Good catch, that's way off the mark. GPT3 was actually trained over 300B tokens, but some of those were repeats (which is now considered bad practice). Based on Table 2.2 in the GPT3 paper, I calculate the actual training set was only ~238B tokens (assuming they shuffle as they should, rather than selecting each minibatch with replacement). That's approximately 1TB of text (~4 characters per token). The source of the 45TB number is this quote from the paper: > The CommonCrawl data was downloaded from 41 shards of monthly CommonCrawl covering 2016 to 2019, constituting 45TB of compressed plaintext before filtering and 570GB after filtering, roughly equivalent to 400 billion byte-pair-encoded tokens.