OpenAI training data oversamples sites which got 3 reddit likes[1], because it improves model. Most large scale training data has some filter on commoncrawl.

OpenAI training data oversamples sites which got 3 reddit likes[1], because it improves model. Most large scale training data has some filter on commoncrawl. [1]: https://paperswithcode.com/dataset/webtext https://paperswithcode.com/dataset/webtext