2 ms·

Best sentence in the article: "It also happens to be trained on a large chunk of Reddit, since the author decided that this was undeniably the perfect location

• czr 7y ago
  While snappy, this sentence is misleading. The model is trained on webpages whose links have received at least 3 karma on reddit, not on reddit comments themselves [0]. The motivation for this choice is not to find impeccable prose, but rather to establish a lower bound on quality for the dataset, and avoid training the model on too much of the spam/nonsense text that is frequent in e.g. CommonCrawl. Also, the "author" is not a single person, but rather 6+ people [1]. [0] https://openai.com/blog/better-language-models/ https://openai.com/blog/better-language-models/ [1] https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf https://d4mucfpksywv.cloudfront.net/better-language-models/l...