3 ms·
> It also happens to be trained on a large chunk of Reddit, since the author decided that this was undeniably the perfect location to obtain high quality, impec
by jancsika 8y ago
> It also happens to be trained on a large chunk of Reddit, since the author decided that this was undeniably the perfect location to obtain high quality, impeccable prose.
Hm, why in the world would they not train it on Sci-hub articles and generate output for which we have plenty of domain experts who can judge the quality?
- czr 8y agoI'm not sure if you're being tongue-in-cheek, but: * Training on Sci-hub or book torrents would probably get them sued, so that's right out. * The point of training the model on reddit-linked-articles with 3+ karma (not reddit comments, as the article suggests) is to filter out worthless content (spam or non-text pages) while still getting a large, diverse sampling of human writing. They're training huge models and they need as much data as possible to do so. * Every native english speaker is a "domain expert" for the purpose of evaluating the model's results. The point at which we need subject-matter experts to evaluate the quality of neural-net generated research papers is many decades away; the excitement around GPT-2 is that it can generate coherent English sentences at all.