32 ms·
But seriously, was Stackoverflow part of the training data used to train GPT-3? Would definitely be an interesting fine tuning experiment
by deeeeplearning 6y ago
But seriously, was Stackoverflow part of the training data used to train GPT-3? Would definitely be an interesting fine tuning experiment
- stochastimus 6y agoFrom what I've read, the answer is "yes", stackoverflow was crawled. EDIT: I looked and stackoverflow is included in the Common Crawl dataset, which is one of the datasets on which GPT-3 was trained. Having said that, it's not clear to me the degree of coverage of that domain guaranteed by that crawl... looks pretty comprehensive, though. http://index.commoncrawl.org/CC-MAIN-2020-24-index?url=*.stackoverflow.com&output=json http://index.commoncrawl.org/CC-MAIN-2020-24-index?url=*.sta...
- sillysaurusx 6y agoThe training data was so large that I don’t think a single epoch was used. I.e. it’s not a guarantee that any specific site is known by GPT-3 just because it’s in the data. But, with the nature of random samples, it’s probably likely that it saw at least a little bit of SO.