4 ms·
What about the training data corpus ? Other than large cos like Google or Meta, can anyone else procure the same ?
by newyankee 4y ago
What about the training data corpus ? Other than large cos like Google or Meta, can anyone else procure the same ?
- loveparade 4y agoLeaving the legal aspects of crawling aside, I think there is an important distinction here between 1. "can you procure it" and 2. "do you have enough money to process it all" 1. Yes, I think almost anyone can write code to procure the training corpus, in theory, and test it on a small scale 2. No, only the biggest labs and universities have enough resources to process such huge amounts of data and iterate on models with that scale. But that's just a matter of resources that can be overcome with partnerships between industry and academia that are common anyway. All the big labs already have huge efforts underway to reproduce GPT-X and it's just a matter of time before they catch up.
- gorbypark 4y agoAt least as far as what the GPT-3 papers claimed, all (or most?) of the data used for training would be freely available for other competitors/researchers to acquire. Wikipedia, Common Crawl data, etc. I don’t believe OpenAI did their own crawling at all. With OpenAI not being really open, it’s hard to say for sure what exactly ended up in the training materials, though. GPT-4 is even more of a black box to anyone outside of OpenAI with very little information released on how it was trained.