3 ms·
Powerful models are trained using large amounts of data that most people do not have access to. They are also trained powerful infrastructure that most people
by 29athrowaway 6y ago
Powerful models are trained using large amounts of data that most people do not have access to.
They are also trained powerful infrastructure that most people do not have access to.
So, to speak of democratization is interpreting the current state of affairs incorrectly.
- boulos 6y agoSee my sibling comment, but the dataset that the GPT-2 folks did was “just grab whatever people on reddit upvoted”. Any grad student might have done the same (and in a sense, James Hays and Alyosha Efros did something like that long ago with flickr data for im2gps). While GPT-3 is trained on a massive V100 cluster, you could probably do so with a much smaller one / there exist interesting smaller models. It’s expensive to rent this class of equipment, but it is available. The distinction is that OpenAI made a focused bet. Most research funding and labs spread their bets heavily (e.g., each institution or researcher gets $50k/yr of funding). OpenAI takes a different stance, but it’s not clear that they’re spending even as much as say Google, Facebook or other large institutions. It’s also not obvious that you even have to play the same game to get similar results.
- mirekrusin 6y agoThey trained it on freely available datasets managed by nonprofits as somebody mentioned. Powerfull infrastructure can be democratized in similar way that SETI@home, cryptocurriencies and other P2P projects do it. In theory this can be done.