See my sibling comment, but the dataset that the GPT-2 folks did was “just grab whatever people on reddit upvoted”. Any grad student might have done the same (a

See my sibling comment, but the dataset that the GPT-2 folks did was “just grab whatever people on reddit upvoted”. Any grad student might have done the same (and in a sense, James Hays and Alyosha Efros did something like that long ago with flickr data for im2gps). While GPT-3 is trained on a massive V100 cluster, you could probably do so with a much smaller one / there exist interesting smaller models. It’s expensive to rent this class of equipment, but it is available. The distinction is that OpenAI made a focused bet. Most research funding and labs spread their bets heavily (e.g., each institution or researcher gets $50k/yr of funding). OpenAI takes a different stance, but it’s not clear that they’re spending even as much as say Google, Facebook or other large institutions. It’s also not obvious that you even have to play the same game to get similar results.