Totally agree. Data is a problem when you need millions of labels, but when your training corpus is just "all the text I can find on the internet" and the trai

Totally agree. Data is a problem when you need millions of labels, but when your training corpus is just "all the text I can find on the internet" and the training objective is self-supervised, it's not what's stopping people. But as you say, actually training a model of this size is really, really tricky. You can't just grab a few ML engineers off the street, write a check to AWS for some compute and have your own GPT-3.