2 ms·
Data isn’t the only bottleneck. Training models at scale is an expensive and difficult engineering task. It’s something that the “data is the new gold” people a
by janalsncm 4y ago
Data isn’t the only bottleneck. Training models at scale is an expensive and difficult engineering task. It’s something that the “data is the new gold” people always seem to overlook.
And then even if you have the data and the engineering talent, it takes competent leadership to deploy a product and shepherd it to success. (Looking at Google’s graveyard here)
- Imnimo 4y agoTotally agree. Data is a problem when you need millions of labels, but when your training corpus is just "all the text I can find on the internet" and the training objective is self-supervised, it's not what's stopping people. But as you say, actually training a model of this size is really, really tricky. You can't just grab a few ML engineers off the street, write a check to AWS for some compute and have your own GPT-3.