3 ms·
> so dropping $1bn or more on scaling GPT up by another factor of 100x is entirely plausible right now. I'd note it's rare that cost of scaling a computing pro
by maerF0x0 6y ago
> so dropping $1bn or more on scaling GPT up by another factor of 100x is entirely plausible right now.
I'd note it's rare that cost of scaling a computing project is a linear growth function.
100x-ing an AI project could be 1100x cost.
- lacker 6y agoFor AI on this scale, there are two important costs. The cost of compute, and the cost of engineer salaries. Scaling the compute is probably a bit above linear due to various overheads, but the cost of engineer salaries is far less than linear, so I would expect the total cost of scaling large-scale AI to be sublinear.
- gwern 6y agoI would expect sublinear scaling in compute cost. Learning curve effects are strong in DL: https://cdn.openai.com/papers/ai_and_efficiency.pdf https://cdn.openai.com/papers/ai_and_efficiency.pdf The more runs you do, the cheaper they get. Plus, OA is no longer running on rented V100s, they have their own MS Azure supercomputer, remember (in fact, GPT-3's evaluation was interrupted because they moved to it), so they remove the enormous cloud margins.