3 ms·
For AI on this scale, there are two important costs. The cost of compute, and the cost of engineer salaries. Scaling the compute is probably a bit above linear
by lacker 6y ago
For AI on this scale, there are two important costs. The cost of compute, and the cost of engineer salaries. Scaling the compute is probably a bit above linear due to various overheads, but the cost of engineer salaries is far less than linear, so I would expect the total cost of scaling large-scale AI to be sublinear.
- gwern 6y agoI would expect sublinear scaling in compute cost. Learning curve effects are strong in DL: https://cdn.openai.com/papers/ai_and_efficiency.pdf https://cdn.openai.com/papers/ai_and_efficiency.pdf The more runs you do, the cheaper they get. Plus, OA is no longer running on rented V100s, they have their own MS Azure supercomputer, remember (in fact, GPT-3's evaluation was interrupted because they moved to it), so they remove the enormous cloud margins.