I would expect sublinear scaling in compute cost. Learning curve effects are strong in DL: https://cdn.openai.com/papers/ai_and_efficiency.pdf https://cdn.opena

I would expect sublinear scaling in compute cost. Learning curve effects are strong in DL: https://cdn.openai.com/papers/ai_and_efficiency.pdf https://cdn.openai.com/papers/ai_and_efficiency.pdf The more runs you do, the cheaper they get. Plus, OA is no longer running on rented V100s, they have their own MS Azure supercomputer, remember (in fact, GPT-3's evaluation was interrupted because they moved to it), so they remove the enormous cloud margins.