3 ms·

The price of a cluster similar to the one OpenAI used is orders of magnitude higher. We are talking about a price tag that at the very minimum would be in the 1

The price of a cluster similar to the one OpenAI used is orders of magnitude higher. We are talking about a price tag that at the very minimum would be in the 150M+ range, you can probably multiply that price by 2 or 3 (10 thousand V100, 300k CPUs, infiniband between every node, network storage etc.). Almost no one has that kind of money to invest up front (especially not an hypothetical open source organization like discussed here). Even if somehow one had that money available, for this to be more economic than renting you need to have it running at full capacity at all times for years, not many research groups can come up with many interesting projects at a scale large enough to utilize that kind of cluster efficiently. So buying something like that is only conceivable for huge research groups with hundreds of researchers and tons of cash: Google Brain, DeepMind, Microsoft and probably Facebook. Even OpenAI wasn't able to afford it directly...

aquajet 6y ago
> especially not an hypothetical open source organization like discussed here Hypothetical open source organization is not hypothetical: https://www.eleuther.ai/home https://www.eleuther.ai/home However, the compute constraints are correct. We plan on using TFRC's TPUs in order to replicate GPT-2 (with a few changes to support scaling to GPT-3 and beyond as well as the larger context length). With success on a GPT-2 model we hope to garner enough interest to get more compute to train a full-sized GPT-3 model. After that, it's 1T or bust.
- webmaven 6y ago
  > After that, it's 1T or bust. A trillion parameters? Damn.
webmaven 6y ago
Existing HPC projects (aka "supercomputers") show that money is available for hardware-intensive research projects (perhaps the lesson here is that the US needs an AI/ML focused National Lab), and as far as the cost of creating a bespoke cluster for a single project, I think that running it full bore for a project and then selling it off ASAP would avoid most of the downsides you've mentioned. Again, I have a feeling that big-budget render farms have trod this path before.

aquajet 6y ago

> especially not an hypothetical open source organization like discussed here Hypothetical open source organization is not hypothetical: https://www.eleuther.ai/home https://www.eleuther.ai/home However, the compute constraints are correct. We plan on using TFRC's TPUs in order to replicate GPT-2 (with a few changes to support scaling to GPT-3 and beyond as well as the larger context length). With success on a GPT-2 model we hope to garner enough interest to get more compute to train a full-sized GPT-3 model. After that, it's 1T or bust.

webmaven 6y ago
> After that, it's 1T or bust. A trillion parameters? Damn.

webmaven 6y ago

Existing HPC projects (aka "supercomputers") show that money is available for hardware-intensive research projects (perhaps the lesson here is that the US needs an AI/ML focused National Lab), and as far as the cost of creating a bespoke cluster for a single project, I think that running it full bore for a project and then selling it off ASAP would avoid most of the downsides you've mentioned. Again, I have a feeling that big-budget render farms have trod this path before.