3 ms·

Access to the right compute infrastructure. While the training itself cost 4.6$M (unsure if this include hyper parameter search done on smaller models beforehan

Access to the right compute infrastructure. While the training itself cost 4.6$M (unsure if this include hyper parameter search done on smaller models beforehand), even if an open source organization had that money available, I don't believe they would be able to access the kind of infra needed to train GPT-3. The closest thing would probably be some TPU pods on GCP, but even then, I think the biggest pod you can publicly access is too small (unless you are ok training for 3 years).

webmaven 6y ago
> even if an open source organization had that money available, I don't believe they would be able to access the kind of infra needed to train GPT-3. The closest thing would probably be some TPU pods on GCP, but even then, I think the biggest pod you can publicly access is too small (unless you are ok training for 3 years). Surely, at that budgetary scale, buying/building your own infrastructure would be cheaper than renting it? This is the same economic calculation large animation studios have to make WRT render farms (which also use similar types of hardware).
- Voloskaya 6y ago
  The price of a cluster similar to the one OpenAI used is orders of magnitude higher. We are talking about a price tag that at the very minimum would be in the 150M+ range, you can probably multiply that price by 2 or 3 (10 thousand V100, 300k CPUs, infiniband between every node, network storage etc.). Almost no one has that kind of money to invest up front (especially not an hypothetical open source organization like discussed here). Even if somehow one had that money available, for this to be more economic than renting you need to have it running at full capacity at all times for years, not many research groups can come up with many interesting projects at a scale large enough to utilize that kind of cluster efficiently. So buying something like that is only conceivable for huge research groups with hundreds of researchers and tons of cash: Google Brain, DeepMind, Microsoft and probably Facebook. Even OpenAI wasn't able to afford it directly...
  - aquajet 6y ago
    > especially not an hypothetical open source organization like discussed here Hypothetical open source organization is not hypothetical: https://www.eleuther.ai/home https://www.eleuther.ai/home However, the compute constraints are correct. We plan on using TFRC's TPUs in order to replicate GPT-2 (with a few changes to support scaling to GPT-3 and beyond as well as the larger context length). With success on a GPT-2 model we hope to garner enough interest to get more compute to train a full-sized GPT-3 model. After that, it's 1T or bust.
    - webmaven 6y ago
      > After that, it's 1T or bust. A trillion parameters? Damn.
  - webmaven 6y ago
    Existing HPC projects (aka "supercomputers") show that money is available for hardware-intensive research projects (perhaps the lesson here is that the US needs an AI/ML focused National Lab), and as far as the cost of creating a bespoke cluster for a single project, I think that running it full bore for a project and then selling it off ASAP would avoid most of the downsides you've mentioned. Again, I have a feeling that big-budget render farms have trod this path before.

webmaven 6y ago

> even if an open source organization had that money available, I don't believe they would be able to access the kind of infra needed to train GPT-3. The closest thing would probably be some TPU pods on GCP, but even then, I think the biggest pod you can publicly access is too small (unless you are ok training for 3 years). Surely, at that budgetary scale, buying/building your own infrastructure would be cheaper than renting it? This is the same economic calculation large animation studios have to make WRT render farms (which also use similar types of hardware).

Voloskaya 6y ago
The price of a cluster similar to the one OpenAI used is orders of magnitude higher. We are talking about a price tag that at the very minimum would be in the 150M+ range, you can probably multiply that price by 2 or 3 (10 thousand V100, 300k CPUs, infiniband between every node, network storage etc.). Almost no one has that kind of money to invest up front (especially not an hypothetical open source organization like discussed here). Even if somehow one had that money available, for this to be more economic than renting you need to have it running at full capacity at all times for years, not many research groups can come up with many interesting projects at a scale large enough to utilize that kind of cluster efficiently. So buying something like that is only conceivable for huge research groups with hundreds of researchers and tons of cash: Google Brain, DeepMind, Microsoft and probably Facebook. Even OpenAI wasn't able to afford it directly...
- aquajet 6y ago
  > especially not an hypothetical open source organization like discussed here Hypothetical open source organization is not hypothetical: https://www.eleuther.ai/home https://www.eleuther.ai/home However, the compute constraints are correct. We plan on using TFRC's TPUs in order to replicate GPT-2 (with a few changes to support scaling to GPT-3 and beyond as well as the larger context length). With success on a GPT-2 model we hope to garner enough interest to get more compute to train a full-sized GPT-3 model. After that, it's 1T or bust.
  - webmaven 6y ago
    > After that, it's 1T or bust. A trillion parameters? Damn.
- webmaven 6y ago
  Existing HPC projects (aka "supercomputers") show that money is available for hardware-intensive research projects (perhaps the lesson here is that the US needs an AI/ML focused National Lab), and as far as the cost of creating a bespoke cluster for a single project, I think that running it full bore for a project and then selling it off ASAP would avoid most of the downsides you've mentioned. Again, I have a feeling that big-budget render farms have trod this path before.