4 ms·

GPT-4 cost about $100 mil to train. That's opex (Renting the GPUs). Expect the corresponding capex (buying the GPUs) on Azure's end to be 10x of that. Hence $1

GPT-4 cost about $100 mil to train. That's opex (Renting the GPUs). Expect the corresponding capex (buying the GPUs) on Azure's end to be 10x of that. Hence $1 bil GPT-5 being 10x the training cost of GPT-4 is a reasonable estimate. GPT-4 was itself 10x the size of GPT-3. So $10 bil. GPT-6 being another 10x over GPT-5 is therefore just a extrapolation, hence $100 bil.

xcv123 3y ago
The cost of computing decreases over time. No one will pay $100 Billion to train GPT-6. That is absurd. The current top supercomputer in the world (Frontier) cost $600M. It is rumoured that GPT-4 was trained on 10,000 A100 processors released in 2020. Total cost is $100M at $10k each. Today they can buy more powerful hardware and train much larger models for the same cost. https://nvidianews.nvidia.com/news/nvidia-blackwell-platform-arrives-to-power-a-new-era-of-computing https://nvidianews.nvidia.com/news/nvidia-blackwell-platform... https://www.cerebras.net/product-chip https://www.cerebras.net/product-chip
- anko 3y ago
  > The cost of computing decreases over time. No one will pay $100 Billion to train GPT-6. That is absurd. The current top supercomputer in the world (Frontier) cost $600M. Just wanted to ask the question - do you think frontier has provided more or less value to the world than gpt-4?
  - xcv123 3y ago
    GPT-4 is not a piece of hardware. The comparison makes no sense. GPT-4 was enabled by hardware. The idea is that you use something like a Frontier to create something like a GPT-4. My point was about cost of computing. What kind of GPT can you train with a $600M computer?
- acchow 3y ago
  It is mind boggling to me that it was so cheap to get GPT4. That is not even 0.1% of California HSR but providing dramatically more value.
  - skybrian 3y ago
    Not hard to beat zero. (So far.) Maybe HSR will work out, but funding is uncertain.
  - deleted 3y ago
    [deleted]
hn_throwaway_99 3y ago
Besides the "GPT-4 cost about $100 mil to train" number, everything else is still just a number you pulled out of your ass. Why would you estimate that the $100 million training bill would require a billion in GPUs? That's kind of like saying getting a $100 million of water would take a billion in plumbing. I don't know what the number is, but I'm not going to start just making them up.
- tanseydavid 3y ago
  Have you ever heard of a "back-of-the-napkin" estimate? Why so much reluctance to even try to make a reasoned guess?
  - hn_throwaway_99 3y ago
    Because his "back-of-the-napkin" estimate that someone is going to spend $100 billion to train a "GPT-6" (whatever that means) is laughably bad. These aren't estimates, these are just uninformed guesses pulled from nowhere.
    - anonylizard 3y ago
      I provided my reasoning, and you just seem incapable of understanding it. Here, the information literally just wrote an article on $100 billion data center: https://www.theinformation.com/articles/microsoft-and-openai-plot-100-billion-stargate-ai-supercomputer https://www.theinformation.com/articles/microsoft-and-openai... It seems the most ignorant also tend to be the most arrogant.

xcv123 3y ago

The cost of computing decreases over time. No one will pay $100 Billion to train GPT-6. That is absurd. The current top supercomputer in the world (Frontier) cost $600M. It is rumoured that GPT-4 was trained on 10,000 A100 processors released in 2020. Total cost is $100M at $10k each. Today they can buy more powerful hardware and train much larger models for the same cost. https://nvidianews.nvidia.com/news/nvidia-blackwell-platform-arrives-to-power-a-new-era-of-computing https://nvidianews.nvidia.com/news/nvidia-blackwell-platform... https://www.cerebras.net/product-chip https://www.cerebras.net/product-chip

anko 3y ago
> The cost of computing decreases over time. No one will pay $100 Billion to train GPT-6. That is absurd. The current top supercomputer in the world (Frontier) cost $600M. Just wanted to ask the question - do you think frontier has provided more or less value to the world than gpt-4?
- xcv123 3y ago
  GPT-4 is not a piece of hardware. The comparison makes no sense. GPT-4 was enabled by hardware. The idea is that you use something like a Frontier to create something like a GPT-4. My point was about cost of computing. What kind of GPT can you train with a $600M computer?
acchow 3y ago
It is mind boggling to me that it was so cheap to get GPT4. That is not even 0.1% of California HSR but providing dramatically more value.
- skybrian 3y ago
  Not hard to beat zero. (So far.) Maybe HSR will work out, but funding is uncertain.
- deleted 3y ago
  [deleted]

hn_throwaway_99 3y ago

Besides the "GPT-4 cost about $100 mil to train" number, everything else is still just a number you pulled out of your ass. Why would you estimate that the $100 million training bill would require a billion in GPUs? That's kind of like saying getting a $100 million of water would take a billion in plumbing. I don't know what the number is, but I'm not going to start just making them up.

tanseydavid 3y ago
Have you ever heard of a "back-of-the-napkin" estimate? Why so much reluctance to even try to make a reasoned guess?
- hn_throwaway_99 3y ago
  Because his "back-of-the-napkin" estimate that someone is going to spend $100 billion to train a "GPT-6" (whatever that means) is laughably bad. These aren't estimates, these are just uninformed guesses pulled from nowhere.
  - anonylizard 3y ago
    I provided my reasoning, and you just seem incapable of understanding it. Here, the information literally just wrote an article on $100 billion data center: https://www.theinformation.com/articles/microsoft-and-openai-plot-100-billion-stargate-ai-supercomputer https://www.theinformation.com/articles/microsoft-and-openai... It seems the most ignorant also tend to be the most arrogant.