5 ms·

That's just $10b a year for 15 years, fairly conservative I'd say. I expect a GPT-5 training cluster to cost $10b (Say 100k blackwell chips + associated infras

That's just $10b a year for 15 years, fairly conservative I'd say. I expect a GPT-5 training cluster to cost $10b (Say 100k blackwell chips + associated infrastructure), and a GPT-6 capable cluster to cost $100b. It sounds like a lot, but its just a typical mega-infrastructure funding. The Californian high speed rail is also around $100b for example, and Microsoft has $90b in cash reserves. The gulf states are also dogpiling in AI, because they finally see a true viable alternative to oil: renting out GPU clusters. Just swapping to renewables is not enough, as their economies is predicated on having a huge, simple, defendable profit source that is exportable. Now they can redeploy the spare oil money to a use case, that has predictable demand, extremely capital intensive, and far more productive than building new cities in the desert. The world in the end built $trillions in value of power stations and roads. I expect the same for data centers, having entire farms being replaced by humming GPU racks. But that will take many decades.

lyime 3y ago
[flagged]
hn_throwaway_99 3y ago
> I expect a GPT-5 training cluster to cost $10b (Say 100k blackwell chips + associated infrastructure), and a GPT-6 capable cluster to cost $100b. You just completely made up those numbers.
- anonylizard 3y ago
  GPT-4 cost about $100 mil to train. That's opex (Renting the GPUs). Expect the corresponding capex (buying the GPUs) on Azure's end to be 10x of that. Hence $1 bil GPT-5 being 10x the training cost of GPT-4 is a reasonable estimate. GPT-4 was itself 10x the size of GPT-3. So $10 bil. GPT-6 being another 10x over GPT-5 is therefore just a extrapolation, hence $100 bil.
  - xcv123 3y ago
    The cost of computing decreases over time. No one will pay $100 Billion to train GPT-6. That is absurd. The current top supercomputer in the world (Frontier) cost $600M. It is rumoured that GPT-4 was trained on 10,000 A100 processors released in 2020. Total cost is $100M at $10k each. Today they can buy more powerful hardware and train much larger models for the same cost. https://nvidianews.nvidia.com/news/nvidia-blackwell-platform-arrives-to-power-a-new-era-of-computing https://nvidianews.nvidia.com/news/nvidia-blackwell-platform... https://www.cerebras.net/product-chip https://www.cerebras.net/product-chip
    - anko 3y ago
      > The cost of computing decreases over time. No one will pay $100 Billion to train GPT-6. That is absurd. The current top supercomputer in the world (Frontier) cost $600M. Just wanted to ask the question - do you think frontier has provided more or less value to the world than gpt-4?
      - xcv123 3y ago
        GPT-4 is not a piece of hardware. The comparison makes no sense. GPT-4 was enabled by hardware. The idea is that you use something like a Frontier to create something like a GPT-4. My point was about cost of computing. What kind of GPT can you train with a $600M computer?
    - acchow 3y ago
      It is mind boggling to me that it was so cheap to get GPT4. That is not even 0.1% of California HSR but providing dramatically more value.
      - skybrian 3y ago
        Not hard to beat zero. (So far.) Maybe HSR will work out, but funding is uncertain.
      - deleted 3y ago
        [deleted]
- deleted 3y ago
  [deleted]
deleted 3y ago
[deleted]
vineyardmike 3y ago
I really thought you were going into quite a tangent until the last paragraph. It really proves your point. Arguably, we’ve already seen trillions in value generated from data centers. I still question GPT-6 having a 100b data center. I expect GPT-6 to be out before 2026, since I expect them to regularly release now models, as a matter of marketing. Neither the cash nor hardware purchasing capacity will exist by then. I don’t give the gulf states that much credit either. I think they’re just spraying money at every opportunity. They were pretty into crypto before too. They’ve been around the valley with outsized checks forever. If I had endless money with an expiration date I’d also invest in everything. What would be interesting is to see them really commit and try to on-shore fabs and silicon engineering. That’s the ultimate move to gain geopolitical protection when oil interest stops.
ksec 3y ago
>That's just $10b a year for 15 years, fairly conservative I'd say. Reposting what I wrote over the years [1] ( 11 months ago ) It was only about a year or so AWS was expanding as fast as they could. Bringing up new Datacenter per week. Getting Graviton 2 Wafer from TSMC whenever extra capacity are available on top of their orders. And they dont see any end of expansion in sight. Now it seems all the demand are suddenly gone. [2] ( ~ 2 years ago ) >Amazon said Thursday that revenue growth in its cloud-computing unit slowed in the third quarter to 27.5%. 27.5%. It is lower that their previous 33% over the past few years, but at the current size of AWS growing 27.5% is still ridiculously good. To put this in perspective, if AWS continues to grow at 33% in 2022 and 2023. Then the whole 2023 33% growth alone, would equal to the size of the entire AWS in 2018. It is not the first time Amazon said they are limited by how fast they are building out Datacenter and getting hardware resources ready. It will be interesting to see further details given out in AWS re:Invent 2022. Especially on Graviton roadmap. It is interesting we have a huge increase in compute density in the past 2 years and upcoming 3-5 years. Where a single socket CPU could have 160 Core 320 Thread. and more. Retrofitting older DC with these type of density will simply increase AWS total compute by 2-3x minimum. At the scale of current AWS. Continue to spend money building DC is pretty impressive in my book. [1] https://news.ycombinator.com/item?id=35753169 https://news.ycombinator.com/item?id=35753169 [2] https://news.ycombinator.com/item?id=33384628 https://news.ycombinator.com/item?id=33384628

lyime 3y ago

[flagged]

hn_throwaway_99 3y ago

> I expect a GPT-5 training cluster to cost $10b (Say 100k blackwell chips + associated infrastructure), and a GPT-6 capable cluster to cost $100b. You just completely made up those numbers.

anonylizard 3y ago
GPT-4 cost about $100 mil to train. That's opex (Renting the GPUs). Expect the corresponding capex (buying the GPUs) on Azure's end to be 10x of that. Hence $1 bil GPT-5 being 10x the training cost of GPT-4 is a reasonable estimate. GPT-4 was itself 10x the size of GPT-3. So $10 bil. GPT-6 being another 10x over GPT-5 is therefore just a extrapolation, hence $100 bil.
- xcv123 3y ago
  The cost of computing decreases over time. No one will pay $100 Billion to train GPT-6. That is absurd. The current top supercomputer in the world (Frontier) cost $600M. It is rumoured that GPT-4 was trained on 10,000 A100 processors released in 2020. Total cost is $100M at $10k each. Today they can buy more powerful hardware and train much larger models for the same cost. https://nvidianews.nvidia.com/news/nvidia-blackwell-platform-arrives-to-power-a-new-era-of-computing https://nvidianews.nvidia.com/news/nvidia-blackwell-platform... https://www.cerebras.net/product-chip https://www.cerebras.net/product-chip
  - anko 3y ago
    > The cost of computing decreases over time. No one will pay $100 Billion to train GPT-6. That is absurd. The current top supercomputer in the world (Frontier) cost $600M. Just wanted to ask the question - do you think frontier has provided more or less value to the world than gpt-4?
    - xcv123 3y ago
      GPT-4 is not a piece of hardware. The comparison makes no sense. GPT-4 was enabled by hardware. The idea is that you use something like a Frontier to create something like a GPT-4. My point was about cost of computing. What kind of GPT can you train with a $600M computer?
  - acchow 3y ago
    It is mind boggling to me that it was so cheap to get GPT4. That is not even 0.1% of California HSR but providing dramatically more value.
    - skybrian 3y ago
      Not hard to beat zero. (So far.) Maybe HSR will work out, but funding is uncertain.
    - deleted 3y ago
      [deleted]
deleted 3y ago
[deleted]

deleted 3y ago

[deleted]

vineyardmike 3y ago

I really thought you were going into quite a tangent until the last paragraph. It really proves your point. Arguably, we’ve already seen trillions in value generated from data centers. I still question GPT-6 having a 100b data center. I expect GPT-6 to be out before 2026, since I expect them to regularly release now models, as a matter of marketing. Neither the cash nor hardware purchasing capacity will exist by then. I don’t give the gulf states that much credit either. I think they’re just spraying money at every opportunity. They were pretty into crypto before too. They’ve been around the valley with outsized checks forever. If I had endless money with an expiration date I’d also invest in everything. What would be interesting is to see them really commit and try to on-shore fabs and silicon engineering. That’s the ultimate move to gain geopolitical protection when oil interest stops.

ksec 3y ago

>That's just $10b a year for 15 years, fairly conservative I'd say. Reposting what I wrote over the years [1] ( 11 months ago ) It was only about a year or so AWS was expanding as fast as they could. Bringing up new Datacenter per week. Getting Graviton 2 Wafer from TSMC whenever extra capacity are available on top of their orders. And they dont see any end of expansion in sight. Now it seems all the demand are suddenly gone. [2] ( ~ 2 years ago ) >Amazon said Thursday that revenue growth in its cloud-computing unit slowed in the third quarter to 27.5%. 27.5%. It is lower that their previous 33% over the past few years, but at the current size of AWS growing 27.5% is still ridiculously good. To put this in perspective, if AWS continues to grow at 33% in 2022 and 2023. Then the whole 2023 33% growth alone, would equal to the size of the entire AWS in 2018. It is not the first time Amazon said they are limited by how fast they are building out Datacenter and getting hardware resources ready. It will be interesting to see further details given out in AWS re:Invent 2022. Especially on Graviton roadmap. It is interesting we have a huge increase in compute density in the past 2 years and upcoming 3-5 years. Where a single socket CPU could have 160 Core 320 Thread. and more. Retrofitting older DC with these type of density will simply increase AWS total compute by 2-3x minimum. At the scale of current AWS. Continue to spend money building DC is pretty impressive in my book. [1] https://news.ycombinator.com/item?id=35753169 https://news.ycombinator.com/item?id=35753169 [2] https://news.ycombinator.com/item?id=33384628 https://news.ycombinator.com/item?id=33384628