6 ms·

GDP adjustments are warranted, but it is more stark than both the estimates suggest. The megaprojects of the previous generations all had decades long deprecia

by manquer 6mo ago

GDP adjustments are warranted, but it is more stark than both the estimates suggest. The megaprojects of the previous generations all had decades long depreciation schedules. Many 50-100+ year old railways, bridges, tunnels or dams and other utilities are still in active use with only minimal maintenance Amortized Y-o-Y the current spends would dwarf everything at the reported depreciation schedule of 6(!) years for the GPUs - the largest line item.

wr2 6mo ago
Also railways would always have alternative uses at that time - e.g. logistics in warfare. What other uses do GPU's have that are critical...? lol In addition to your points, this is why I always laugh when people do backward comparisons. What characteristics do they share in common? Very little.
- jamesknelson 6mo ago
  GPUs do have a use in warfare though. I mean, LLMs are basically offensive weapons disguised as software engineers. Sure, LLMs can kind of put together a prototype of some CRUD app, so long as it doesn’t need to be maintainable, understandable, innovative or secure. But they excel at persisting until some arbitrary well defined condition is met, and it appears to be the case that “you gain entry to system X” works well as one of those conditions. Given the amount of industrial infrastructure connected to the internet, and the ways in which it can break, LLMs are at some point going to be used as weapons. And it seems likely that they’ll be rather effective. FWIW, people first saw TNT as a way to dye things yellow, and then as a mining tool. So LLMs starting out as chatbots and then being seen as (bad) software engineers does put them in good company.
  - bigfatkitten 6mo ago
    > GPUs do have a use in warfare though. Unclassified public cloud GPUs are completely useless when your warfighting workloads are at the SECRET level or above.
    - jhide 6mo ago
      They’re unclassified public cloud GPUs today, much the same as the massive industrial base of the United States was churning out harmless consumer widgets in 1939. Those widget makers happened to be reconfigurable into weapon makers, and so wartime production exploded from 2% to 40% of GDP in 5 years [1]. But the total industrial output of course didn’t expand by nearly that much. I think it’s maybe plausible that private compute feels similar in the next do-or-die global war. [1] https://eh.net/encyclopedia/the-american-economy-during-world-war-ii/#:~:text=from%20just%20two%20percent https://eh.net/encyclopedia/the-american-economy-during-worl...
      - bigfatkitten 6mo ago
        The United States has almost no domestic capability to produce advanced semiconductors. There is no abundance of industrial capacity cranking out GPUs that can be quickly diverted from AI companies into weapon systems. Even if private compute was at a level of maturity where you could use it for classified workloads, knowing that the infrastructure is being managed by someone in India or China, securely getting data into and out of that infrastructure is still a mostly unsolvable problem.
        tester756 6mo ago
        wut? Intel with 18A can do it
        bigfatkitten 6mo ago
        Its low yields and tiny volumes are part of what gets the US from “no capacity” to “almost no capacity.”
        tester756 6mo ago
        yields are constantly improving on monthly basis, according to executives around 7% per month, so the capability is definitely there, but yields still needs some time
  - wr2 6mo ago
    Imagine comparing something that has a useful life of 100+ years vs a thing that is worn out, much less durable, and needs replacing much more often and can become obselete from innovation within its own product category. Comical. China can continue innovating on GPUs and all this existing spend to stock up on compute is a waste. Again, comical. Moreover China has energy capacity that the US does not. Meaning all those GPU's that deliver less performance per watt? Yep going in the bin. So yeah.. carry on telling me how this is going to yield some supreme advantage lmao.
- jhide 6mo ago
  On the topic of warfare, wars are fought differently now. Compute will be mentioned in the same breath as total manufacturing output if a global war between superpowers erupts. In highly competitive industries this is already the case. Compute will be part of industrial mobilization in the same way that physical manufacturing or transportation capacity were mobilized in WWII. I’m not an expert on military computing but my intuition is that FLOPS are probably even more easily fungible into wartime compute than widget makers, and the US was able to go widgets->weapons on an unbelievable scale last time.
  - andrewljohnson 6mo ago
    You could argue that compute was a decisive factor in World War II even (used in code breaking and designing nuclear weapons).
  - AngryData 6mo ago
    There are plenty of military uses for computing, but I also find it hard to believe anything but a handful of datacenters are or could be a major factor in anything but a completely 1 sided war. They are very vulnerable targets that are easy to locate and require large amounts of power and cooling. I also just don't see the application, encryption capabilities far exceed the compute available needed for decryption and computing precision and speed with even 20 year old tech far exceeds the precision of anything you would want to control. Even with tangible banefits, say 10% more or less casualties than there would be otherwise, in an exchange with anything resembling a peer military force im not sure it matters because everybody already loses.
  - 7952 6mo ago
    Is that in terms of data centres or chips on the battlefield? Surely the latter is most important. Or will war alwys have perfect connectivity.
- naasking 6mo ago
  > What other uses do GPU's have that are critical...? lol GPUs are essential to every kind of scientific and engineering simulation you can think of. AI-accelerated simulations are a huge deal now.
  - wr2 6mo ago
    GPUs that have lives of..? Now compare that with the life a rail road. Amusing.
    - naasking 6mo ago
      Some of those railroad bridges might never have been constructed without those simulations.
deleted 6mo ago
[deleted]
gravypod 6mo ago
The side effects of spending funds on these mega projects is also something to consider. NASA spending has created a huge pile of technologies that we use day to day: https://en.wikipedia.org/wiki/NASA_spin-off_technologies https://en.wikipedia.org/wiki/NASA_spin-off_technologies.
- delusional 6mo ago
  > NASA spending has created a huge pile of technologies that we use day to day We're a little too early to know if that's the case here too. I do foresee a chance at a reality where AI is a dead end, but after it we have a ton of cheap GPU compute lying about, which we all rush to somehow convert into useful compute (by emulating CPU's or translating traditional algorithms into GPU oriented ones or whatever).
  - PunchyHamster 6mo ago
    I think there is little chance it is a "dead end", it's here to stay but at least LLMs seem to have hit the diminishing returns curve already, despise what investors might think, and so far none of the big providers actually makes money for all that investment
    - etempleton 6mo ago
      I think for many, if LLMs and AI only improves marginally in the next 5-10 years it is effectively a dead end. The capital expenditure necessitates AI does something exponentially more valuable than what it does now. I think we are saying the same thing.i just think the pull back on AI will be dramatic unless something amazing happens very soon.
      - brookst 6mo ago
        I just don’t see it. Both professionally and personally I’m producing so much more now. Back burner projects that weren’t worth months of my time are easily worth a few hours and $20 or whatever. Why would I pull back?
rayiner 6mo ago
Great point!
Lerc 6mo ago
The shovels and labour used to make those things where not depreciated. The GPUs are the shovels, not the project. AI at any capability will retain that capbibilty forever. It only gets reduced in value by superior developments. Which are built upon technologies that the previous generation developed.
- jiggawatts 6mo ago
  > retain that capbibilty forever Not really. The base training data cutoff will quickly render models useless as they fail to keep up with developments. Translating some Farsi news articles about the war was hilarious, Gemini Pro got into a panic. ChatGPT either accused me of spreading fake news, or assumed this was some sort of fantasy scenario.
  - m00x 6mo ago
    That's GPT4 thinking. New models use tools to look at current events or latest versions, and rely very little on weight knowledge.
    - zozbot234 6mo ago
      You can pull new information into the context via RAG, but that is expensive and only gives very shallow understanding compared to retraining.
  - nl 6mo ago
    Not really. For coding I care mostly about reasoning ability which is uncorrelated with cut off
  - jeremyjh 6mo ago
    Karpathy - and others - consider the pre-training knowledge as much a liability as an asset. If we could just retain the emergent reasoning and language capability without the hazy recollections the models would likely be stronger.
- loandbehold 6mo ago
deleted 6mo ago
[deleted]
pembrook 6mo ago
Only half of the rail capacity that existed during the railroad boom times was still in use by the 1970s. Lots of it was never really used at all after various railroads went bankrupt. But your point still stands. That said, I'm pretty sure in a compute-hungry AI world you aren't going to retire GPUs every 6 years anymore. Even if compute capacity jumps such that current H100s only represent 10% of total compute available in 6 years, you're still running those H100s until they turn to dust. I just think it's hard to compare localized railroad infrastructure to globalized AI capacity and say one was more rational than the other on a % of GDP basis until the history actually plays out. If you compare global investment in nuclear weapons it would dwarf the manhattan project and AI thus far, and yet, 99.99999% of nuclear weapons investment is just "wasted" capacity in that it has never been "used." But the value it has created in other ways (MAD-enabled peace) has surely been profitable on net. Nobody would have predicted this at the time. Playing armchair internet pessimist about the "new thing" always makes you feel smart but is usually not a good idea since you always mis-price what you don't know about the future (which is almost everything).
brookst 6mo ago
I’m not sure tax depreciation rates are the best measure here. Those GPUs will be used for much longer than 6 years, and the returns from the businesses will be an order of magnitude longer.
- mfuzzey 6mo ago
  actually the physical lifetime (not financial depreciation) for AI data center GPUs is even lower (3 to 4 years)
  - brookst 6mo ago
    Like, they break? Or it just becomes more profitable for the data center to replace them?
    - manquer 6mo ago
      It will become more expensive to fix than replace. Also more energy intensive than newer generation to operate. MBTF is significant the older the fleet gets higher the failure rates . A typical node today is 8 GPU node today , you have to keep replacing failed GPUs by cannibalizing parts from other GPUs as nobody is selling new GPUs of that model anymore at higher frequencies. In addition to outright failure there are higher error rates in computation in graphics it tends to be flickers or screen artifacts and so on. Azure operated K-80s and P-100s for 9 and 7 years respectively but they were running at 2 GPU nodes and of course were much simpler compared to today’s HBM behomouths on 2/5 nm processor nodes . Google operates their custom ASIC TPUs for about 8-9 years . With custom inference ASICs like cerebras hitting production the cascading of training NVIDIA chips to inference to get the 5-6 year useful life is also not clear.
- vmbm 6mo ago
  The jury is still out on this. Those tax based deprecation schedules are largely a relic of traditional data centers, where workloads are fairly moderate compared to AI use cases. Additionally, power and rack space constraints can complicate things quite a bit. If next gen chips are significantly more efficient and you are currently constrained by power availability, you might pull your old servers and replace them with the newer ones regardless of how much useful life you have left.
elil17 6mo ago
I think there's more nuance to it. The real asset is the models that are being created. Imagine this world: the bubble "pops" in a couple years. The GPUs stick around for a few more years after that. At the end, we pretty much don't train new foundation models anymore - no one wants to spend the money on the hardware needed to make a real advance. People continue to refine, distill, and optimize the existing foundation models for the next century or two, just like people keep laying new track over old railway right of ways.
phreeza 6mo ago
That's definitely true for some of them, but for others it's not so clear, like the Apollo or Manhattan projects? Those of course also have lasting impact but it's more in terms of knowledge, which at least arguably we are also accruing with these data centers.
- manquer 6mo ago
  Not just knowledge. RS-25 - It was designed as HG-3 during the 60s for Saturn-V and manufactured for the Space Shuttle and refurbished for SLS and just launched last month. Vehicle assembly building - Built for Saturn-V launches been in active use and continues today . Crawler-transporters - Hanz and Franz were built in 1966 for Apollo and still used for launches. There are plenty of other examples from Apollo program of actual hardware being repurposed and used for later missions. In other mega space projects, Hubble is still doing active research, 35 years after launch, voyager is sending data close to 50 years later. It is a whole another topic whether they should be used, how NASA is funded , and this is why makes programs like SLS or the shuttle are so expensive and so forth. The point is these mega projects had a long lifetime of value, albeit with higher maintenance costs for the tech heavy ones like Apollo than say a bridge or a dam does.

by manquer 6mo ago

wr2 6mo ago

Also railways would always have alternative uses at that time - e.g. logistics in warfare. What other uses do GPU's have that are critical...? lol In addition to your points, this is why I always laugh when people do backward comparisons. What characteristics do they share in common? Very little.

jamesknelson 6mo ago
GPUs do have a use in warfare though. I mean, LLMs are basically offensive weapons disguised as software engineers. Sure, LLMs can kind of put together a prototype of some CRUD app, so long as it doesn’t need to be maintainable, understandable, innovative or secure. But they excel at persisting until some arbitrary well defined condition is met, and it appears to be the case that “you gain entry to system X” works well as one of those conditions. Given the amount of industrial infrastructure connected to the internet, and the ways in which it can break, LLMs are at some point going to be used as weapons. And it seems likely that they’ll be rather effective. FWIW, people first saw TNT as a way to dye things yellow, and then as a mining tool. So LLMs starting out as chatbots and then being seen as (bad) software engineers does put them in good company.
- bigfatkitten 6mo ago
  > GPUs do have a use in warfare though. Unclassified public cloud GPUs are completely useless when your warfighting workloads are at the SECRET level or above.
  - jhide 6mo ago
    They’re unclassified public cloud GPUs today, much the same as the massive industrial base of the United States was churning out harmless consumer widgets in 1939. Those widget makers happened to be reconfigurable into weapon makers, and so wartime production exploded from 2% to 40% of GDP in 5 years [1]. But the total industrial output of course didn’t expand by nearly that much. I think it’s maybe plausible that private compute feels similar in the next do-or-die global war. [1] https://eh.net/encyclopedia/the-american-economy-during-world-war-ii/#:~:text=from%20just%20two%20percent https://eh.net/encyclopedia/the-american-economy-during-worl...
    - bigfatkitten 6mo ago
      The United States has almost no domestic capability to produce advanced semiconductors. There is no abundance of industrial capacity cranking out GPUs that can be quickly diverted from AI companies into weapon systems. Even if private compute was at a level of maturity where you could use it for classified workloads, knowing that the infrastructure is being managed by someone in India or China, securely getting data into and out of that infrastructure is still a mostly unsolvable problem.
      - tester756 6mo ago
        wut? Intel with 18A can do it
        bigfatkitten 6mo ago
        Its low yields and tiny volumes are part of what gets the US from “no capacity” to “almost no capacity.”
        tester756 6mo ago
        yields are constantly improving on monthly basis, according to executives around 7% per month, so the capability is definitely there, but yields still needs some time
- wr2 6mo ago
  Imagine comparing something that has a useful life of 100+ years vs a thing that is worn out, much less durable, and needs replacing much more often and can become obselete from innovation within its own product category. Comical. China can continue innovating on GPUs and all this existing spend to stock up on compute is a waste. Again, comical. Moreover China has energy capacity that the US does not. Meaning all those GPU's that deliver less performance per watt? Yep going in the bin. So yeah.. carry on telling me how this is going to yield some supreme advantage lmao.
jhide 6mo ago
On the topic of warfare, wars are fought differently now. Compute will be mentioned in the same breath as total manufacturing output if a global war between superpowers erupts. In highly competitive industries this is already the case. Compute will be part of industrial mobilization in the same way that physical manufacturing or transportation capacity were mobilized in WWII. I’m not an expert on military computing but my intuition is that FLOPS are probably even more easily fungible into wartime compute than widget makers, and the US was able to go widgets->weapons on an unbelievable scale last time.
- andrewljohnson 6mo ago
  You could argue that compute was a decisive factor in World War II even (used in code breaking and designing nuclear weapons).
- AngryData 6mo ago
  There are plenty of military uses for computing, but I also find it hard to believe anything but a handful of datacenters are or could be a major factor in anything but a completely 1 sided war. They are very vulnerable targets that are easy to locate and require large amounts of power and cooling. I also just don't see the application, encryption capabilities far exceed the compute available needed for decryption and computing precision and speed with even 20 year old tech far exceeds the precision of anything you would want to control. Even with tangible banefits, say 10% more or less casualties than there would be otherwise, in an exchange with anything resembling a peer military force im not sure it matters because everybody already loses.
- 7952 6mo ago
  Is that in terms of data centres or chips on the battlefield? Surely the latter is most important. Or will war alwys have perfect connectivity.
naasking 6mo ago
> What other uses do GPU's have that are critical...? lol GPUs are essential to every kind of scientific and engineering simulation you can think of. AI-accelerated simulations are a huge deal now.
- wr2 6mo ago
  GPUs that have lives of..? Now compare that with the life a rail road. Amusing.
  - naasking 6mo ago
    Some of those railroad bridges might never have been constructed without those simulations.

deleted 6mo ago

[deleted]

gravypod 6mo ago

The side effects of spending funds on these mega projects is also something to consider. NASA spending has created a huge pile of technologies that we use day to day: https://en.wikipedia.org/wiki/NASA_spin-off_technologies https://en.wikipedia.org/wiki/NASA_spin-off_technologies.

delusional 6mo ago
> NASA spending has created a huge pile of technologies that we use day to day We're a little too early to know if that's the case here too. I do foresee a chance at a reality where AI is a dead end, but after it we have a ton of cheap GPU compute lying about, which we all rush to somehow convert into useful compute (by emulating CPU's or translating traditional algorithms into GPU oriented ones or whatever).
- PunchyHamster 6mo ago
  I think there is little chance it is a "dead end", it's here to stay but at least LLMs seem to have hit the diminishing returns curve already, despise what investors might think, and so far none of the big providers actually makes money for all that investment
  - etempleton 6mo ago
    I think for many, if LLMs and AI only improves marginally in the next 5-10 years it is effectively a dead end. The capital expenditure necessitates AI does something exponentially more valuable than what it does now. I think we are saying the same thing.i just think the pull back on AI will be dramatic unless something amazing happens very soon.
    - brookst 6mo ago
      I just don’t see it. Both professionally and personally I’m producing so much more now. Back burner projects that weren’t worth months of my time are easily worth a few hours and $20 or whatever. Why would I pull back?

rayiner 6mo ago

Great point!

Lerc 6mo ago

The shovels and labour used to make those things where not depreciated. The GPUs are the shovels, not the project. AI at any capability will retain that capbibilty forever. It only gets reduced in value by superior developments. Which are built upon technologies that the previous generation developed.

jiggawatts 6mo ago
> retain that capbibilty forever Not really. The base training data cutoff will quickly render models useless as they fail to keep up with developments. Translating some Farsi news articles about the war was hilarious, Gemini Pro got into a panic. ChatGPT either accused me of spreading fake news, or assumed this was some sort of fantasy scenario.
- m00x 6mo ago
  That's GPT4 thinking. New models use tools to look at current events or latest versions, and rely very little on weight knowledge.
  - zozbot234 6mo ago
    You can pull new information into the context via RAG, but that is expensive and only gives very shallow understanding compared to retraining.
- nl 6mo ago
  Not really. For coding I care mostly about reasoning ability which is uncorrelated with cut off
- jeremyjh 6mo ago
  Karpathy - and others - consider the pre-training knowledge as much a liability as an asset. If we could just retain the emergent reasoning and language capability without the hazy recollections the models would likely be stronger.
loandbehold 6mo ago

deleted 6mo ago

[deleted]

pembrook 6mo ago

Only half of the rail capacity that existed during the railroad boom times was still in use by the 1970s. Lots of it was never really used at all after various railroads went bankrupt. But your point still stands. That said, I'm pretty sure in a compute-hungry AI world you aren't going to retire GPUs every 6 years anymore. Even if compute capacity jumps such that current H100s only represent 10% of total compute available in 6 years, you're still running those H100s until they turn to dust. I just think it's hard to compare localized railroad infrastructure to globalized AI capacity and say one was more rational than the other on a % of GDP basis until the history actually plays out. If you compare global investment in nuclear weapons it would dwarf the manhattan project and AI thus far, and yet, 99.99999% of nuclear weapons investment is just "wasted" capacity in that it has never been "used." But the value it has created in other ways (MAD-enabled peace) has surely been profitable on net. Nobody would have predicted this at the time. Playing armchair internet pessimist about the "new thing" always makes you feel smart but is usually not a good idea since you always mis-price what you don't know about the future (which is almost everything).

brookst 6mo ago

I’m not sure tax depreciation rates are the best measure here. Those GPUs will be used for much longer than 6 years, and the returns from the businesses will be an order of magnitude longer.

mfuzzey 6mo ago
actually the physical lifetime (not financial depreciation) for AI data center GPUs is even lower (3 to 4 years)
- brookst 6mo ago
  Like, they break? Or it just becomes more profitable for the data center to replace them?
  - manquer 6mo ago
    It will become more expensive to fix than replace. Also more energy intensive than newer generation to operate. MBTF is significant the older the fleet gets higher the failure rates . A typical node today is 8 GPU node today , you have to keep replacing failed GPUs by cannibalizing parts from other GPUs as nobody is selling new GPUs of that model anymore at higher frequencies. In addition to outright failure there are higher error rates in computation in graphics it tends to be flickers or screen artifacts and so on. Azure operated K-80s and P-100s for 9 and 7 years respectively but they were running at 2 GPU nodes and of course were much simpler compared to today’s HBM behomouths on 2/5 nm processor nodes . Google operates their custom ASIC TPUs for about 8-9 years . With custom inference ASICs like cerebras hitting production the cascading of training NVIDIA chips to inference to get the 5-6 year useful life is also not clear.
vmbm 6mo ago
The jury is still out on this. Those tax based deprecation schedules are largely a relic of traditional data centers, where workloads are fairly moderate compared to AI use cases. Additionally, power and rack space constraints can complicate things quite a bit. If next gen chips are significantly more efficient and you are currently constrained by power availability, you might pull your old servers and replace them with the newer ones regardless of how much useful life you have left.

elil17 6mo ago

I think there's more nuance to it. The real asset is the models that are being created. Imagine this world: the bubble "pops" in a couple years. The GPUs stick around for a few more years after that. At the end, we pretty much don't train new foundation models anymore - no one wants to spend the money on the hardware needed to make a real advance. People continue to refine, distill, and optimize the existing foundation models for the next century or two, just like people keep laying new track over old railway right of ways.

phreeza 6mo ago

That's definitely true for some of them, but for others it's not so clear, like the Apollo or Manhattan projects? Those of course also have lasting impact but it's more in terms of knowledge, which at least arguably we are also accruing with these data centers.

manquer 6mo ago
Not just knowledge. RS-25 - It was designed as HG-3 during the 60s for Saturn-V and manufactured for the Space Shuttle and refurbished for SLS and just launched last month. Vehicle assembly building - Built for Saturn-V launches been in active use and continues today . Crawler-transporters - Hanz and Franz were built in 1966 for Apollo and still used for launches. There are plenty of other examples from Apollo program of actual hardware being repurposed and used for later missions. In other mega space projects, Hubble is still doing active research, 35 years after launch, voyager is sending data close to 50 years later. It is a whole another topic whether they should be used, how NASA is funded , and this is why makes programs like SLS or the shuttle are so expensive and so forth. The point is these mega projects had a long lifetime of value, albeit with higher maintenance costs for the tech heavy ones like Apollo than say a bridge or a dam does.