6 ms·
One aspect Paul Kedrosky mentioned recently is the concept of „duration mismatch“. The price per token goes down over time (either because the AI vendor reduces
by dgellow 5mo ago
One aspect Paul Kedrosky mentioned recently is the concept of „duration mismatch“. The price per token goes down over time (either because the AI vendor reduces due to competition pressure, or because customers are now incentivized to use older cheaper models). But datacenters are financed through debt, with the assumption their revenue increases over time. Quoting him: „[AI vendors are] paying for a fixed cost with a depreciating commodity“[0].
So you have on one end the token revenue trending down, on the other end the training cost going up for the next frontier models, and you need to pay back your 10y debt.
0: https://youtu.be/wGZboZcSGDY?is=64GuKyqBh_4aSjTE https://youtu.be/wGZboZcSGDY?is=64GuKyqBh_4aSjTE
- bijowo1676 5mo agodo GPU chips really depreciate physically? There are no moving parts, I dont think memory chips or GPU chips deteriorate naturally. I think its only accounting depreciation. I have been using my laptop for a decade, what is stopping datacenters from using the purchased GPU chips for a decade?
- dgellow 5mo agoGPU do depreciate indeed, but here the depreciating commodity is the token, not the hardware. You sell cheaper token with the same hardware
- xyzsparetimexyz 5mo agoWhen everything is said and done it'll be datacenters in American competing with ones in China that have several times lower electricity prices. Token prices will drop to a level that will be unprofitable for American data centers and they will need to close. Thats the main issue here.
- dgellow 5mo agoYep, that’s one source of pressure, for sure
- bigfishrunning 5mo agoYour laptop doesn't have a 100% duty cycle. If you ran it like a data center it would indeed wear out much faster.
- whateverboat 5mo agoToday's data center GPUs are essentially overclocked, and so at limit of how much the chip materials can physically handle, and therefore degrade over time. For example, GH200s operate at 1W/superchip but the actual safe power is somewhere around 650W which will allow them to function for a decade or more. But that leads to around 15% slowdown and that is unacceptable in today's competition. So current GPUs are destined to be depreciating assets. In future, we might have fixed cost GPUs but not today.
- missedthecue 5mo agoI would presume the reason they are overclocked is because they are trying to make up for the shortage. In time, the shortage of computing components will be remedied, and tokens produced at lower power pulls will be cheaper.
- bijowo1676 5mo agoi think its reasonable to give up 15% of speed for a decade more lifetime. This depreciation change alters economics of GPU
- nothercastle 5mo agoThat extra decade might provide almost no revenue. The long tail isn’t profitable
- Aurornis 5mo agoThere are data centers that use and rent out 10 year old server GPUs. They can't run larger modern models. They can't run smaller models as fast as newer servers. So their remaining market is applications where customers are okay with older, smaller models and slower performance. They have to price the service lower than competitors due to the lower performance. The older GPUs are less efficient so it costs them more to keep them running. They're paid off, but they're taking up valuable power, space, and cooling in a data center. Eventually there is a tipping point where it's better to replace that space and power budget with something new that has more demand. The parts are sold off on the open market. There's an equilibrium demand for the parts from other data centers keeping older servers running and from hobby people who are okay with a jet engine sounding toaster of a GPU running in their home.
- jmalicki 5mo agoAs long as the demand for GPUs keeps increasing, there are more data centers being built to house them. When you have waitlists for many many months for Blackwell GPUs, keeping the old ones around as long as customers are willing to pay for them is great. If I as a customer have a use case for a machine learning model I developed awhile ago, so an insect identification model, I had an ML researcher/eng develop it back in 2019, and it runs fine on a 2018-era T4 GPU (NVidia 2080 era), why mess with it?
- HumanOstrich 5mo agoWe aren't talking about insect identification models from 2019.
- sandworm101 5mo agoYes, even if the hardware is untouched. As technology advances, the power cost per compute cycle goes down. A gpu using old tech costs progressively more to operate compared to the newer models. So its value goes down over time = depreciation. As for duty cycles, the chips are perfectly happy at 100% operation. Cooling and power componants fail, not the chips. But it costs manpower to repair such things and manpower is inconveniant these days. A gpu with any sort of fault just gets dumped.
- munk-a 5mo agoIn addition to the physical depreciations other comments mentioned I'd also mention that old chips will settle into a low price and then actually go up on a per unit basis if you're trying to buy a significant amount of them. With a limitation on fabrication facilities continuing to pump out older cards is an opportunity cost to the manufacturers that would prefer to be producing newer cards. If you were in a place where you suddenly wanted to buy 10,000 3080s, as an example, I'm not certain if the market could actually fulfill that demand and no one with the ability to increase the available supply to meet that demand actually wants to do so. Chips do wear out and need to be replaced (entropy do be like that and durability is not a primary concern for chip design) so you'll need to refresh your stock and, even if you don't need cutting edge models, the price of all chips at scale will go up over time. It may feel unintuitive since, when the PS3 was released PS1s were extremely cheap - but if you're struggling to understand this effect from your experiences in the consumer market you're actually looking at the price factor that starts making antiques increase in value since at a certain point they become scarce goods. The market price for an NES is higher today than it was in 2003 because the price had already bottomed out from demand from the general consumer market but the demand remaining (speedrunners and the like) is now fixed or growing while the supply is inevitably shrinking.
- foobarian 5mo ago> There are no moving parts, I dont think memory chips or GPU chips deteriorate naturally I believe they do, but I too would love to know more details because there are several ways this can happen. Electromigration, package failures, VRAM failures, dielectric breakdown... Hopefully there will be studies soon similar to that old Google paper on HDD failures!
- hgoel 5mo agoCurrently it's a pretty big ask to look at the several hundred billion transistors and the interconnects between them to find what broke. Though, those capabilities are maybe just a few years out, funnily it's taking AI to make it potentially doable.
- threetonesun 5mo agoI assumed the issue was similar to crypto mining, where given finite amounts of space and power it makes sense to always be running the latest and most powerful GPUs instead of keeping older hardware running. There's definitely a secondary market for these GPUs as well.
- vb-8448 5mo agoI used to work in datacenters, during spinning disk era we had technicians from vendors basically every couple of days to replace some broken part. When the massive switch to ssd happened instead of having them every couple of days it was 3 or 4 times per month. Despite no moving parts things broke anyway and, even if it doesn't break, the vendor can make you change the technology just by playing with maintenance cost of the older one, limiting or removing spare parts from the market.
- _fizz_buzz_ 5mo agoMy understanding is that a lot of AI data centers are still heavily relying on spinning HDDs, which is why seagate, western digital are selling more HDDs than ever before.
- simonw 5mo agoHuh, TIL. Here's the Seagate financials for Q3FY26: https://s24.q4cdn.com/101481333/files/doc_financials/2026/q3/STX-FQ3-26-Supplemental.pdf https://s24.q4cdn.com/101481333/files/doc_financials/2026/q3... "Hard Drive exabyte shipments of 199EB, up 39% YoY, with ~90% shipped to data center customers" "Data center revenue of $2.5B, up 55% YoY, driven by strengthening cloud and enterprise demand" And an article: https://www.seagate.com/stories/articles/the-ai-era-doesnt-run-on-compute-alone-it-runs-on-storage-economics/ https://www.seagate.com/stories/articles/the-ai-era-doesnt-r...
- hylaride 5mo agoSpinning drives are still the "best" for data density and if the IO is sequential (which wouldn't surprise me with AI training workloads), the performance delta may not be that bad vs SSDs. As always, it depends on use case. I know that a lot of cloud storage has tiered models, where the "expensive, but faster" tiers are SSDs, but then the slower cheaper tiers are HDDs, and the "cold storage" can be HDDs that are turned off all the way to tiers like AWS's S3 "deep archive glacier" tier being tape drives.
- manyatoms 5mo agothe hardware itself is still useful, but random failures happen every so often, so if you're trying to run a fixed sized fleet then your fleet shrinks when you can't get spares any more
- numpad0 5mo agoChips do deteriorate and fail naturally at datacenter scale or in timescales of decades, though not exactly like on financial reports. Leak current increases or electro-migrations occur at junctions or whatever those words mean. And yeah, it does feel like GPUs will start losing values slower going forward with Moore's Law being dead for a while. It used to be that 3-5 years old GPUs were more useful as space heaters than GPUs, but that's much less of the case today.
- malfist 5mo agoThey do degrade physically, but the bigger thing is they stop being competitive quickly. Each year or so we see doubling of GPU speeds for the same amount of power. If you build a 100MW data center with GPU compute and three years laster a new data center opens with the same cost for GPUs and same electricity cost you do, but can do twice as much compute, you quickly lose business unless the market is just so constrained customers can't afford to be picky. But the moment there's slack in the market you'll see major migrations off of providers that have the same cost but half, or quarter of the same performance. So when you see someone talking about GPUs fully deprecating in value in 1-3 years this is what they're talking about. Right now it's not a big deal because there's no slack in the market. But once there is, the bottom will drop out.
- tardedmeme 5mo agoGradually, and especially when hot. Modern chips are pretty close to the physical limits of how small they can be made, and that means atomic/chemical effects like electromigration are accounted for and determine the lifetime. Every extra 10 degrees Celsius of temperature doubles the speed of chemical reactions. When they stray too close to the line ... you get Intel's 13/14th gen chips that wear out after 1-2 years instead of 10-20 years. Intel calls it "Vmin drift" because that doesn't sound scary, but the actual point is that various wear-out mechanisms push the chip outside of its design envelope - increasing the voltage or lowering the clock speed may get it to run for a while longer, but you're living on borrowed time as the various circuits just stop working right and you get unpredictable instruction mis-execution: https://fgiesen.wordpress.com/2025/05/21/oodle-2-9-14-and-intel-13th-14th-gen-cpus/ https://fgiesen.wordpress.com/2025/05/21/oodle-2-9-14-and-in...
- bijowo1676 5mo agosounds like planned depreciation on Intel's part, they definitely do not design server grade chips for longevity since that would harm their own revenues
- HDBaseT 5mo agoIt was not planned depreciation, as many chips were failing even before 2 years and this impacted not only PC Builders and Gamers, but also some server infra providers too. This was simply poor design, it took Intel ages to really figure out what went wrong and "resolve" it. It cost them far more than it made.
- bgnn 5mo agoChips age and fail with age. You can check hot-carrier injection, bias-temperature instability and electromigration as they are the main aging mechanisms. All if these are a linear function of time but exponentieal of temperature. 90-100C these chips are running at are really tough, so they are likely to fail at couple of percent to 10% range in 2-3 years depending on the margins they have in the design. The solder joints are notorious to fail at a high rate too.
- consp 5mo agoIf those don't go the caps and coils will eventually.
- lazide 5mo agoCaps also have a rapid aging with temp.
- chadgpt3 5mo agothose are easy and cheap to replace
- jetbalsa 5mo agoDepends, the SMD caps spread across the board the tiny ones do start to fail and go out of spec over time. they are a right pain to replace and hard to spot one that has gone out of spec to cause the chip to start crashing.
- mattalex 5mo agoNothing is stopping them, it's just not worth it: Have a look at e.g. vast.ai's pricing (https://vast.ai/pricing https://vast.ai/pricing). The V100 (2017 -> 9 years old) can be rented from $0.02 to $0.37/h (right now I can find a V100 with a Xeon Gold 6140 and 48GB RAM for $0.165/h). Let's assume the guy you rent it to pins it at its 250W TDP and let's ignore the running costs of CPU/RAM/etc... Then you draw 1/4 kwh for that compute hour. The industrial electricity prices in the US vary between 7.5 and 25 ct per kwh (depending on state, time of day, etc...), so at 100% efficiency, assuming nothing ever breaks, and the CPU consumes 0W you earn about 14ct/h. And remember: V100s hours are sometimes sold at 1/10th the price. If I pick average conditions you need to start thinking of whether it is worth it to rent them out: Usually it isn't unless you have them anyways and just sell idle capacity. It's barely worth it to run them in a pure "is it profitable" sense, if we also account for the opportunity cost of taking up a slot in your datacenter it seizes to be worth it really quickly.
- ozim 5mo agoTransistors do wear out. Not going to elaborate as it is easy to ask GPT
- fooker 5mo agoWhen it was profitable to mine crypto with GPUs people used to sell these miner GPUs on the used market after about two years. These were about half of the cost of an used GPU just used for gaming. By that pricr, I'd say a GPU kept busy has twice as high a chance of failure after two years of use. Not great, not terrible.
- missedthecue 5mo ago"So you have on one end the token revenue trending down, on the other end the training cost going up for the next frontier models, and you need to pay back your 10y debt." Not necessarily, the bond holders could simply take a massive hair cut and lose shitloads of money. On the topic of bubbles and exuberance, Jeff Bezos made the salient point that there was a massive over-invested biotech boom in the 1990s and tons of sophisticated investors ended up losing lots of money. But humanity still kept the medical advancements made by the boom. Stocks going down didn't un-research drugs, and it won't un-research new GPUs or un-build datacenters.
- biztos 5mo agoIn order to not un-build the data centers, they at least have to make more than it costs to operate them, and also not have some attractive liquidation value (the land, maybe). I could imagine something like “inference is done at home or in China, that’s the price to beat” and it’s not worth keeping all those GPUs cool out in Nevada.
- missedthecue 5mo agoBut the parent comment was that one of the bigger costs in these data centers was the interest expense on the borrowed money. A restructuring removes or heavily reduces that amount. The fiber laid during the dotcom bubble never paid back the investors or lenders, but it's still profitably connecting customers all these years later.
- nothercastle 5mo agoIt’s true once built the data center can operate right up to a financed data center value of zero. The investors will loose money but the costs of AI will go down as they do
- lazide 5mo agoYup, that is the real economic benefit of bankruptcy - a reset.
- 5mo ago
- geysersam 5mo agoCurrent AI datacenter/model development investment rate is roughly 1T/year. That's a lot. But the US economy is 33T/year. So the investment pays back (roughly) over ten years if, each year, the AI investments increase overall productivity by 0.6%, assuming the AI companies can capture half of the value of that productivity gain. > „[AI vendors are] paying for a fixed cost with a depreciating commodity“ That's just a confusing way to say you don't think future models will be worth the development costs. Because if future models are significantly better, why would the price of tokens to access those models deprecate?
- jiggawatts 5mo agoThe $1T number seems more promises than reality, which is closer to the $300B to $500B level. Still a big number, but between a third and a half of the value used in the popular media.
- flextheruler 5mo agoThese are similar numbers to the dotcom bubble. With GDP growth and the percentage of productivity AI contributes staying the same in this scenario this requires regular gains in revenue or growth. If things just stumble, like with most datacenters going unbuilt the bubble will pop.
- timacles 5mo agoI'm surprised people think LLMs, a thing which mainly excels at advertising, spam and writing code is going to generate that much economic activity.
- ashdksnndck 5mo agoCompanies whose main core competency is writing code were already making up a big chunk of the economy before AI. Also, less wealthy companies were constrained in their use of software by the inability to afford the salaries of talented programmers (and ripoff practices from software consulting companies who in theory could help). Lowering the cost of building software systems ought to unblock a good amount of economic activity as the technology diffuses.
- bethekidyouwant 5mo agoUsing a shittier model is just more work for the user, I’m not sure why anyone does it, unless they’re playing with it like a toy.
- SoMomentary 5mo agoLocal privacy respecting inference can be worth it. I use a local model to log everything I do all week to automate my timesheet. I also have it do a bunch of other data tasks. I won't say that larger SOTA models wouldn't do these tasks better than a local model but PII is a concern and my employer wouldn't approve of me just setting tokens on fire everyday to do what I could do myself.
- jurgenburgen 5mo ago> I use a local model to log everything I do all week to automate my timesheet. Isn’t that just more work than logging it yourself?
- SoMomentary 5mo agoNot at all! My company has 100s of clients and we track time in 6 minute increments. I feed in my browser history, terminal logs, session scripts, calendar, git commits, etc etc into it and voila it produces a highly accurate timesheet in no time flat. Automating it has been way better for me than the alternative of breaking my flow whenever I'm switching tasks to chart my time, or logging all my hours for the week in one sitting. Different strokes for different folks I suppose.
- jurgenburgen 5mo ago> My company has 100s of clients and we track time in 6 minute increments. That’s an insane level of detail and I can understand why it makes sense to use an LLM to remove that busywork, otherwise you’d be spending half of your time on this bureaucracy. For the company I would imagine it could be more cost efficient to stop being so onerous instead of burning tokens.
- try-working 5mo agoIf you have a good model router, you can route to older, cheaper models that run on older hardware, for simpler tasks. That helps labs extend the economic life of their hardware investments. They will likely fight it at first though as they see it as reducing ASP. This is why I'm building role-model, a routing protocol and a router runtime: https://role-model.dev/ https://role-model.dev/
- jurgenburgen 5mo agoRunning cheaper models on newer hardware is always going to beat running them on older hardware.
- bandrami 5mo agoThe other part of that is that while price per token may be going down, tokens per task is going up
- no-name-here 5mo agoFor ~equivalent tasks/results, or because we’re expecting more or better from tasks? The real measure should be cost per ~equivalent task result, not cost per token nor tokens per task.
- bandrami 5mo agoFor better performance of ~equivalent tasks. That's what all the harness tooling people are using does: (often) increasing output quality by significantly increasing token counts.
- Forgeties79 5mo agoI really wouldn’t be surprised if we saw some of these data centers scrapped in the next few years
- treis 5mo agoRelative to the current usage demand for tokens is effectively unlimited. If the price of tokens go down people will send more tokens to compensate. We are very very far away from a cost per token where people run out of things they want to send through an LLM.
- dgellow 5mo agoYou’re describing the issue. The problem isn’t that datacenters are under utilized, it’s that they are used to generate something that has a value going down over time, while being financed through debt with the assumption their revenue grow over time. That’s where you have a mismatch. To simplify let’s assume a given datacenter can generate a fixed number of token per second (in reality that depends on the model tokenizer, and other model specific factors). And let’s say it is used at full capacity, split between inference and training. If people are mostly using the frontier models the datacenter might be profitable, with enough margin to cover training of future frontier model, operating costs, and repay the debt used for construction. However if the token price goes down and the demand doesn’t change you now risk to run the datacenter at loss, with risk to not be able to repay your debt. If price reduce and demands increase, you can reduce the part dedicated to training, but eventually run the datacenter at full capacity to generate tokens that are becoming less and less valuable over time
- treis 5mo agoThe value is not going down. The cost is.
- alfalfasprout 5mo agoRight. Which means tokens are actually being priced well under cost once you factor in all this datacenter/GPU capex. Also worth noting the datacenters are not purely for training. They're for inference too.