10 ms·
Nobody knows what a used GPU cluster is worth
- grim_io 3mo agoNothing, because those "GPU's" are special proprietary hardware and are not what most people are capable of plugging in into anything.
- cgyvbunji 3mo agoPeople are selling adapter boards to plug some data center GPUs into a regular pcie slot.
- tyfon 3mo agoI half expect Nvidia to have buyback contacts like Ferrari with the larger customers to prevent a price crash when they all upgrade and to keep them scarce. I hope not though, perhaps I can pick up a H100 in a few years if they get sold on the open market.
- someguyiguess 3mo agoThat should be illegal. Sounds like a very fraudulent business tactic.
- rcxdude 3mo agoFraudulent, I wouldn't say so. Anti-consumer or anti-competitive? Sounds like.
- 2III7 3mo ago[flagged]
- HPsquared 3mo agoGood old "win-win-lose"
- throw1234567891 3mo agoWould you call a trade-in a fraudulent tactic?
- InsideOutSanta 3mo agoTrade-ins are voluntary.
- deleted 3mo ago[deleted]
- efficax 3mo agoso are buybacks. you choose to sign the contract. there's no way they didn't have an escape clause, although likely it meant not using the cloud provider anymore
- InsideOutSanta 3mo ago> so are buybacks. you choose to sign the contract Right, so they're not voluntary.
- anomaly_ 3mo agoLiterally what? Do you understand what is being proposed?
- InsideOutSanta 3mo agoYes. "You choose to sign the contract" is not a reasonable argument, for two reasons: 1. There is one supplier, so you have no choice. 2. Even if you had a choice to sign the contract, this still means that it's not the same as a trade-in, because trade-ins are always voluntary, but once you have signed the contract, a right of first refusal is not. In general, the "you chose to sign the contract" argument is a poor justification for bad contracts. If the contract is bad, it is bad regardless of whether you chose to sign it.
- cyanydeez 3mo agoThe Grift Economy places all legalities on the marks and their inability to form legal fights.
- rbanffy 3mo agoIt’s common for car companies when they enter a new market. It removes uncertainty from the second hand market. By doing that, you know upfront what the value of your used hardware will be at the time you decommission it. It removes a lot of the risk for buyers in a volatile market.
- nl 3mo agoThe hyperscalers signing these contracts have decent legal departments. Think about Oracle for example - I'm pretty sure they know every trick there is about beneficial contract drafting. I don't think they need some special protection against this kind of contract.
- KaiserPro 3mo agoIts restrictive, but not that different from other types of deals. For example, when I worked at a VFX software company, we were exclusively with one hardware partner. This unlocked something like a 50% discount across all our infra needs (they were big enough to provide switches, racks, servers, storage) another company it unlocked a 75% discount.
- lelanthran 3mo ago> That should be illegal. Sounds like a very fraudulent business tactic. a) It's not fraudulent, and b) Nvidia has already signed on to buyback any unused capacity from the DCs it is selling to.
- zenbane 3mo agoit's also cruel. thankfully orgs like www.siliconstrays.com
- echelon 3mo agoWouldn't that be crazy - a hobbyist market for H100s? Maybe someone could start a business buying up and rehousing these.
- rtkwe 3mo agoThey're pretty specific to the datacenter use case with no outputs and they need to be cooled externally, principally through the very loud high speed fans used in data centers. I suppose you could strap a fan to one and put it in a normal case or maybe make a dedicated after market cooler (like the water blocks made for water cooling cases).
- bayindirh 3mo agoOr you can secure yourself a DLC one and try to feed it with the correct regime (liquid composition, temperature and flow rate). I'd say good luck. These things get hot and are fussy about their requirements.
- jtolmar 3mo agoI think there's a market for a home AI server that can run an LLM or video gen model behind a web frontend. Not literally a raspberry pi strapped to an H100, but something with lopsided enough specs that people joke it is. And precisely because it's such a huge headache to do yourself, I think a small company could make a nice business wrapping up used datacenter cards in that sort of server.
- everforward 3mo agoI'm doubtful they can, based on current inference prices. An H100 draws about 50W while idling, which is ~$8/month at average US electricity prices. They also draw ~400W while active. The electricity prices are relevant because if you paid $0 for your H100 and didn't use it a single time, you could buy millions of tokens in inference just on the electricity it draws while idling. If you can't keep that thing saturated through the night, you're probably underwater overnight. Likewise, it's too small to run even the frontier open source models so you need to be able to live with worse models. Max power matters because you aren't going to run many of those H100s before you blow breakers in most houses. Newer houses in the US are 15A service to non-kitchen breakers, so 1650W (that might be peak rather than continuous, not sure). If you're plugging that into an existing run, you could maybe run 2 before you start blowing breakers? You can't just plug 4 H100s into the wall in a normal house. Maybe I'm wrong, though. I'd be curious, it'd be neat to run my own inference for something more than what'll run on a 3080.
- rickypp 3mo agoI worked at an org that had a substantial on-prem GPU datacenter. We transitioned to <Big Cloud Provider> with a substantial negotiated discount rate, with part of the contract being we would sell them all of our hardware and not purchase any more.
- throwaway85825 3mo agoHow does this work? Is the OEM giving the cloud provider a big discount or is the cloud provider giving you a teaser rate to lock up your business.
- wmf 3mo agoThe list price of cloud is 10x higher than the price of hardware itself so that leaves room for some discounting.
- to11mtm 3mo agoYeah.... Where folks (often) get lazy is the resulting math over what the real bean-counters care about (but are too lazy to check often). In a past life, I worked on costing models for a Cable/Fiber contract house, to help the company decide 'what was profitable to keep in house' versus 'what do we subcontract' (sometimes that could even mean we just 'rented' a machine and had a qualified operator using it, based on that employee's hourly rate and expected L2R for taxes... so many spreadsheets...) And from from my 'I don't know all the factors for this but I've seen how people screw up the big ones' view (and frankly, I'm guessing a lot of us have seen and dealt with the same category of 'bad math' around outsourcing IT work...) An on-prem data center means: - You need to account for electricity costs - i.e. CA vs midwest electric rates. - cooling and power backup capability - Smaller factor but real - personnel cost - e.x. there's probably cases where a smaller org could be better off with 'on-site' server admins that have other roles based on local wages. Kinda case specfic but it's a case. - whatever the 'space' holding the stuff costs - Sardonic take :Hey, let's have another unused meeting room instead! (e.x. In the case of on-prem shops that simply fled to AWS in their migration from VMware) - the cost of licensing whatever is running - In defense of this, In one of my earliest IT lives, AWS handling the Oracle licensing for a DB was a *huge* win as far as making it as easy as possible to ensure whatever was going on we couldn't have the Oracle licensing folks 'ding' us on whatever infraction occurred between reviews (that could not be understood by the majority of the company, often including the accused. I was never guilty but I saw it happen to others.) - OTOH I know lots of folks who just want to be lazy about what they have to document. Still, IMO a lot of orgs don't do the right math around these decisions, or just buy into the 'Well trends can change' as though they can decide as an org they need to suddenly triple capacity in a month and it would be able to organically happen in the first place. Frankly, the orgs that 'might' need that either have their arch set up where they are in cloud, or they are onprem but can scale to cloud if needed in interim.
- nylonstrung 3mo agoThere's going to be a golden age of GPGPU compute in the next few years once A100/H100 are fully obsolete for running frontier models efficiently and the price plummets It will be perfect for stuff like GPU-accelerated query engines, "classical ML" and every other CPU-based workload that could conceivably be offloaded to GPU
- throwaway894345 3mo agoGPGPU? General Purpose GPU? If embarrassingly parallel CPU algorithms weren't offloaded to the GPU previously, why would the A100/H100 price drop make a difference? We had cheap GPU in the past and we still left plenty of performance on the table with CPU programs because they were easier to build. Is the idea that previously maintaining GPU programs was expensive whereas now AI makes it cheap? If so, I could buy that line of reasoning. Maybe relatedly, I expect (hope) the hardware manufacturers will ramp up supply in the meanwhile which would also put downward pressure on GPUs. Right now though this hardware crunch is making me sad, not even because of GPUs but also because of general memory / disk.
- christina97 3mo agoGPGPU programming has become significantly easier now and the payoff is bigger (better hardware), due to the immense investment in this due to ML/AI.
- NuclearPM 3mo agoWhat are the best tools for this?
- girvo 3mo agoWe've not had cheap GPUs with this much VRAM before, though. Might be an interesting change, though I also doubt it personally.
- nl 3mo agoAs noted in my other comment you can get obsolete GPUs (P100s, BC250s) with lots of RAM on AliExpress now. It hasn't proven revolutionary.
- deleted 3mo ago[deleted]
- moffkalast 3mo agoThe only reason a datacenter would ditch their H100 is if it becomes uneconomical to run them, with newer silicon providing much more power efficiency. When that happens they'll look like a used V100 looks today: horribly inefficient, lacking modern data types and engines, requiring screaming server fans with weird adapters to not melt, way beyond end of life in terms of cuda support. Almost completely damn useless unless you really have no other alternative.
- mertleee 3mo ago[dead]
- imtringued 3mo agoHonestly, the primary reason you would ditch a H100 is because the ML models have moved onto a different data type, e.g. nobody uses fp8 anymore and everyone jumped onto fp4 or some custom block float thing.
- sidewndr46 3mo agoisn't Ferrari the brand that requires any purchaser to be an existing owner? I could see NVIDIA going for something like that.
- metadat 3mo agoInvite-only is only for special edition hypercars and halo models. Standard models can be purchased by anyone with the funds and desire.
- m463 3mo agotesla did that for the last 250 model s cars. https://electrek.co/2026/04/11/tesla-signature-series-model-s-x-final-run-pricing/ https://electrek.co/2026/04/11/tesla-signature-series-model-...
- driverdan 3mo agoPorsche is doing that for RS cars.
- olyjohn 3mo agoYeah and now they have lost 98% of their profits in the last year. How's that catering to the ultra-wealthy working out for them? They used to build attainable cars that were nearly as cheap as a Corvette. Now they're double the price, and most certainly aren't double the performance.
- rgmerk 3mo agoFrom what I’ve read the only part of their business that is holding up is the 911 and particularly the high-end variants. It’s everything else that’s struggling.
- semiquaver 3mo agoMaybe this article has something useful to say but the painfully LLM-generated prose is too distracting to make it evident.
- mrhottakes 3mo agoDid an AI make all the text and kerning so ugly? Why do web sites look like this now?
- aqfamnzc 3mo agoIsn't this just Substack?
- ijidak 3mo agoHow do you know? I feel that it's just poorly edited. In my experience, AI is easier to read than this was.
- Game_Ender 3mo agoSome sentences feel pretty AI like: > These are not catastrophic events. They are the steady state. > There is no GPU futures market, no standardized residual value curve, and no way to lock in a forward rental rate. The premium is is the price of underwriting in the dark. The headings are also AI like, a lot of essays before usually did not have titled sections but now they do and they all feel like these. In addition the diagrams themselves look pretty AI generated.
- owebmaster 3mo agoSoon we will have people asking if these comments are AI-generated.
- acd 3mo agoUsed GPU hosting for pension funds. You go to ebay search for a used GPU. You get a price. Neither is used servers a new thing or used routers. There are established used server companies. https://en.wikipedia.org/wiki/The_Emperor%27s_New_Clothes https://en.wikipedia.org/wiki/The_Emperor%27s_New_Clothes
- deleted 3mo ago[deleted]
- BenFranklin100 3mo agoWould it make economic sense to strip it down and sell for parts? That’s how it’s done now for older data centers, where the obsolete equipment is sent off to China, stripped down for parts, and sold on the secondary market. I’ve picked up several older but still useful RAID hardware cards off eBay this way. I’m mainly interested in getting some DDR4/5 and RTX5090s on the cheap :).
- latchkey 3mo agoI like Meg a lot a human, but Meg is all doom and gloom. Every single post she makes is about how GPUs fail [0] and now she's onto how financing is a big thing just waiting to crash and "nobody knows what a used GPU cluster is worth"... Actually, we do, people offer them to me all the time. A used box of MI300x is $257k. "There is no GPU futures market"... actually there are a few of them that people have pitched to me. This article is a lot of words from someone who isn't actually buying or deploying compute. My point is... take it all with a grain of salt. [0] https://x.com/meggmcnulty/status/2040851080066859386 https://x.com/meggmcnulty/status/2040851080066859386
- cgyvbunji 3mo agoSomehow at any given moment every possible topic to discuss on HN belongs to either the set of "it's amazing and nobody can say anything bad about it" or the set of "this thing sucks and nobody is allowed to say anything positive about it" and I never have ANY idea which one any given topic will be in on any given day.
- fragmede 3mo agolatchkey is being restrained. He runs a data center filled with AMD GPUs. He's got a lot more insight to the business of it than the post does.
- cgyvbunji 3mo agoThat's great, their comment should get more attention then. I was complaining that it's obviously incorrect that nobody knows what used GPUs are worth, not about latchkey. I have NO idea why anyone is upvoting a post titled "nobody knows what a used GPU cluster is worth", that is a WILD claim.
- latchkey 3mo agoshe makes a lot of wild claims for clicks and gets to the front page with them... sigh.
- aslkalska 3mo agoso you're telling me you can't use 1 or 2% percent of 5 billions dollars to rebuild a team that runs GPU clusters for like a couple of years ?! and the guy that borrowed billions from these banks would want to mess up that relationship for what? I mean he is a stupid narcissist but not to that degree. This whole article makes no sense to me.
- cgyvbunji 3mo agoIf nobody knows what a used GPU cluster is worth it means nobody is doing anything important with GPUs - time to short everything. Do you believe it?
- pier25 3mo agoso when will xAI default on its debt?
- wmf 3mo agoNever? They're currently renting out those GPUs at a profit.
- owebmaster 3mo agoTheir debt expects a much (much!) bigger margin tho
- fancyfredbot 3mo agoThe relatively slow depreciation of GPU value is an artifact of supply constraints. If you run fp4 inference and could choose freely between Hopper and a Rubin, the performance per watt would make the Hopper unattractive even if you paid zero for the hardware and only for the power. You can't get the Rubin, or even the Blackwell, so you will pay for the H100 but this won't last if fabs ramp up capacity.
- tracker1 3mo agoNot to mention, physical limits to lithography are slowing down significantly... so tech will continue to evolve more slowly... it'll never be the jump from 1080-1990 again, for example, even though 1990-2000 was pretty close, 2000-2010 much slower and since 2010 slower still. What's as or more weird is how much hardware is backordered, and how much live hardware is allocated, but waiting on facilities for operation. And how many facilities are years behind at this point already... all on various credit and dept swaps between all the involved companies... it's not just a balloon, it's a house of cards balanced on a balloon.
- cyanydeez 3mo agosupposedly, the next step is into fiber & optics. but I generally agree, people put a lot of faith in the exponential leaps vs the exponential space. You tell them we're not living on mars any time soon and they'll bring up christopher columbus.
- 3mo ago
- 1-6 3mo agoThe GPU cluster's RAM is probably worth more than the flops these days.
- trollbridge 3mo agoI’ve wondered when people will start finding a way to disconnect the HBM DRAM from e.g. old V100s.
- 1-6 3mo agoHBMs are tightly connected to the GPU die via a silicon interposer. It's basically soldered onto the processor and impossible to separate.
- timmmmmmay 3mo ago[flagged]
- mlyle 3mo agoThat still doesn't tell you what GPUs will be worth in 5 years, because it depends upon what inference demand is like and what the alternatives are. What will an hour of NVL72 be worth in 2029? (And other things, that we know partially but not fully-- like what failure rate for current generation parts will be under this loading). So we have big uncertainties about the revenue, moderate uncertainty about the proportion of the asset that will survive, and some uncertainty about what operating costs will be. It's difficult to turn this into a residual value. Finally, the whole "operating the big facility" thing is not likely to be plug-and-play for a new technical team following a default. How much outage/disruption ensues?
- munchler 3mo agoNobody knows what anything is going to be worth in 5 years.
- throwaway85825 3mo agoIf that was true the insurance and loan industries wouldn't exist.
- mlyle 3mo agoStrictly true, but finance being able to predict this pretty closely is how the entire modern economy works.
- timmmmmmay 3mo agoperhaps! but you can look at what a five year old GPU (Nvidia A100 80gb) sells for today. it's not cheap! I wish it was!
- mlyle 3mo agoYes, because we're currently in a GPU and RAM supply crunch, things have held value well compared to historical averages. We are increasing production of GPUs and RAM a lot.
- DiabloD3 3mo agoGPU clusters have, largely, no actual value. If anything, you might have to pay to have them disposed of, they don't really have any meaningful used eBay market outside of the randos that want to do high end extreme local inference in their basement. Also, as for RAMmageddon, the inference SBCs that all of the AI bros bought don't have DIMMs, they're not even the right chip: its all GDDR and LPDDR. The only DDR DIMMs being consumed are for regular non-inference machines that help run the business and service infrastructure behind the scenes.
- Zigurd 3mo agoTime to dig up the spreadsheet models of what dark fiber is worth.
- busymom0 3mo ago> Silent data corruption (SDC) is the most expensive, where a faulty GPU produces wrong answers without crashing anything, which means a multi-day training run can complete normally and the resulting model weights are quietly poisoned. Anyone know how these get caught ultimately?
- wmf 3mo agoUsually silent data corruption isn't caught. If a file fails to open you might realize it's been corrupted.
- Joel_Mckay 3mo agoGiven the price for a rack of bc-250 after the crypto hype cycle, the expected value of the hardware will be around 5% to 10% of the original retail price. Without other market influences, that is a >90% expected discount when the over-provisioned market must inevitably self-correct. If the Market follows what Samsung/SK Hynix did to the South Korean exchange this week, than the "AI" bubble will hit harder than the dot com crash. I like the Shrek Movie correlation theory, as they always happen just before Debt-backed investors get hit hard... And the new film is due out in 2027. =3
- owebmaster 3mo ago> If the Market follows what Samsung/SK Hynix did to the South Korean exchange this week, than the "AI" bubble will hit harder than the dot com crash Can you tell us more about this? Or some link
- Joel_Mckay 3mo agoDon't worry about it... I am more focused on the bizarre Shrek film timing phenomena, and pondering whether the pattern will hold again. =3 Patrick Boyle gives a summary of the situation, but not the underlying Shrek issue: https://www.youtube.com/watch?v=nJtL9MBVj48 https://www.youtube.com/watch?v=nJtL9MBVj48
- HAL3000 3mo agoSamsung and SK Hynix together account for around 60% of the Kospi's (SK stock exchange) market capitalization. Over the past few weeks Kospi index has tumbled 25% since its June peak, resulting in a $1 trillion wipeout and its chipmaker duo have both lost at least 30% of their value. There have been days of near 10% plunges followed by sharp rebounds driven entirely by shifting confidence in whether AI spending is sustainable.
- totetsu 3mo agoI am hearing this point come up a lot. That in previous bubbles, there at least was useful infrastructure left in the aftermath. In the case of an AI bubble there is a lot of shortlived chips, and high maintenance infrastructure being invested into.
- narrator 3mo agoOne data point: 512GB Mac Studios are selling on ebay for double what they were selling for new at the beginning of the year.
- wmf 3mo agoAnything with RAM in it is an appreciating asset.
- mtmail 3mo agohttps://news.ycombinator.com/item?id=49005798 https://news.ycombinator.com/item?id=49005798 "I have an M3 ultra mac studio with 512 GB of memory. I want to sell it, [...] Given the high cost of the Mac studio ($20,000 or more)"
- girvo 3mo agoMy DGX Spark-alike is currently selling in my country for nearly double what I paid for it too. We live in a silly, silly world.
- oersted 3mo agoI don't understand why, that is not HBM nor a real GPU. Fine the model might fit in RAM, kinda. And it a bit like a server that you can keep on all the time, which must feel nice if you only ever had a MacBook. But it can still not run truly capable coding agents at reasonable speeds right? That doesn't make sense for the cost of quite a nice new car. If you want to spend that much, build yourself a proper workstation at least. Clearly a computer with the form-factor of an Apple TV is not going to be ideal for getting the highest performance for your buck. Is it just wannabe vibentrepreneurs with too much spare money hyping each other up?
- cmiles8 3mo agoAll indications are there will be a lot of repossessed GPUs appearing on the market before too long. Likely to be messy for a while but will open up a lot of possibilities when it’s easy to get your hands on some secondhand GPUs.
- tomaskafka 3mo agoThis sounds like weapons market after the crash of soviet republic. Never has been a better time to get a functioning tank or parts of nukes.
- deleted 3mo ago[deleted]
- pinkmuffinere 3mo agoWhat are people going to do with all the GPUs? I guess we'll all have really good gaming rigs, lol. Sincerely, perhaps there will be a new flood of crypto farms, which I suppose would drive the prices of crypto down. Other ideas for what changes are enabled by massive supply of cheap GPUs?
- baalimago 3mo agoRun AI locally instead of in cloud
- ponkpanda 3mo agoVery difficult to sense check that substack post without access to the TLB credit agreement. There are likely management service agreements from xAI proper -> SPV to cover precisely what the author talks about. Clearly, xAI could play games but without seeing the docs (which are not public), it's very difficult. This article's basic point is right though. On the other hand, the LTV of this deal was approx 50% debt-financed (not too high; very much depends on the "V"). At 12.5%, it's not as if its being priced as a high quality asset. Overall, substack post was too bearish. The wider point is that there's a lot of froth tied to what has now become systemically opaque - namely the circular deal flow that every hyperscaler, nvidia, neoclouds and friends are now engaged in. When the proverbial hits the fan, that stuff will be difficult to price and find few willing buyers with the competence to underwrite. The systemic issues are the bigger concern than one specific deal imo.
- Animats 3mo ago"Secondary market data shows moderately-used 2 to 3 year old GPUs trading at 50% to 70% of new pricing under normal conditions." That's for good NVidia H100 units.[1] There's a shortage of those. That seems to be the price after removal, cleaning, testing and refurbishing. Raw units removed from a shutdown will not be as valuable. H100 units are available on eBay, but multiple sellers are using the same picture of a new unit in its original packaging, a bad sign.[2] Some even have pictures with the logos of a competitor. [1] https://introl.com/blog/secondary-gpu-markets-buying-selling-used-hardware-guide-2025 https://introl.com/blog/secondary-gpu-markets-buying-selling... [2] https://www.ebay.com/shop/nvidia-h100-gpu?_nkw=nvidia+h100+gpu&msockid=3f8b9a1d8ff965f92c9d881c8e9664ed https://www.ebay.com/shop/nvidia-h100-gpu?_nkw=nvidia+h100+g...
- lightbendover 3mo agoWho is buying these at 70% of new pricing given the sky high likelihood of them being shot? Maybe it's safe to buy from small labs that went under quickly, but I can't imagine a cluster that has been operating near its thermal limits for a couple years fetching that kind of resale.
- threetonesun 3mo agoIt was true of crypto GPUs too, although mostly people picking them up for gaming. Always seems high to me too but if you can get any guarantee of them not being on fire when they were pulled the bathtub curve keeps you pretty safe, thermal limits are limits for a reason.
- gnfargbl 3mo ago> The job of an operations team is to keep all of this in steady state. They know which racks run hot in summer, which cooling loops have been flaky since the last firmware update, which jobs to re-route when a node degrades but has not failed yet. None of that knowledge is written down. It lives in the team. Hmmn. All of this information should live in the monitoring system, in which case any frontier model will be able to get to grips with it in short order. It feels like the author doesn't really fully understand the changes brought about by the systems they are writing about.
- Planktonne 3mo agoIt's generated prose; the author's understanding doesn't factor in, because they didn't actually write it.
- hellohackers127 3mo ago[flagged]
- NohatCoder 3mo agoI suspect you are right, but it is getting really difficult to tell the difference between AI and someone who simply don't know what they are writing about.
- somat 3mo agoIsn't it worth exactly what you can sell it for. a few ways to do this. slow and awkward, best market match: The auction. sell to highest bidder. faster and more customer friendly but poor market match until a lot of units sold: The store. guess price, adjust up or down to reach sell frequency desired. fast and good market match but takes a knowledgeable customer base: The reverse auction. Start with price too high lower it over time until it sells.
- charcircuit 3mo ago>adjust up or down to reach sell frequency desired. >Start with price too high lower it over time until it sells. These are the same strategy.
- deleted 3mo ago[deleted]
- tim-tday 3mo agoI’ll take “people who sell used hardware” for 500 Alex.
- Zarathustra30 3mo agoIs the 30-50% of face value realistic for "Liquidation Value"? If one organization has to liquidate, sure. But if the bubble pops and many groups have to liquidate at once? Owners will be lucky if they can dodge the recycling fees.
- nikanj 3mo agoEssentially nothing, fractions of a penny on a dollar, because players who could afford paying real money won't risk the crusty old hardware - so you're limited to buyers who still need a massive cluster but don't have AI infinite money glitch enabled
- andix 3mo agoAre failing GPUs really a big issue? I don't know how they behave when they die, but if it can be detected quickly, the affected nodes can just be removed from the pool.
- claytonjy 3mo agoI think there’s a bit more nuance and complexity here. I built and operated an application that used 1-2000 L4 GPUs in production for a couple of years. Long-term GCP reservations running in a GKE cluster. At that scale we had a few GPU failures per week, and once saw 3 in a day. Nearly every failure required an engineer to manually intervene to get rid of the bad node, and then file a ticket with GCP support as they requested. NVIDIA GPUs throw an “XID” code when they fail, which can be seen from serial port logs for GCP compute nodes (not through k8s!). If you’re lucky, it fires right as your application starts to fail, but there’s often a delay of several minutes. Even when you get an XID, by default GKE only responds to one or a few of them. You can expand the list via configuration, but the reconciliation loop is so slow that might still take 10-15 minutes during which a pod is puking errors and someone might be getting paged. They’re working on it, and we never saw a single XID after we migrated to H100s, so the situation is improving. I imagine other clouds are even worse, though iirc Azure was leading some effort to improve k8s node problem detector to include accelerator problems so maybe they have a better story. Training workloads are rather more sensitive to a node failure given that many modern training runs (including SFT etc) need multiple nodes where topology matters, and they might not have another e.g. 8xH100 box in the right place when one fails.
- jmyeet 3mo agoThis is a good article. I've been curious about how this is going to play out. A couple of data points: 1. An enthusiast had a project to get a V100 working on his PC [1]. This was a ~$10k GPU 10 years ago. It's now sold for scrap; 2. The A100 came out in 2020 and cannot run a large model like DeepSeek v4 Pro. It can run Flash. You need a 16xH100 cluster to run Pro and that's a ~4 year old GPU and AFAICT 8xB100 or 4xB200; 3. We're about to roll out R100/R200s. I'm surprised that NVidia is moving to a 1 year product cycle (per this article) because the big question I've had is what's that going to do to existing investments in GPUs. Why? Because if 4xR100 can do the work of 32xH100 then that's a massive advantage in performance-per-Watt, which I think is going to be the only metric that ends up mattering. In addition to raw power, new capabilities are developed and come online. For example, certain smaller, more efficient quantization methods just didn't exist on older hardware. Oh, another thought from this: a 9% annual failure rate just goes to show you how ridiculous the idea of orbital data centers really is. Orbital DCs were always just a pump-and-dump scheme for SpaceX's IPO. Currently it gets expensive to run models larger than ~31B locally. You start to need some pretty expensive hardware. That's going to change. I don't expect we'll be running 1T+ models on a Macbook Pro within 5 years (at reasonable inference rates) but I think people today will be shocked at what's being run locally in 5 years and that'll easily be 100-200B+ models. [1]: https://www.hackster.io/news/hacking-a-server-grade-nvidia-gpu-into-a-home-desktop-674ada7032a7 https://www.hackster.io/news/hacking-a-server-grade-nvidia-g...
- keeda 3mo agoA relevant article I found from an industry insider (which could indicate bias but also relevance): https://www.whitefiber.com/blog/understanding-gpu-lifecycle https://www.whitefiber.com/blog/understanding-gpu-lifecycle Which has this anecdotal data point: >... when I left Paperspace in mid-2024, our M4000 GPUs, nine-year-old GPUs, were still consistently utilized at near-total capacity. That’s not a typo. Nine. Years. Old. Still booked, still working, still generating revenue. Also, I won't claim to understand accounting, but in general it seems it is advantageous to accelerate depreciation schedules for high CapEx industries because they lower taxes: https://leyton.com/us/insights/articles/what-is-accelerated-depreciation-and-why-is-it-a-tax-advantage/ https://leyton.com/us/insights/articles/what-is-accelerated-... As such you'd assume these cloud providers to want faster depreciation of their GPU assets rather than slower? I suppose in this case they do have an incentive to show bigger revenue numbers, but there seems to be a trade-off here that is not being discussed?
- mertleee 3mo ago[dead]
- chasil 3mo agoShould I be worried that my plant is owned by Apollo? No, because I am retiring next month!
- andix 3mo agoI'm wondering if all those GPUs ever end up on the second hand market for us to buy. Or if they will get refurbished multiple times and get used in second tier data centers until the chips die.
- trollbridge 3mo agoThey’re available but rarely a good idea for the average person who wands small scale AI - for the same money you’re far better off getting an RTX 6000, 5090, or R9700.
- AYBABTME 3mo agoI don't get how this is unique to GPU clusters. As a general rule, underwriters are not qualified to operate and maintain the assets they underwrite loans for. That's why houses, cars, equipment, ... go at auction at a fraction of their value. And why lenders have insurance.
- venussnatch 3mo agoOr... auctions are a terrible way to buy big expensive risky purchases. Auction prices account for that risk, you don't get to do all of the verification you do for normal purchases.
- roadbuster 3mo ago> I don't get how this is unique to GPU clusters It isn't, and you're right. This is just a long-winded article by someone who thinks they've come across a deep, crucial insight. I witnessed a bank foreclosure stemming from large, unpaid loans to a lumber mill. The bank absolutely didn't want to take possession of the operation, but had no choice when the founder decided to call it a day. The bank had no idea what to do with finished lumber sitting in the drying ovens, let alone the entirety of mill infrastructure itself. After struggling to find a buyer, they hired the founder as a consultant to handle liquidation. The same would happen with a repossessed datacentre.
- marticode 3mo ago[dead]
- danielmarkbruce 3mo agoIn many cases they could literally do it but just don't have the scale - a small team of folks can finance a lot of real estate development for example. They can't do all the work of the development itself even if capable. It's just too much work.
- AtlasBarfed 3mo agoWell it certainly doesn't appreciate. And I feel that AI hardware is going to rapidly depreciate faster than even typical PC hardware
- jujube3 3mo agoA lot of hardware literally has appreciated in the last year or two.
- dfedbeef 3mo ago\__
- thelastgallon 3mo agoWhy would anyone buy these used when the new one, Vera Rubin can generate 10× more tokens per megawatt? Electricity costs are the biggest cost. More importantly, DCs are of a fixed power capacity. A power efficient GPU will mean it can fit in legacy DCs, don't need mega scale newfangled DCs. Old GPUs in the recent years should be worthless.
- imtringued 3mo agoThe price of the used stuff will drop until both options look equally good. The thing that might be difficult to understand is that the new operators might not run the GPUs at full capacity and therefore their energy bills are much lower than you're assuming. Hence the price doesn't drop to nothing, it stays at some non zero value.
- trklausss 3mo agoThe article frames a GPU cluster as if it was rocket science. If you swap a GPU cluster with, say, a power plant, the question doesn't change a bit. Also high complexity, multiple variables, you don't know the current output power, scheduled maintenance, skilled labor needed etc. In the end, they will either figure it out, or contract a company that can figure it out. Truth is, there will be many companies wanting to do the rental of those clusters for AI...
- possiblyburrito 3mo ago[flagged]
- rahulladumor 3mo ago[flagged]
- PeterStuer 3mo agoLearning from crypto, a GPU's life foes not end because it fails, althougg ofc that happens, but because an alternative comes along that makes running the old equipment no longer power efficient under the new market conditions.
- thhfgujgik5 3mo agoExcept, there's no real secondary market for SXM cards.
- PeterStuer 3mo agoThere are plenty SXM to PCIe adapters and I do expect these to keep up with newer generations as demand surfaces. But apart from homelabbers, will there be enough demand to have a reasonable secondary market outlet? The scenario after all is either a 'bubble burst' meaning thete will be oversupply of new gen, or tech advances rendering previous gen commercially obsolete in a commodity market. The latter you can compare to 2 gen old server equipment on eBay. That is going for 10% of original rather than 50-70%, so basically a full writeoff.
- petesergeant 3mo ago“Nobody knows” = “the method sophisticated lenders are using to determine this is a commercial secret” If you are the CFO of a credible AI company you can pick up the phone and get some advice exceptionally accurate numbers on what the market things the value is very quickly.
- guess_who_is 3mo agoAmerican AI corps will soon find out the worth once Huawei brings the cheap accelerators in the market. 6-9 months left.
- greenoracle9 3mo ago[dead]
- gadders 3mo agoA lot of big infra projects end up going bust, and then someone else buying up the company and its assets because they no longer have the original build costs to bear - EG EuroTunnel, Iridium etc. Someone should stand up a vulture fund to buy these data centres off companies when they go bust.
- jgalt212 3mo ago9% failure rate seems very high. What's the failure rate of a server CPU? Is the GPU failure rate high because it's 1000s of CUDA cores instead of 128 (Xeon)? If so, can't the H100 just work around the busted CUDA core(s)?
- baalimago 3mo agoIs this the CDOs of the AI-bubble?
- effnorwood 3mo agoI do. As much as they will pay.
- LakshmiKiranG 3mo ago[dead]