5 ms·

I really don't understand the argument that nvidia GPUs only work for 1-3 years. I am currently using A100s and H100s every day. Those aren't exactly new anymor

• iancmceachern 9mo ago
  They're no longer energy competitive. I.e. the amount of power per compute exceeds what is available now. It's like if your taxi company bought taxis that were more fuel efficient every year.
  • bob1029 9mo ago
    Margins are typically not so razor thin that you cannot operate with technology from one generation ago. 15 vs 17 mpg is going to add up over time, but for a taxi company it's probably not a lethal situation to be in.
    • deleted 9mo ago
      [deleted]
    • iancmceachern 9mo ago
      Tell that to the airline industry
      • bob1029 9mo ago
        I don't think the airline industry is a great example from an IT perspective, but I agree with regard to the aircraft.
      • hibikir 9mo ago
        And yet they aren't running planes and engines all from 2023 or beyond: See the MD-11 that crashed in Louisville: Nobody has made a new MD-11 in over 20 years. Planes move to less competitive routes, change carriers, and eventually might even stop carrying people and switch to cargo, but the plane itself doesn't get to have zero value when the new one comes out. An airline will want to replace their planes, but a new plane isn't fully amortized in a year or three: It still has value for quite a while
    • SchemaLoad 9mo ago
      At least with crypto mining this was the case. Hardware from 6 months ago is useless ewaste because the new generation is more power efficient. All depends on how expensive the hardware is vs the cost of power.
  • mikkupikku 9mo ago
    If a taxi company did that every year, they'd be losing a lot of money. Of course new cars and cards are cheaper to operate than old ones, but is that difference enough to offset buying a new one every one to three years?
    • wordpad 9mo ago
      If your competitor refreshes their cards and you dont, they will win on margin. You kind of have to.
      • lazide 9mo ago
        Not necessarily if you count capital costs vs operating costs/margins. Replacing cars every 3 years vs a couple % in efficiency is not an obvious trade off. Especially if you can do it in 5 years instead of 3.
        • zozbot234 9mo ago
          You can sell the old, less efficient GPUs to folks who will be running them with markedly lower duty cycles (so, less emphasis on direct operational costs), e.g. for on-prem inference or even just typical workstation/consumer use. It ends up being a win-win trade.
          • lazide 9mo ago
            Then you’re dealing with a lot of labor to do the switches (and arrange sales of used equipment), plus capital float costs while you do it. It can make sense at a certain scale, but it’s a non trivial amount of cost and effort for potentially marginal returns.
            • pixl97 9mo ago
              Building a new data center and getting power takes years to double your capacity. Swapping out out a rack that is twice as fast takes very little time in comparison.
  • echelon 9mo ago
    Nvidia has plenty of time and money to adjust. They're already buying out upstart competitors to their throne. It's not like the CUDA advantage is going anywhere overnight, either. Also, if Nvidia invests in its users and in the infrastructure layouts, it gets to see upside no matter what happens.
• linkregister 9mo ago
  The common factoid raised in financial reports is GPUs used in model training will lose thermal insulation due to their high utilization. The GPUs ostensibly fail. I have heard anecdotal reports of GPUs used for cryptocurrency mining having similar wear patterns. I have not seen hard data, so this could be an oft-repeated, but false fact.
  • munk-a 9mo ago
    I can't confirm that fact - but it's important to acknowledge that consumer usage is very different from the high continuous utilization in mining and training. It is credulous that the wear on cards under such extreme usage is as high as reported considering that consumers may use their cards at peak 5% of waking hours and the wear drop off is only about 3x if it is used near 100% - that is a believable scale for endurance loss.
  • zozbot234 9mo ago
    > I have heard anecdotal reports of GPUs used for cryptocurrency mining having similar wear patterns. If this was anywhere close to a common failure mode, I'm pretty sure we'd know that already given how crypto mining GPUs were usually ran to the max in makeshift settings with woefully inadequate cooling and environmental control. The overwhelming anecdotal evidence from people who have bought them is that even a "worn" crypto GPU is absolutely fine.
  • Melatonic 9mo ago
    It's the opposite actually - most GPU used for mining are run at a consistent temp and load which is good for long term wear. Peaky loads where the GPU goes from cold to hot and back leads to more degradation because of changes in thermal expansion. This has been known for some time now.
    • Yizahi 9mo ago
      That is commonly repeated idea, but it doesn't take into account countless token farms which are smaller than a datacenter. Basically anything from a single MB with 8 cards to a small shed with rigs, all of which tend to disregard common engineering practices and run hardware into a ground to maximize output until next police raid or difficulty bump. Plenty of photos in the internet of crappy rigs like that, and no one guarantees which GPU comes whom where. Another commonly forgotten issue is that many electrical components are rated by hours of operation. And cheaper boards tend to have components with smaller tolerances. And that rated time is actually a graph, where hour decrease with higher temperature. There were instances of batches of cards failing due to failing MOSFETs for example.
• savorypiano 9mo ago
  You aren't trying to support ad-based demand like OpenAI is.
• linuxftw 9mo ago
  I think the story is less about the GPUs themselves, and more about the interconnects for building massive GPU clusters. Nvidia just announced a massive switch for linking GPUs inside a rack. So the next couple of generations of GPU clusters will be capable of things that were previously impossible or impractical. This doesn't mean much for inference, but for training, it is going to be huge.
• legitster 9mo ago
  From an accounting standpoint, it probably makes sense to have their depreciation be 3 years. But yeah, my understanding is that either they have long service lives, or the customers sell them back to the distributor so they can buy the latest and greatest. (The distributor would sell them as refurbished)
• denimnerd42 9mo ago
  1-3 is too short but they aren’t making new A100s, theres 8 in a server and when one goes bad what do you do? you wont be able to renew a support contract. if you wanna diy you eventually you have to start consolidating pick and pulls. maybe the vendors will buy them back from people who want to upgrade and resell them. this is the issue we are seeing with A100s and we are trying to see what our vendor will offer for support.
• mbesto 9mo ago
  Not saying your wrong. A few things to consider: (1) We simply don't know what the useful life is going to be because of how new the advancements of AI focused GPUs used for training and inference. (2) Warranties and service. Most enterprise hardware has service contracts tied to purchases. I haven't seen anything publicly disclosed about what these contracts look like, but the speculation is that they are much more aggressive (3 years or less) than typical enterprise hardware contracts (Dell, HP, etc.). If it gets past those contracts the extended support contracts can typically get really pricey. (3) Power efficiency. If new GPUs are more power efficient this could be huge savings on energy that could necessitate upgrades.
  • epolanski 9mo ago
    Nvidia is moving to a 1 year release life cycle for data center, and in Jensen's words once a new gen is released you lose money for being on the older hardware. It makes no longer financially sense to run it.
    • pixl97 9mo ago
      That will come back to bite them in the ass if money leaves the AI race.
    • aurareturn 9mo ago
      Do you not see the bad logic? Companies can’t buy new Nvidia GPUs because their older Nvidia GPUs are obsolete. However, the old GPUs are only obsolete if companies buy the new Nvidia GPUs.
  • pvab3 9mo ago
    based on my napkin math, an H200 needs to run for 4 years straight at maximum power (10.2 kW) to consume its own price of $35k worth of energy (based on 10 cents per kWh)
• mbrumlow 9mo ago
  It’s not that they don’t work. It’s how businesses handle hardware. I worked at a few data centers on and off in my career. I got lots of hardware for free or on the cheap simply because the hardware was considered “EOL” after about 3 years, often when support contracts with the vendor ends. There are a few things to consider. Hardware that ages produce more errors, and those errors cost, one way or another. Rack space is limited. A perfectly fine machine that consumes 2x the power for half the output cost. It’s cheaper to upgrade a perfectly fine working system simply because it performs better per watt in the same space. Lastly. There are tax implications in buying new hardware that can often favor replacement.
  • fooker 9mo ago
    I’ll be so happy to buy a EOL H100! But no, there’s none to be found, it is a 4 year, two generations old machine at this point and you can’t buy one used at a rate cheaper than new.
    • aswegs8 9mo ago
      Not sure why this "GPUs obsolete after 3 years" gets thrown around all the time. Sounds completely nonsensical.
      • belval 9mo ago
        Especially since AWS still have p4 instances that are 6 years old A100s. Clearly even for hyperscalers these have a useful life longer than 3 years.
      • bmurphy1976 9mo ago
        It's because they run 24/7 in a challenging environment. They will start dying at some point and if you aren't replacing them you will have a big problem when they all die en masse at the same time. These things are like cars, they don't last forever and break down with usage. Yes, they can last 7 years in your home computer when you run it 1% of the time. They won't last that long in a data center where they are running 90% of the time.
• swalsh 9mo ago
  If power is the bottleneck, it may make business sense to rotate to a GPU that better utilizes the same power if the newer generation gives you a significant advantage.