4 ms·

Keeping 20k GPUs healthy

134 points by jxmorris12 9mo ago

bflesch 9mo ago
In his newsletter Ed Zitron hammered down the point that GPUs depreciate quickly, but these kind of reliability issues are shocking to read. The GPUs are so common to fail that they hang out in a 24/7 slack channel with customers like Meta (who apparently can't set up a cluster themselves..). Ed Zitron also called out the business model of GPU-as-a-service middleman companies like modal deeply unsustainable, and I also don't see how they can make a profit if they are only reselling public clouds. Assuming they are VC funded the VCs need returns for their funds. Unlike fiber cable during the dot com boom the currently used GPUs eventually end up in the trash bin. These GPUs are treated like toilet paper, you use them and throw them away, nothing you will give to the next generation. Who will be the one who marks down these "assets"? Who is providing money to buy the next batch of GPUs, now that billions are already spent? Maybe we'll see a wave of retirements soon. > It’s underappreciated how unreliable GPUs are. NVIDIA’s hardware is a marvel, the FLOPs are absurd. But the reliability is a drag. A memorable illustration of how AI/ML development is hampered by reliability comes from Meta’s paper detailing the training process for the LLaMA 3 models: “GPU issues are the largest category, accounting for 58.7% of all unexpected issues.” > Imagine the future we’ll enjoy when GPUs are as reliable as CPUs. The Llama3 team’s CPUs were the problem only 0.5% of the time. In my time at Modal we can’t remember finding a single degraded CPU core. > For our Enterprise customers we use a shared private Slack channel with tight SLAs. Slack is connected to Pylon, tracking issues from creation to resolution. Because Modal is built on top of the cloud giants and designed for dynamic compute autoscaling, we can replace bad GPUs pretty fast!
- ares623 9mo ago
  I suppose NVidia could invest in making their GPUs more reliable? But then that'll make everything else even more expensive lol. If only one of the companies on the chain can take one for the team.
  - touisteur 9mo ago
    And NVIDIA supposedly has the exact knowhow for reliablity, as their Jetson 'industrial' parts are qualified for 10-15 years at maximal temp. Of course Jetson is on another point of the flops and watts curve. Just wondering, if reliability increases if you slow down your use of GPUs a bit. Like pausing more often and stopping chasing every bubble and nvlink-all-reduce optimization.
    - dsrtslnd23 9mo ago
      Jetson uses LPDDR though. H100 failures seem driven by HBM heat sensitivity and the 700W+ envelope. That is a completely different thermal density I guess.
      - zozbot234 9mo ago
        Reliability also depends strongly on current density and applied voltage, even more perhaps than on thermal density itself. So "slowing down" your average GPU use in a long-term sustainable way ought to improve those reliability figures via multiple mechanisms. Jetsons are great for very small-scale self-contained tasks (including on a performance-per-watt basis) but their limits are just as obvious, especially with the recently announced advances wrt. clustering the big server GPUs on a rack- and perhaps multi-rack level.
        touisteur 9mo ago
        I don't have first-hand knowledge on HBM GPUs but on the RTX Blackwell 6000 Pro Server, the perf difference between the free up-to-600W and the same GPU capped at 300W is less than 10% on any workload I could (including Tensor Core-heavy ones) throw at it. That's a very expensive 300W and I wonder what tradeoff made them go for this, and whether capping is here a way to increase reliability. ... Wonder whether there's any writeup on those additional 300 Watts...
        storystarling 9mo ago
- pixl97 9mo ago
  >These GPUs are treated like toilet paper, you use them and throw them away, nothing you will give to the next generation. I'm guessing this may be highly dependant on what the bathtub curve looks like, and how much the provider wants to spend on cooling. Of course with Nvidia being a near monopoly here, they might just not give a fuck and will pump out cards/servers with shitty reliability rates simply because people keep buying them and they don't suffer any economic loss or have to sit in front of a judge. Be interesting to see what the error rate per TFLOP (no /s, we're looking at operations not time) is compared to older generation cards.
  - topaz0 9mo ago
    > Of course with Nvidia being a near monopoly here, they [...] will pump out cards/servers with shitty reliability rates simply because people keep buying them and they don't suffer any economic loss or have to sit in front of a judge. Presumably this can't last that much longer, because the people that are buying/running these are already taking on loads of debt/venture capital to buy the past/current round of hardware without seeing much revenue from it. It's much harder to ask investors for multiples of your annual revenue just to maintain your current capabilities than it was a couple years ago to ask for many multiples of your revenue to expand your capabilities dramatically.
- charles_irl 9mo ago
  > Ed Zitron also called out the business model of GPU-as-a-service middleman companies like modal deeply unsustainable, and I also don't see how they can make a profit if they are only reselling public clouds. You got a link for that? I work on Modal and would be interested in seeing the argument! We think building a proper software layer for multitenant demand aggregation on top of the public clouds is sufficient value-add to be a sustainable business (cf DBRX and Snowflake).
  - bflesch 9mo ago
    It was on his last newsletter, but I can't link it right now.
  - pphysch 9mo ago
    Snowflake and Databricks provide data storage and pipeline features and therefore have extraordinary lock-in potential, which allows them to have sustainable business models. GPU compute is essentially fungible. That's quite a stretch to compare those business models. Snowflake and Databricks don't necessarily have the best "value-add" and they don't need to.
deleted 9mo ago
[deleted]
bluedino 9mo ago
I help run a fleet of GPU servers, and I might see 1 DIMM or SSD failure for every 50-100 GPU failures. I realize NVIDIA is just cranking them out as fast as they can, but the quality on them is terrible. They overheat, disappear after you reboot, they fall off the bus, memory failures, and then mix in all the software crashes your users generate... Our current server vendor is actually good at replacing them, unlike our previous vendor, but the failure rates are just insane. If any other component failed this much we'd have the vendor buy the servers back.
- dlcarrier 9mo ago
  They're also run far closer to the edge of their operational limits than CPUs, so you're far more likely to get one that barely passes manufacturing tests, then degrades just a little tiny bit and stops working.
- bigwheels 9mo ago
  FWIW, NVIDIA enterprise hardware does come with good warranty and prompt RMA service. A deep dive on why these beastly cards fail so frequently compared to all other common current day hardware would be fascinating!
  - indoordin0saur 9mo ago
    I don't know much about the subject but GPUs were originally meant for gaming and would run for a few hours to several hours a day and then would get rest periods. The amount of power draw on them would also vary throughout the time they were being actively used. With constant 24/7 usage at max capacity is it just possible that they are being pushed beyond what they were originally engineered for?
    - ls65536 9mo ago
      My intuition would be that constant usage (not exceeding maximum rated capacity/thermals/etc.) should generally result in less wear compared to the more frequent thermal cycling that you might expect from intermittent use, but maybe there's something else going on here too. I suppose this would depend on what exactly the cause of the failure is. Either way, these are obviously being intentionally sold to be used for non-gaming-type workloads, so it wouldn't be a good argument to state that they're just being (ab)used beyond what they were inteded for...unless somehow they really are being pushed beyond design limits, but given the cost of these things I can't imagine anyone doing this willingly with a whole fleet of them.
eleventyseven 9mo ago
> Today, we’re sharing our GPU reliability system as both a demonstration of our commitment to Modal customers and as a guide for fellow travelers renting hyperscaler or neocloud cards. It’s dangerous to go alone! Take this. > We’ve chosen not to refer to cloud providers directly, but instead give them anonymized A, B, C, D identifiers. If you want know who’s who, track the clues or buy us a beer sometime. Come on, either name names or admit it is pure PR. Edit: or will someone who can decode the clues weigh in?
- pests 9mo ago
  I was curious and don't know enough of the cloud internals so asked an LLM: Cloud A: AWS (Amazon Web Services) Cloud B: Azure (Microsoft Azure) Cloud C: GCP (Google Cloud Platform) Cloud D: OCI (Oracle Cloud Infrastructure) Gemini had some decent evidence for each choice too, but I didn't confirm anything.
  - danpalmer 9mo ago
    Would you like to share that evidence?
    - FergusArgyll 9mo ago
      ChatGPT concurs. https://chatgpt.com/share/6972c8fa-e03c-8003-a562-ab99f78d025a https://chatgpt.com/share/6972c8fa-e03c-8003-a562-ab99f78d02...
    - pests 9mo ago
      Here was the chat: https://gemini.google.com/share/3f209aa5f6b8 https://gemini.google.com/share/3f209aa5f6b8
  - tryauuum 9mo ago
    wouldn't they run on neoclouds? paying AWS for GPUs is shooting yourself in the foot money-wise
smsx 9mo ago
Are the numbers in the H100 PCIE vs SXM table swapped for rows 3 onwards? It looks to me like the PCI is showing higher GiB/s numbers, which is counter to expectations. Or am I misunderstanding those benchmarks?
- thundergolfer 9mo ago
  You're not misunderstanding, the PCIe does indeed outperform on the memory bandwidth tests. But it gets dominated on FLOP/s and real-world application bencharks.
zkvx7a 9mo ago
A taxonomy and statistics of GPU failures are described in this paper Story of Two GPUs: Characterizing the Resilience of Hopper H100 and Ampere A100 GPUs https://dl.acm.org/doi/10.1145/3712285.3759821 https://dl.acm.org/doi/10.1145/3712285.3759821
- magicalhippo 9mo ago
  Very interesting. From the paper: H100 shows 3.2 × lower per-GPU mean time between errors (MTBE) compared to A100 for uncorrectable ECC memory errors. The per-GB MTBE of the H100’s HBM3 memory is 24% lower (∼ 8.5M hours) than the A100’s HBM2e memory (∼ 11.3M hours). We conjecture that the reduction in memory resilience stems from H100’s higher memory capacity. We attribute the decrease in resilience is primarily due to the higher memory capacity (96 GB vs. 40 GB, a 2.4 × increase), which increases the chances of bit flips. We additionally hypothesize that H100 memory resilience is worse due to (a) a lower signaling voltage that increases susceptibility to bit flips and (b) an increased number of stacks that make heat dissipation challenging and degrade the resilience of memory modules, of the HBM3 memory. Increasing voltage just makes the heat dissipation problem worse, so probably can't just crank that up. From what I can gather, a typical A100 or H100 is air cooled. Sounds like liquid cooling them might help, or at least allow you to bump up those voltages without thermal issues.
gregjm 9mo ago
I wonder why H100 H2D and D2H unpinned memcpy bandwidth is *faster* on PCIe with vendor B than on SXM with vendor D. Is resizable BAR available on PCIe but not SXM? Or, could it be a software configuration difference? The driver API flag CU_MEMHOSTREGISTER_IOMEMORY states that host memory being physically contiguous may matter to the driver, in this context for memory-mapped memory. If vendor B has THP enabled or configured differently than vendor D, small allocations up to 2 MiB could be physically contiguous which may result in higher efficiency/more bytes transferred per request. At a higher level: unpinned memcpy is a performance antipattern. Perhaps vendor D has fewer clients using unpinned memcpy in their workloads than vendor B, or they decided not to dedicate support to it for this reason. TensorFlow will go to great lengths to copy unpinned memory to a pinned staging buffer if you feed unpinned host memory tensors to a graph.
- checker659 9mo ago
  Are both using a PCIe switch? If one is and the other isn't, it could be about PCIe credit based flow control kicking in.
Surac 9mo ago
I recently had to route a PCB for a fpga using DDR3. It needed 3 designs to get the ram interface good. Dont get me wrong i have designed such things before but there are so may external factors. Now think of DDR of higher order. I think they are on the edge what can be done on todays PCB design
squeefers 9mo ago
i thought this kind of thing was the cloud providers job? this just seems to me very similar to the classic server rental model
- squeefers 9mo ago
  being downvoted for nothing only makes me sling more shit
pyuser583 9mo ago
My experience with GPU failures is they have trouble loading tons of data - suggesting it’s also the stress such a highly performant part puts on the system.

134 points by jxmorris12 9mo ago

In his newsletter Ed Zitron hammered down the point that GPUs depreciate quickly, but these kind of reliability issues are shocking to read. The GPUs are so common to fail that they hang out in a 24/7 slack channel with customers like Meta (who apparently can't set up a cluster themselves..). Ed Zitron also called out the business model of GPU-as-a-service middleman companies like modal deeply unsustainable, and I also don't see how they can make a profit if they are only reselling public clouds. Assuming they are VC funded the VCs need returns for their funds. Unlike fiber cable during the dot com boom the currently used GPUs eventually end up in the trash bin. These GPUs are treated like toilet paper, you use them and throw them away, nothing you will give to the next generation. Who will be the one who marks down these "assets"? Who is providing money to buy the next batch of GPUs, now that billions are already spent? Maybe we'll see a wave of retirements soon. > It’s underappreciated how unreliable GPUs are. NVIDIA’s hardware is a marvel, the FLOPs are absurd. But the reliability is a drag. A memorable illustration of how AI/ML development is hampered by reliability comes from Meta’s paper detailing the training process for the LLaMA 3 models: “GPU issues are the largest category, accounting for 58.7% of all unexpected issues.” > Imagine the future we’ll enjoy when GPUs are as reliable as CPUs. The Llama3 team’s CPUs were the problem only 0.5% of the time. In my time at Modal we can’t remember finding a single degraded CPU core. > For our Enterprise customers we use a shared private Slack channel with tight SLAs. Slack is connected to Pylon, tracking issues from creation to resolution. Because Modal is built on top of the cloud giants and designed for dynamic compute autoscaling, we can replace bad GPUs pretty fast!

ares623 9mo ago
I suppose NVidia could invest in making their GPUs more reliable? But then that'll make everything else even more expensive lol. If only one of the companies on the chain can take one for the team.
- touisteur 9mo ago
  And NVIDIA supposedly has the exact knowhow for reliablity, as their Jetson 'industrial' parts are qualified for 10-15 years at maximal temp. Of course Jetson is on another point of the flops and watts curve. Just wondering, if reliability increases if you slow down your use of GPUs a bit. Like pausing more often and stopping chasing every bubble and nvlink-all-reduce optimization.
  - dsrtslnd23 9mo ago
    Jetson uses LPDDR though. H100 failures seem driven by HBM heat sensitivity and the 700W+ envelope. That is a completely different thermal density I guess.
    - zozbot234 9mo ago
      Reliability also depends strongly on current density and applied voltage, even more perhaps than on thermal density itself. So "slowing down" your average GPU use in a long-term sustainable way ought to improve those reliability figures via multiple mechanisms. Jetsons are great for very small-scale self-contained tasks (including on a performance-per-watt basis) but their limits are just as obvious, especially with the recently announced advances wrt. clustering the big server GPUs on a rack- and perhaps multi-rack level.
      - touisteur 9mo ago
        I don't have first-hand knowledge on HBM GPUs but on the RTX Blackwell 6000 Pro Server, the perf difference between the free up-to-600W and the same GPU capped at 300W is less than 10% on any workload I could (including Tensor Core-heavy ones) throw at it. That's a very expensive 300W and I wonder what tradeoff made them go for this, and whether capping is here a way to increase reliability. ... Wonder whether there's any writeup on those additional 300 Watts...
        storystarling 9mo ago
pixl97 9mo ago
>These GPUs are treated like toilet paper, you use them and throw them away, nothing you will give to the next generation. I'm guessing this may be highly dependant on what the bathtub curve looks like, and how much the provider wants to spend on cooling. Of course with Nvidia being a near monopoly here, they might just not give a fuck and will pump out cards/servers with shitty reliability rates simply because people keep buying them and they don't suffer any economic loss or have to sit in front of a judge. Be interesting to see what the error rate per TFLOP (no /s, we're looking at operations not time) is compared to older generation cards.
- topaz0 9mo ago
  > Of course with Nvidia being a near monopoly here, they [...] will pump out cards/servers with shitty reliability rates simply because people keep buying them and they don't suffer any economic loss or have to sit in front of a judge. Presumably this can't last that much longer, because the people that are buying/running these are already taking on loads of debt/venture capital to buy the past/current round of hardware without seeing much revenue from it. It's much harder to ask investors for multiples of your annual revenue just to maintain your current capabilities than it was a couple years ago to ask for many multiples of your revenue to expand your capabilities dramatically.
charles_irl 9mo ago
> Ed Zitron also called out the business model of GPU-as-a-service middleman companies like modal deeply unsustainable, and I also don't see how they can make a profit if they are only reselling public clouds. You got a link for that? I work on Modal and would be interested in seeing the argument! We think building a proper software layer for multitenant demand aggregation on top of the public clouds is sufficient value-add to be a sustainable business (cf DBRX and Snowflake).
- bflesch 9mo ago
  It was on his last newsletter, but I can't link it right now.
- pphysch 9mo ago
  Snowflake and Databricks provide data storage and pipeline features and therefore have extraordinary lock-in potential, which allows them to have sustainable business models. GPU compute is essentially fungible. That's quite a stretch to compare those business models. Snowflake and Databricks don't necessarily have the best "value-add" and they don't need to.

deleted 9mo ago

[deleted]

bluedino 9mo ago

I help run a fleet of GPU servers, and I might see 1 DIMM or SSD failure for every 50-100 GPU failures. I realize NVIDIA is just cranking them out as fast as they can, but the quality on them is terrible. They overheat, disappear after you reboot, they fall off the bus, memory failures, and then mix in all the software crashes your users generate... Our current server vendor is actually good at replacing them, unlike our previous vendor, but the failure rates are just insane. If any other component failed this much we'd have the vendor buy the servers back.

dlcarrier 9mo ago
They're also run far closer to the edge of their operational limits than CPUs, so you're far more likely to get one that barely passes manufacturing tests, then degrades just a little tiny bit and stops working.
bigwheels 9mo ago
FWIW, NVIDIA enterprise hardware does come with good warranty and prompt RMA service. A deep dive on why these beastly cards fail so frequently compared to all other common current day hardware would be fascinating!
- indoordin0saur 9mo ago
  I don't know much about the subject but GPUs were originally meant for gaming and would run for a few hours to several hours a day and then would get rest periods. The amount of power draw on them would also vary throughout the time they were being actively used. With constant 24/7 usage at max capacity is it just possible that they are being pushed beyond what they were originally engineered for?
  - ls65536 9mo ago
    My intuition would be that constant usage (not exceeding maximum rated capacity/thermals/etc.) should generally result in less wear compared to the more frequent thermal cycling that you might expect from intermittent use, but maybe there's something else going on here too. I suppose this would depend on what exactly the cause of the failure is. Either way, these are obviously being intentionally sold to be used for non-gaming-type workloads, so it wouldn't be a good argument to state that they're just being (ab)used beyond what they were inteded for...unless somehow they really are being pushed beyond design limits, but given the cost of these things I can't imagine anyone doing this willingly with a whole fleet of them.

eleventyseven 9mo ago

> Today, we’re sharing our GPU reliability system as both a demonstration of our commitment to Modal customers and as a guide for fellow travelers renting hyperscaler or neocloud cards. It’s dangerous to go alone! Take this. > We’ve chosen not to refer to cloud providers directly, but instead give them anonymized A, B, C, D identifiers. If you want know who’s who, track the clues or buy us a beer sometime. Come on, either name names or admit it is pure PR. Edit: or will someone who can decode the clues weigh in?

pests 9mo ago
I was curious and don't know enough of the cloud internals so asked an LLM: Cloud A: AWS (Amazon Web Services) Cloud B: Azure (Microsoft Azure) Cloud C: GCP (Google Cloud Platform) Cloud D: OCI (Oracle Cloud Infrastructure) Gemini had some decent evidence for each choice too, but I didn't confirm anything.
- danpalmer 9mo ago
  Would you like to share that evidence?
  - FergusArgyll 9mo ago
    ChatGPT concurs. https://chatgpt.com/share/6972c8fa-e03c-8003-a562-ab99f78d025a https://chatgpt.com/share/6972c8fa-e03c-8003-a562-ab99f78d02...
  - pests 9mo ago
    Here was the chat: https://gemini.google.com/share/3f209aa5f6b8 https://gemini.google.com/share/3f209aa5f6b8
- tryauuum 9mo ago
  wouldn't they run on neoclouds? paying AWS for GPUs is shooting yourself in the foot money-wise

smsx 9mo ago

Are the numbers in the H100 PCIE vs SXM table swapped for rows 3 onwards? It looks to me like the PCI is showing higher GiB/s numbers, which is counter to expectations. Or am I misunderstanding those benchmarks?

thundergolfer 9mo ago
You're not misunderstanding, the PCIe does indeed outperform on the memory bandwidth tests. But it gets dominated on FLOP/s and real-world application bencharks.

zkvx7a 9mo ago

A taxonomy and statistics of GPU failures are described in this paper Story of Two GPUs: Characterizing the Resilience of Hopper H100 and Ampere A100 GPUs https://dl.acm.org/doi/10.1145/3712285.3759821 https://dl.acm.org/doi/10.1145/3712285.3759821

magicalhippo 9mo ago
Very interesting. From the paper: H100 shows 3.2 × lower per-GPU mean time between errors (MTBE) compared to A100 for uncorrectable ECC memory errors. The per-GB MTBE of the H100’s HBM3 memory is 24% lower (∼ 8.5M hours) than the A100’s HBM2e memory (∼ 11.3M hours). We conjecture that the reduction in memory resilience stems from H100’s higher memory capacity. We attribute the decrease in resilience is primarily due to the higher memory capacity (96 GB vs. 40 GB, a 2.4 × increase), which increases the chances of bit flips. We additionally hypothesize that H100 memory resilience is worse due to (a) a lower signaling voltage that increases susceptibility to bit flips and (b) an increased number of stacks that make heat dissipation challenging and degrade the resilience of memory modules, of the HBM3 memory. Increasing voltage just makes the heat dissipation problem worse, so probably can't just crank that up. From what I can gather, a typical A100 or H100 is air cooled. Sounds like liquid cooling them might help, or at least allow you to bump up those voltages without thermal issues.

gregjm 9mo ago

I wonder why H100 H2D and D2H unpinned memcpy bandwidth is *faster* on PCIe with vendor B than on SXM with vendor D. Is resizable BAR available on PCIe but not SXM? Or, could it be a software configuration difference? The driver API flag CU_MEMHOSTREGISTER_IOMEMORY states that host memory being physically contiguous may matter to the driver, in this context for memory-mapped memory. If vendor B has THP enabled or configured differently than vendor D, small allocations up to 2 MiB could be physically contiguous which may result in higher efficiency/more bytes transferred per request. At a higher level: unpinned memcpy is a performance antipattern. Perhaps vendor D has fewer clients using unpinned memcpy in their workloads than vendor B, or they decided not to dedicate support to it for this reason. TensorFlow will go to great lengths to copy unpinned memory to a pinned staging buffer if you feed unpinned host memory tensors to a graph.

checker659 9mo ago
Are both using a PCIe switch? If one is and the other isn't, it could be about PCIe credit based flow control kicking in.

Surac 9mo ago

I recently had to route a PCB for a fpga using DDR3. It needed 3 designs to get the ram interface good. Dont get me wrong i have designed such things before but there are so may external factors. Now think of DDR of higher order. I think they are on the edge what can be done on todays PCB design

squeefers 9mo ago

i thought this kind of thing was the cloud providers job? this just seems to me very similar to the classic server rental model

squeefers 9mo ago
being downvoted for nothing only makes me sling more shit

pyuser583 9mo ago

My experience with GPU failures is they have trouble loading tons of data - suggesting it’s also the stress such a highly performant part puts on the system.