3 ms·

According to the article these cards have a 9% annual failure rate.

htrp 3mo ago
>The number traces to Meta’s Llama 3 technical report, which documented 419 unforeseen disruptions across 16,384 H100s over 54 days of training, of which 148 were GPU failures and 72 were HBM3 memory failures. From an annualized number on the llama 3 training report. would be interesting to see if we have a better idea given that we're already on rubin.
christina97 3mo ago
It depends entirely on the time-to-failure distribution though whether used cards are a good deal or not. Often this kind of hardware has a bathtub shaped hazard rate, actually getting burned in cards may mean you get the weeded out solid specimens, and forgo the lemons.
lmm 3mo ago
Most hardware has an exponential (memoryless) failure distribution in practice. The bathtub curve is a myth.
- fc417fc802 3mo ago
  A bathtub curve is also memoryless though, and the back portion can be exponential. Consider two sub-populations of the same hardware. The first is small, evaded QA, and will fail early. Failures follow an exponential fall off. The second is much larger and failures rise exponentially with age. Also consider mixing in "was dropped during shipping" or "was stored improperly".
  - eru 3mo ago
    How is a bathtub curve memoryless?
    - fc417fc802 3mo ago
      My mistake, it seems I misunderstood what OP meant by that. Ignore that semantic detail and my position still stands. You've got a mix of subpopulations and many of the failure modes are definitely time dependent in some manner at least in the real world.
      - eru 3mo ago
        No worries. I just thought I was misunderstanding something.
- Xalutiono 3mo ago
  Where is this exponential thing coming from? Why would hardware failure raise exponential with linear scale? Doesn't make sense
  - eru 3mo ago
    > Where is this exponential thing coming from? That's what you get from a constant failure rate per unit of time. It's saying that hardware fails in the same way that Carbon-14 dating works.
    - Xalutiono 3mo ago
      Lets hope they add more context to it. Most if not all hardware failure on my side is def not memoryless besides the one lightning damage thingy but thats not hardwares fault.
      - lmm 3mo ago
        https://en.wikipedia.org/wiki/Memorylessness https://en.wikipedia.org/wiki/Memorylessness
onlyrealcuzzo 3mo ago
A 9% (static) failure rate, implies an average lifespan of 11 years...
- camkego 3mo ago
  With a median lifespan of 7.35 years, although I doubt that 9% per year remains constant as the number of years increases.
  - aaron695 3mo ago
    [dead]

htrp 3mo ago

>The number traces to Meta’s Llama 3 technical report, which documented 419 unforeseen disruptions across 16,384 H100s over 54 days of training, of which 148 were GPU failures and 72 were HBM3 memory failures. From an annualized number on the llama 3 training report. would be interesting to see if we have a better idea given that we're already on rubin.

christina97 3mo ago

It depends entirely on the time-to-failure distribution though whether used cards are a good deal or not. Often this kind of hardware has a bathtub shaped hazard rate, actually getting burned in cards may mean you get the weeded out solid specimens, and forgo the lemons.

lmm 3mo ago

Most hardware has an exponential (memoryless) failure distribution in practice. The bathtub curve is a myth.

fc417fc802 3mo ago
A bathtub curve is also memoryless though, and the back portion can be exponential. Consider two sub-populations of the same hardware. The first is small, evaded QA, and will fail early. Failures follow an exponential fall off. The second is much larger and failures rise exponentially with age. Also consider mixing in "was dropped during shipping" or "was stored improperly".
- eru 3mo ago
  How is a bathtub curve memoryless?
  - fc417fc802 3mo ago
    My mistake, it seems I misunderstood what OP meant by that. Ignore that semantic detail and my position still stands. You've got a mix of subpopulations and many of the failure modes are definitely time dependent in some manner at least in the real world.
    - eru 3mo ago
      No worries. I just thought I was misunderstanding something.
Xalutiono 3mo ago
Where is this exponential thing coming from? Why would hardware failure raise exponential with linear scale? Doesn't make sense
- eru 3mo ago
  > Where is this exponential thing coming from? That's what you get from a constant failure rate per unit of time. It's saying that hardware fails in the same way that Carbon-14 dating works.
  - Xalutiono 3mo ago
    Lets hope they add more context to it. Most if not all hardware failure on my side is def not memoryless besides the one lightning damage thingy but thats not hardwares fault.
    - lmm 3mo ago
      https://en.wikipedia.org/wiki/Memorylessness https://en.wikipedia.org/wiki/Memorylessness

onlyrealcuzzo 3mo ago

A 9% (static) failure rate, implies an average lifespan of 11 years...

camkego 3mo ago
With a median lifespan of 7.35 years, although I doubt that 9% per year remains constant as the number of years increases.
- aaron695 3mo ago
  [dead]