6 ms·

SSD drives’ 40k-hour “death bug” continues to catch enterprises unawares

109 points by ghostpepper 4y ago

zitterbewegung 4y ago
This is quite meta (literally and also not related to the company).
- sprayk 4y ago
  Meta (the company) too, maybe. They have a pretty big datacenter footprint.
  - notacoward 4y ago
    They also know about mixing brands/batches/etc. to avoid exactly this kind of correlated failure. During one of my oncalls, we had an uncomfortably large number of failures fitting this kind of pattern, but they were well enough distributed and we had enough data redundancy that there was no data loss. Sometimes there's safety in numbers, if you know how to make those numbers work for you rather than against you.
  - loeg 4y ago
    Mostly HDDs, though.
    - toast0 4y ago
      They've got a lot of flash too. I think most of the small nodes are running from flash, and most of the databases too? There's a lot of longer term storage that's going to be rotational though, sure.
pclmulqdq 4y ago
I wonder how the institutional knowledge on correlated disk failure got lost. This used to be common with HDDs, and one large enterprise I am aware of makes sure that boot drives in a datacenter (the only SSDs of this type) are always from a mix batches at a mix of vendors.
- reaperducer 4y ago
  Possibly because so much of what happens in the tech industry has been abstracted away. The vast majority of the people responsible for the servers on the billion-dollar company I work for have never physically touched a server, a rack, or an Ethernet cable. There's a whole generation of "point and click" admins quickly gaining prominence. I suspect cloud services have contributed to this lack of knowledge, because if all you know is cloud, you don't need to know things like this.
  - hinkley 4y ago
    That, and the curse of working the same 18 months 4 times on the way to 'senior'. All of these little companies starting with brand new hardware as well. Far from all but a few of those 'garage startups' I knew of did limp along on used hardware for some time. It happened again a bit after the dotcom bubble burst (especially for office equipment) and again around 2008. But for drives in particular, people are pretty superstitious about giving them up.
    - anitil 4y ago
      > the same 18 months 4 times This feels honestly like my whole career. Just get to a point of competence and then get an offer you can't refuse.
      - hinkley 4y ago
        The best I can say is try not to keep coming into the same point in the lifecycle of each new company, and try to pay attention to things you didn't last time instead of just jumping into your comfort zone. It can help seeing people struggle with tech debt or bad decisions you had nothing to do with, if you can put yourself into the shoes of the person who did it. The wise man learns from the mistakes of others. But it does mean you have to do a little more sleuthing than the typical postmortem that just blames everything on an engineer who quit six months ago. That's facile, and neglects the contributions of people who still work here and/or think the same way.
  - 4y ago
goatcode 4y ago
It's been a while since I've been in the hardware-level IT side of things, but is it not unwise to run a production server with SanDisk drives?
- batmansmom1 4y ago
  What is the industry standard for this?
  - goatcode 4y ago
    No idea now. I usually look to BackBlaze's periodic drive performance reports when I'm shopping, but even that's been a while for me.
    - walrus01 4y ago
      at a large enough scale, almost everything backblaze uses is sufficiently reliable as measured by percentage of any pool of 100 drives that fails in a year, as they have a large amount of parity / data duplication across multiple pods, hot spares, etc. I haven't seen any drives that really stood out as terrible in their reports in the last couple of years for 12TB+ spinning disks, although the HGST product do seem to be the most reliable.
      - goatcode 4y ago
        The manufacturers whose drives they use provide a good basis for my consideration. I don't see certain names on their lists sometimes, so my search usually is narrowed down by that.
        walrus01 4y ago
        there's literally only 3 spinning HDD manufacturers on earth now so if a name doesn't appear on their list now it's because it doesn't exist. some of the things on their list are now brands owned by a single company through consolidation.
tjoff 4y ago
> The update corrects a logging check: “Assert had a bad check to validate the value of circular buffer’s index value. Instead of checking the max value as N, it checked for N-1,” Dell’s advisory said. So the code was never tested even once?
- nix0n 4y ago
  It could have been checked once, it just wasn't checked N times in a row.
- swatcoder 4y ago
  Possibly, but if you don’t think that’s common you may be grossly overestimating how thorough even the best testing process can be. Engineering processes are systems with their own bugs, and the people/systems that implement them can’t deliver perfectly either. There are always going to be things that slip through, and some of them are bound to look embarrassingly sloppy.
  - tjoff 4y ago
    Testing that a circular buffer can wrap is a very very low bar though. I'm not talking automated testes or anything. Just once, on your own machine. I agree that it is a bit unfair to pick an example after the fact. But it does not inspire confidence.
    - rmatt2000 4y ago
      > I'm not talking automated testes or anything. I'd love to see an entire thread on this topic alone.
  - hinkley 4y ago
    A lot of preventable accidents come from the human brain discounting the threat after a large number of positive interactions. For all we know there once was a test three refactorings ago and someone fenced it away.
- deleted 4y ago
  [deleted]
notacoward 4y ago
> in a specific line of older, end-of-life SanDisk SAS SSDs They weren't end-of-life when they were sold this way. That's just the PR department trying to make their failure seem less severe than it was.
- toast0 4y ago
  Well, we now know the lifetime for those SSDs is 40k hours. At least we don't have to wonder.
  - detaro 4y ago
    Or you can choose to install the firmware update fixing the issue and get back that sweet sweet uncertainty that your drive might live longer.
treeman79 4y ago
Can’t find it. But there was an old article where some young guy at a conference was explaining to Netflix why SSDs weren’t going to work due to drive failure cost. Response was epic. Everything you say is true. But we don’t care, we rent the servers.
- supernova87a 4y ago
  > Response was epic. As in, no one believed him at the time? And what does Netflix have a need for SSDs for? Their load seems entirely predictable / serveable / stable with spinning disks and reasonably (but not lightning) fast access patterns? I would have guessed the network latency is always dominant.
  - jasonwatkinspdx 4y ago
    There's still a huge long tail, so iops matter. Netflix definitely wants to minimize any time you ever see a waiting on network spinner. I dunno what talk the above comment is referring to but most likely the young hero was saying something somewhat clueless, not being kept down by the tyranny of older idiots or whatever is being insinuated. Netflix's content is pre baked. They have to do like a dozen versions of every episode to get the best experience across the variety of hardware, but it's not livestreaming or such. Where they use SSDs it's not an erase/update heavy workload, at least for the CDN part (Netflix does a ton of other stuff including basically running entire movie studios out of the cloud, not referring to that). Density and power also matter a ton as they want their POPs to be as easy as possible to convince people to run at peering points.
  - SteveNuts 4y ago
    Netflix has a lot more to it than just serving video data
    - NavinF 4y ago
      Sure but don't they use AWS for everything else? I have no idea what that guy is talking about or why Netflix would care about SSD failures more than a normal company that just uses mirrors/raidz for everything.
stefan_ 4y ago
Because "SSD fails at a fixed amount of hours" is a recall-level bug and well.. where is the recall then? This should be a class-action lawsuit.
- kabdib 4y ago
  Is there anything on the SanDisk site about this? I checked a few weeks ago and saw nothing.
- jonas21 4y ago
  There's no need for a recall because it can be fixed with a firmware update. And as the article mentions, vendors did issue urgent/critical firmware updates for it back in 2020. https://www.dell.com/support/home/en-us/drivers/driversdetails?driverid=8h6hj&oscode=w12r2 https://www.dell.com/support/home/en-us/drivers/driversdetai... https://support.hpe.com/hpesc/public/docDisplay?docLocale=en_US&docId=a00097382en_us https://support.hpe.com/hpesc/public/docDisplay?docLocale=en...
  - koheripbal 4y ago
    Firmware updates on hardware have a very very low update rate outside of professional datacenters.
    - deleted 4y ago
      [deleted]
  - DannyBee 4y ago
    Uh, what? In most industries, it would be a recall and fix. The recall is used to notify people, and to pull unfixed products off shelves.
    - wink 4y ago
      I think the problem is like.. if you buy a car, then they will call you if a recall happens. If they'd call you every month to sell you something you'd probably tell them off in such a way that they would stop. If food is recalled you may see it at the checkout in the grocery store. No one registers their hardware at the vendor because all they do is spam you with offers then. Or they change their system and despite you registering you'll still get no notice.
ksec 4y ago
Well, I thought it should be well known by now since it took down HN a while back.

109 points by ghostpepper 4y ago

zitterbewegung 4y ago

This is quite meta (literally and also not related to the company).

sprayk 4y ago
Meta (the company) too, maybe. They have a pretty big datacenter footprint.
- notacoward 4y ago
  They also know about mixing brands/batches/etc. to avoid exactly this kind of correlated failure. During one of my oncalls, we had an uncomfortably large number of failures fitting this kind of pattern, but they were well enough distributed and we had enough data redundancy that there was no data loss. Sometimes there's safety in numbers, if you know how to make those numbers work for you rather than against you.
- loeg 4y ago
  Mostly HDDs, though.
  - toast0 4y ago
    They've got a lot of flash too. I think most of the small nodes are running from flash, and most of the databases too? There's a lot of longer term storage that's going to be rotational though, sure.

pclmulqdq 4y ago

I wonder how the institutional knowledge on correlated disk failure got lost. This used to be common with HDDs, and one large enterprise I am aware of makes sure that boot drives in a datacenter (the only SSDs of this type) are always from a mix batches at a mix of vendors.

reaperducer 4y ago
Possibly because so much of what happens in the tech industry has been abstracted away. The vast majority of the people responsible for the servers on the billion-dollar company I work for have never physically touched a server, a rack, or an Ethernet cable. There's a whole generation of "point and click" admins quickly gaining prominence. I suspect cloud services have contributed to this lack of knowledge, because if all you know is cloud, you don't need to know things like this.
- hinkley 4y ago
  That, and the curse of working the same 18 months 4 times on the way to 'senior'. All of these little companies starting with brand new hardware as well. Far from all but a few of those 'garage startups' I knew of did limp along on used hardware for some time. It happened again a bit after the dotcom bubble burst (especially for office equipment) and again around 2008. But for drives in particular, people are pretty superstitious about giving them up.
  - anitil 4y ago
    > the same 18 months 4 times This feels honestly like my whole career. Just get to a point of competence and then get an offer you can't refuse.
    - hinkley 4y ago
      The best I can say is try not to keep coming into the same point in the lifecycle of each new company, and try to pay attention to things you didn't last time instead of just jumping into your comfort zone. It can help seeing people struggle with tech debt or bad decisions you had nothing to do with, if you can put yourself into the shoes of the person who did it. The wise man learns from the mistakes of others. But it does mean you have to do a little more sleuthing than the typical postmortem that just blames everything on an engineer who quit six months ago. That's facile, and neglects the contributions of people who still work here and/or think the same way.
- 4y ago

goatcode 4y ago

It's been a while since I've been in the hardware-level IT side of things, but is it not unwise to run a production server with SanDisk drives?

batmansmom1 4y ago
What is the industry standard for this?
- goatcode 4y ago
  No idea now. I usually look to BackBlaze's periodic drive performance reports when I'm shopping, but even that's been a while for me.
  - walrus01 4y ago
    at a large enough scale, almost everything backblaze uses is sufficiently reliable as measured by percentage of any pool of 100 drives that fails in a year, as they have a large amount of parity / data duplication across multiple pods, hot spares, etc. I haven't seen any drives that really stood out as terrible in their reports in the last couple of years for 12TB+ spinning disks, although the HGST product do seem to be the most reliable.
    - goatcode 4y ago
      The manufacturers whose drives they use provide a good basis for my consideration. I don't see certain names on their lists sometimes, so my search usually is narrowed down by that.
      - walrus01 4y ago
        there's literally only 3 spinning HDD manufacturers on earth now so if a name doesn't appear on their list now it's because it doesn't exist. some of the things on their list are now brands owned by a single company through consolidation.

tjoff 4y ago

> The update corrects a logging check: “Assert had a bad check to validate the value of circular buffer’s index value. Instead of checking the max value as N, it checked for N-1,” Dell’s advisory said. So the code was never tested even once?

nix0n 4y ago
It could have been checked once, it just wasn't checked N times in a row.
swatcoder 4y ago
Possibly, but if you don’t think that’s common you may be grossly overestimating how thorough even the best testing process can be. Engineering processes are systems with their own bugs, and the people/systems that implement them can’t deliver perfectly either. There are always going to be things that slip through, and some of them are bound to look embarrassingly sloppy.
- tjoff 4y ago
  Testing that a circular buffer can wrap is a very very low bar though. I'm not talking automated testes or anything. Just once, on your own machine. I agree that it is a bit unfair to pick an example after the fact. But it does not inspire confidence.
  - rmatt2000 4y ago
    > I'm not talking automated testes or anything. I'd love to see an entire thread on this topic alone.
- hinkley 4y ago
  A lot of preventable accidents come from the human brain discounting the threat after a large number of positive interactions. For all we know there once was a test three refactorings ago and someone fenced it away.
deleted 4y ago
[deleted]

notacoward 4y ago

> in a specific line of older, end-of-life SanDisk SAS SSDs They weren't end-of-life when they were sold this way. That's just the PR department trying to make their failure seem less severe than it was.

toast0 4y ago
Well, we now know the lifetime for those SSDs is 40k hours. At least we don't have to wonder.
- detaro 4y ago
  Or you can choose to install the firmware update fixing the issue and get back that sweet sweet uncertainty that your drive might live longer.

treeman79 4y ago

Can’t find it. But there was an old article where some young guy at a conference was explaining to Netflix why SSDs weren’t going to work due to drive failure cost. Response was epic. Everything you say is true. But we don’t care, we rent the servers.

supernova87a 4y ago
> Response was epic. As in, no one believed him at the time? And what does Netflix have a need for SSDs for? Their load seems entirely predictable / serveable / stable with spinning disks and reasonably (but not lightning) fast access patterns? I would have guessed the network latency is always dominant.
- jasonwatkinspdx 4y ago
  There's still a huge long tail, so iops matter. Netflix definitely wants to minimize any time you ever see a waiting on network spinner. I dunno what talk the above comment is referring to but most likely the young hero was saying something somewhat clueless, not being kept down by the tyranny of older idiots or whatever is being insinuated. Netflix's content is pre baked. They have to do like a dozen versions of every episode to get the best experience across the variety of hardware, but it's not livestreaming or such. Where they use SSDs it's not an erase/update heavy workload, at least for the CDN part (Netflix does a ton of other stuff including basically running entire movie studios out of the cloud, not referring to that). Density and power also matter a ton as they want their POPs to be as easy as possible to convince people to run at peering points.
- SteveNuts 4y ago
  Netflix has a lot more to it than just serving video data
  - NavinF 4y ago
    Sure but don't they use AWS for everything else? I have no idea what that guy is talking about or why Netflix would care about SSD failures more than a normal company that just uses mirrors/raidz for everything.

stefan_ 4y ago

Because "SSD fails at a fixed amount of hours" is a recall-level bug and well.. where is the recall then? This should be a class-action lawsuit.

kabdib 4y ago
Is there anything on the SanDisk site about this? I checked a few weeks ago and saw nothing.
jonas21 4y ago
There's no need for a recall because it can be fixed with a firmware update. And as the article mentions, vendors did issue urgent/critical firmware updates for it back in 2020. https://www.dell.com/support/home/en-us/drivers/driversdetails?driverid=8h6hj&oscode=w12r2 https://www.dell.com/support/home/en-us/drivers/driversdetai... https://support.hpe.com/hpesc/public/docDisplay?docLocale=en_US&docId=a00097382en_us https://support.hpe.com/hpesc/public/docDisplay?docLocale=en...
- koheripbal 4y ago
  Firmware updates on hardware have a very very low update rate outside of professional datacenters.
  - deleted 4y ago
    [deleted]
- DannyBee 4y ago
  Uh, what? In most industries, it would be a recall and fix. The recall is used to notify people, and to pull unfixed products off shelves.
  - wink 4y ago
    I think the problem is like.. if you buy a car, then they will call you if a recall happens. If they'd call you every month to sell you something you'd probably tell them off in such a way that they would stop. If food is recalled you may see it at the checkout in the grocery store. No one registers their hardware at the vendor because all they do is spam you with offers then. Or they change their system and despite you registering you'll still get no notice.

ksec 4y ago

Well, I thought it should be well known by now since it took down HN a while back.