5 ms·

Do you have a postmortem on why both servers died so fast?

dang 4y ago
It was an SSD that failed in each case, and in a similar way (e.g. both were in RAID arrays but neither could be rebuilt from the array - but I am over my skis in reporting this, as I barely know what that means). The disks were in two physically separate servers that were not connected to each other. I believe, however, that they were of similar make and model. So the leading hypothesis seems to be that perhaps the SSDs were from the same manufacturing batch and shared some defect. In other words, our servers were inbred! Which makes me want to link to the song 'Second Cousin' by Flamin' Groovies. The HN hindsight consensus, to judge by the replies to https://news.ycombinator.com/item?id=32026606 https://news.ycombinator.com/item?id=32026606, is that this happens all the time, is not surprising at all, and is actually quite to be expected. Live and learn!
- whitepoplar 4y ago
  Do you happen to know the make and model of SSD?
  - wolfgang42 4y ago
    Just posted to the linked thread: kabdib> Let me narrow my guess: They hit 4 years, 206 days and 16 hours . . . or 40,000 hours. And that they were sold by HP or Dell, and manufactured by SanDisk. mikiem> These were made by SanDisk (SanDisk Optimus Lightning II) and the number of hours is between 39,984 and 40,032...
    - dang 4y ago
      https://news.ycombinator.com/item?id=32028511 https://news.ycombinator.com/item?id=32028511
- hoofhearted 4y ago
  I believe a more plausible scenario could be that each drive failed during the RAID rebuild and restriping process. This is a known issue in NAS systems, and Freenas always recommended running two raid arrays with 3 disks in each array for mission critical equipment. By doing so, you can lose a disk in each array and keep on trucking without any glitches. Then if you happen to kill another disk during restriping, it would failover to the second mirrored array. You could hotswap any failed disks in this setup without any downtime. The likelihood of losing 3 drives together in a server would be highly unlikely. https://www.45drives.com/community/articles/RAID-and-RAIDZ/ https://www.45drives.com/community/articles/RAID-and-RAIDZ/
  - aaaaaaaaata 4y ago
    Unless it was due to end of life, or power surge related failure. Then more than one failing simultaneously isn't so inconceivable.
  - pmoriarty 4y ago
    Ideally, there should be redundancy in servers, too.. With different hardware, on different sides of the planet, on different service providers.
    - hoofhearted 4y ago
      Correct.. Your production server with dual mirrored arrays should have an identical warm spare. If it's in the same data center, then you need a separate offsite backup in case of a worst case disaster such as a tornado, fire, or nuclear strike.
- loxias 4y ago
  > So the leading hypothesis seems to be that perhaps the SSDs were from the same manufacturing batch and shared some defect. Really sorry that you had to learn the hard way, but this is unfortunately common knowledge :/ Way back (2004) when I was shadowing-eventually-replacing a mentor that handled infrastructure for a major institution, he gave me a rule I took to heart from then forward: Always diversify. Diversify across manufacturer, diversify across make/model, hell, if it's super important, diversify across _technology stacks_ if you can. It was policy within our (infrastructure) group that /any/ new server or service must be build-able from at least 2 different sources of components before going live, and for mission critical things, 3 is better. Anything "production" had to be multihomed if it connects to the internet. Need to build a new storage server service? Get a Supermicro board _and_ a Tyan (or buy an assortment of Dell & IBM), then populate both with an assortment of drives picked randomly across 3 manufacturers, with purchases spread out across time (we used 3months) as well as resellers. Any RAID array with more than 4 drives had to include a hot spare. For even more peace of mind, add a crappy desktop PC with a ton of huge external drives and periodically sync to that. He also taught me that it's not done until you do a few live "disaster tests" (yanking drives out of fully powered up servers, during heavy IO. Brutally ripping power cables out, quickly plugging it back in, then yanking it out again once you hear the machine doing something, then plug back in...), without giving anyone advance notice. Then, and only then, is a service "done". I thought "Wow, $MENTOR is really into overkill!!" at the time, but he was right. I credit his "rules for building infrastructure" for having a zero loss track record when it comes to infra I maintain, my whole life.
  - erik_seaberg 4y ago
    Didn’t Intel grant AMD some kind of license because the US government refused to buy x86 CPU models that only have one source?
    - trasz 4y ago
      I believe it was IBM, not the government.
  - dang 4y ago
- ksec 4y ago
  I mean to be fair, had this not been a SSD defect, the probability of four of them dying at the same time ( or in extreme close proximity of a few hours ) is indeed very unlikely. And choosing different SSD vendor would have prevented this happening even if one did have counter overflow. Or in this case the 2nd Server could have been a different Model or vendor.
  - dang 4y ago
    I'm not sure if 4 failed for that reason, or if only 2 failed for that reason and then the attempts to restore them from the RAID array failed for a different reason. 2 failures within a few hours is unlikely enough already though, unless there was a common variable (which there clearly was).

dang 4y ago

It was an SSD that failed in each case, and in a similar way (e.g. both were in RAID arrays but neither could be rebuilt from the array - but I am over my skis in reporting this, as I barely know what that means). The disks were in two physically separate servers that were not connected to each other. I believe, however, that they were of similar make and model. So the leading hypothesis seems to be that perhaps the SSDs were from the same manufacturing batch and shared some defect. In other words, our servers were inbred! Which makes me want to link to the song 'Second Cousin' by Flamin' Groovies. The HN hindsight consensus, to judge by the replies to https://news.ycombinator.com/item?id=32026606 https://news.ycombinator.com/item?id=32026606, is that this happens all the time, is not surprising at all, and is actually quite to be expected. Live and learn!

whitepoplar 4y ago
Do you happen to know the make and model of SSD?
- wolfgang42 4y ago
  Just posted to the linked thread: kabdib> Let me narrow my guess: They hit 4 years, 206 days and 16 hours . . . or 40,000 hours. And that they were sold by HP or Dell, and manufactured by SanDisk. mikiem> These were made by SanDisk (SanDisk Optimus Lightning II) and the number of hours is between 39,984 and 40,032...
  - dang 4y ago
    https://news.ycombinator.com/item?id=32028511 https://news.ycombinator.com/item?id=32028511
hoofhearted 4y ago
I believe a more plausible scenario could be that each drive failed during the RAID rebuild and restriping process. This is a known issue in NAS systems, and Freenas always recommended running two raid arrays with 3 disks in each array for mission critical equipment. By doing so, you can lose a disk in each array and keep on trucking without any glitches. Then if you happen to kill another disk during restriping, it would failover to the second mirrored array. You could hotswap any failed disks in this setup without any downtime. The likelihood of losing 3 drives together in a server would be highly unlikely. https://www.45drives.com/community/articles/RAID-and-RAIDZ/ https://www.45drives.com/community/articles/RAID-and-RAIDZ/
- aaaaaaaaata 4y ago
  Unless it was due to end of life, or power surge related failure. Then more than one failing simultaneously isn't so inconceivable.
- pmoriarty 4y ago
  Ideally, there should be redundancy in servers, too.. With different hardware, on different sides of the planet, on different service providers.
  - hoofhearted 4y ago
    Correct.. Your production server with dual mirrored arrays should have an identical warm spare. If it's in the same data center, then you need a separate offsite backup in case of a worst case disaster such as a tornado, fire, or nuclear strike.
loxias 4y ago
> So the leading hypothesis seems to be that perhaps the SSDs were from the same manufacturing batch and shared some defect. Really sorry that you had to learn the hard way, but this is unfortunately common knowledge :/ Way back (2004) when I was shadowing-eventually-replacing a mentor that handled infrastructure for a major institution, he gave me a rule I took to heart from then forward: Always diversify. Diversify across manufacturer, diversify across make/model, hell, if it's super important, diversify across _technology stacks_ if you can. It was policy within our (infrastructure) group that /any/ new server or service must be build-able from at least 2 different sources of components before going live, and for mission critical things, 3 is better. Anything "production" had to be multihomed if it connects to the internet. Need to build a new storage server service? Get a Supermicro board _and_ a Tyan (or buy an assortment of Dell & IBM), then populate both with an assortment of drives picked randomly across 3 manufacturers, with purchases spread out across time (we used 3months) as well as resellers. Any RAID array with more than 4 drives had to include a hot spare. For even more peace of mind, add a crappy desktop PC with a ton of huge external drives and periodically sync to that. He also taught me that it's not done until you do a few live "disaster tests" (yanking drives out of fully powered up servers, during heavy IO. Brutally ripping power cables out, quickly plugging it back in, then yanking it out again once you hear the machine doing something, then plug back in...), without giving anyone advance notice. Then, and only then, is a service "done". I thought "Wow, $MENTOR is really into overkill!!" at the time, but he was right. I credit his "rules for building infrastructure" for having a zero loss track record when it comes to infra I maintain, my whole life.
- erik_seaberg 4y ago
  Didn’t Intel grant AMD some kind of license because the US government refused to buy x86 CPU models that only have one source?
  - trasz 4y ago
    I believe it was IBM, not the government.
- dang 4y ago
ksec 4y ago
I mean to be fair, had this not been a SSD defect, the probability of four of them dying at the same time ( or in extreme close proximity of a few hours ) is indeed very unlikely. And choosing different SSD vendor would have prevented this happening even if one did have counter overflow. Or in this case the 2nd Server could have been a different Model or vendor.
- dang 4y ago
  I'm not sure if 4 failed for that reason, or if only 2 failed for that reason and then the attempts to restore them from the RAID array failed for a different reason. 2 failures within a few hours is unlikely enough already though, unless there was a common variable (which there clearly was).