6 ms·

the chance of two SSD's failing at the same time under normal circumstances is extremely slim. So this might actually be a good cause of this incident.

MBCook 4y ago
Especially since one pair was a nearly unused backup server that had a totally different use profile.
Group_B 4y ago
If both SSD's are from the same lot number and one fails, the chances of the second failing go up by a high amount. Both failing at the same time though is extremely rare.
- oogali 4y ago
  We (as an industry) went through this bad batch madness with the IBM DeskStar 75GXP hard drives, which were affectionately referred to as "IBM Deathstar"[1]. It's rare, but it's not _that_ rare. You have to make the effort to understand why it failed. I had a situation where I deployed Toshiba SLC SSDs (that were purchased over the course of several months) and a piece of software that synchronized to disk frequently, resulting in about 1GB of writes per hour. After ~11 months in service, most of the drives died in the same 4 week period. We were astounded that everything failed so close to each other, including instances where both drives in a RAID 1 set were toast. We did extensive troubleshooting between the failed servers and the remaining servers and figured out that write volume (by proxy of in-service date) was the one predictor of failure. Shortly thereafter, wear leveling and TRIM became things we sought out mentions of when spec'ing out hardware. 1: https://en.wikipedia.org/wiki/Deskstar https://en.wikipedia.org/wiki/Deskstar
  - jaclaz 4y ago
    There was also more recently the case of the Seagate 7200.11, see my previous comment: https://news.ycombinator.com/item?id=32053477 https://news.ycombinator.com/item?id=32053477
  - dredmorbius 4y ago
    Surprisingly few HN submissions under either term: IBM Deskstar: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=ibm%20deskstar%20&sort=byPopularity&type=story https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que... IBM Deathstar: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=ibm%20deathstar%20&sort=byPopularity&type=story https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
    - xyzzy_plugh 4y ago
      Why would there be submissions? These drives predate HN.
      - dredmorbius 4y ago
        HN occasionally discusses issues pre-dating itself.
        dredmorbius 4y ago
        ... and the lifetime (deathtime?) award for the Deathstar only predated HN by a few months: May 26, 2006: https://www.pcworld.com/article/535838/worst_products_ever.html https://www.pcworld.com/article/535838/worst_products_ever.h... October 9, 2006: https://news.ycombinator.com/item?id=1 https://news.ycombinator.com/item?id=1 Memory would still have been reasonably green.
      - deleted 4y ago
        [deleted]
    - solardev 4y ago
      Back then people still used Slashdot: Wondering if it's real... https://m.slashdot.org/story/20680 https://m.slashdot.org/story/20680 Years later, it's a widespread phenomenon: https://m.slashdot.org/story/43312 https://m.slashdot.org/story/43312 It was affectionately called the "click of death".
      - 4y ago
  - mst 4y ago
    The deathstars were fantastic, they almost always failed on the outer edges of the platters. So if you only formatted them (filesystem wise) out to capacity-2Gb they were a really cheap option at the time.
    - lostlogin 4y ago
      That’s a very different definition of fantastic than the one I use.
      - mst 4y ago
        Once I'd worked it out - which was after the problems were public and therefore the price had utterly cratered - they were by far the cheapest storage per Gb available at the time (think "by a factor of two"). I would not have let a normal business user near one, but the developers I was supporting were most pleased about their larger than expected scratch disks for test databases and intermediate compilation artifacts. Everything breaks. Things that at least break predictably make me happier than the alternative.
  - rasz 4y ago
    It usually takes something really bad happening before things get better. Absolute best mechanical drives available until quite recently can be traced back to Deatstar. Deskstar 7K4000 were absolute best in class. https://en.wikipedia.org/wiki/Deskstar https://en.wikipedia.org/wiki/Deskstar Hitachi bought IBM hard drive business in 2003 for $2B. Sadly Its now owned by WD.
    - jasomill 4y ago
      Of the ~20 assorted Deskstar and related Ultrastar drives of that vintage I've had in regular use over the past decade, exactly one has failed...because I dropped it. HGST SAS SSDs (the ones that pair Intel NAND and Hitachi SAS controllers) have also been reliable performers in my home office experience, even without (RAID controller support for) UNMAP (SCSI "TRIM"). Incidentally, these now appear to be selling on eBay for more than I paid for them "lightly used" several years ago.
dredmorbius 4y ago
It seems more likely it was four drives (though dang and Mike both refer to "two" in the earlier thread). Both primary and failover servers had RAID arrays. I suspect RAID 10 (striped mirror), which would mean two drives would have to fail to take down a single server. Four drives of the same manufacturer spec and batch would do that.
- Silhouette 4y ago
  This is what always worries me about our home server. It's running ZFS with multiple redundant drives but the supplier refused (when I explicitly asked) to supply it with disks known to be from different batches claiming that the odds of multiple failures close together were negligible. Obviously we have backups as well but the time and cost to restore a full server from online backups can be significant.
  - dredmorbius 4y ago
    Within an earlier thread "Tell HN: HN Moved from M5 to AWS", there's an excellent comment by loxias about risk diversification across multiple factors. Well worth reading: https://news.ycombinator.com/item?id=32031655 https://news.ycombinator.com/item?id=32031655 I've increasingly come to view systems operations / SRE as a risk management exercise, where the goal is to reduce the odds of a catastrophic failure. Total system outage is one level, unrecoverable total system outage is even worse. Having multiple redundant backups / storage systems, in different locations, with different vendor hardware / stacks, all helps reduce risk of a single-factor outage. Though complexity risk is its own issue.
    - kqr 4y ago
      > I've increasingly come to view systems operations / SRE as a risk management exercise, where the goal is to reduce the odds of a catastrophic failure. s/reduce/find an appropriate level for/ It's a common misconception that risk management and risk reduction are synonyms. Risk management is about finding the right level of risk given external factors. Sometimes that means maintaining the current level of risk or even increasing it in favour of other properties.
pca006132 4y ago
perhaps they are the same model. iirc people recommend not to use the same model of hardware to provide redundancy.
jeffreygoesto 4y ago
True. But it's more about the probability of things being "normal", isn't it? I had multiple Evo970 fail within a very short time. Turns out to be a systematic problem of drives produced in one specific month. Just how much difference is enough to be safe is the price question...
pmlnr 4y ago
Heh, no. We had a fleet of HPE Cloudline (CL3100) failing at the same time because the SSDs exhausted the writes.
somat 4y ago
The way I always put it is, you have identical drives, with identical usage, powered for identical times. and you are still surprised when a second drive fails under the high stress environment of rebuilding after the first drive fails.

MBCook 4y ago

Especially since one pair was a nearly unused backup server that had a totally different use profile.

Group_B 4y ago

If both SSD's are from the same lot number and one fails, the chances of the second failing go up by a high amount. Both failing at the same time though is extremely rare.

oogali 4y ago
We (as an industry) went through this bad batch madness with the IBM DeskStar 75GXP hard drives, which were affectionately referred to as "IBM Deathstar"[1]. It's rare, but it's not _that_ rare. You have to make the effort to understand why it failed. I had a situation where I deployed Toshiba SLC SSDs (that were purchased over the course of several months) and a piece of software that synchronized to disk frequently, resulting in about 1GB of writes per hour. After ~11 months in service, most of the drives died in the same 4 week period. We were astounded that everything failed so close to each other, including instances where both drives in a RAID 1 set were toast. We did extensive troubleshooting between the failed servers and the remaining servers and figured out that write volume (by proxy of in-service date) was the one predictor of failure. Shortly thereafter, wear leveling and TRIM became things we sought out mentions of when spec'ing out hardware. 1: https://en.wikipedia.org/wiki/Deskstar https://en.wikipedia.org/wiki/Deskstar
- jaclaz 4y ago
  There was also more recently the case of the Seagate 7200.11, see my previous comment: https://news.ycombinator.com/item?id=32053477 https://news.ycombinator.com/item?id=32053477
- dredmorbius 4y ago
  Surprisingly few HN submissions under either term: IBM Deskstar: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=ibm%20deskstar%20&sort=byPopularity&type=story https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que... IBM Deathstar: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=ibm%20deathstar%20&sort=byPopularity&type=story https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
  - xyzzy_plugh 4y ago
    Why would there be submissions? These drives predate HN.
    - dredmorbius 4y ago
      HN occasionally discusses issues pre-dating itself.
      - dredmorbius 4y ago
        ... and the lifetime (deathtime?) award for the Deathstar only predated HN by a few months: May 26, 2006: https://www.pcworld.com/article/535838/worst_products_ever.html https://www.pcworld.com/article/535838/worst_products_ever.h... October 9, 2006: https://news.ycombinator.com/item?id=1 https://news.ycombinator.com/item?id=1 Memory would still have been reasonably green.
    - deleted 4y ago
      [deleted]
  - solardev 4y ago
    Back then people still used Slashdot: Wondering if it's real... https://m.slashdot.org/story/20680 https://m.slashdot.org/story/20680 Years later, it's a widespread phenomenon: https://m.slashdot.org/story/43312 https://m.slashdot.org/story/43312 It was affectionately called the "click of death".
    - 4y ago
- mst 4y ago
  The deathstars were fantastic, they almost always failed on the outer edges of the platters. So if you only formatted them (filesystem wise) out to capacity-2Gb they were a really cheap option at the time.
  - lostlogin 4y ago
    That’s a very different definition of fantastic than the one I use.
    - mst 4y ago
      Once I'd worked it out - which was after the problems were public and therefore the price had utterly cratered - they were by far the cheapest storage per Gb available at the time (think "by a factor of two"). I would not have let a normal business user near one, but the developers I was supporting were most pleased about their larger than expected scratch disks for test databases and intermediate compilation artifacts. Everything breaks. Things that at least break predictably make me happier than the alternative.
- rasz 4y ago
  It usually takes something really bad happening before things get better. Absolute best mechanical drives available until quite recently can be traced back to Deatstar. Deskstar 7K4000 were absolute best in class. https://en.wikipedia.org/wiki/Deskstar https://en.wikipedia.org/wiki/Deskstar Hitachi bought IBM hard drive business in 2003 for $2B. Sadly Its now owned by WD.
  - jasomill 4y ago
    Of the ~20 assorted Deskstar and related Ultrastar drives of that vintage I've had in regular use over the past decade, exactly one has failed...because I dropped it. HGST SAS SSDs (the ones that pair Intel NAND and Hitachi SAS controllers) have also been reliable performers in my home office experience, even without (RAID controller support for) UNMAP (SCSI "TRIM"). Incidentally, these now appear to be selling on eBay for more than I paid for them "lightly used" several years ago.

dredmorbius 4y ago

It seems more likely it was four drives (though dang and Mike both refer to "two" in the earlier thread). Both primary and failover servers had RAID arrays. I suspect RAID 10 (striped mirror), which would mean two drives would have to fail to take down a single server. Four drives of the same manufacturer spec and batch would do that.

Silhouette 4y ago
This is what always worries me about our home server. It's running ZFS with multiple redundant drives but the supplier refused (when I explicitly asked) to supply it with disks known to be from different batches claiming that the odds of multiple failures close together were negligible. Obviously we have backups as well but the time and cost to restore a full server from online backups can be significant.
- dredmorbius 4y ago
  Within an earlier thread "Tell HN: HN Moved from M5 to AWS", there's an excellent comment by loxias about risk diversification across multiple factors. Well worth reading: https://news.ycombinator.com/item?id=32031655 https://news.ycombinator.com/item?id=32031655 I've increasingly come to view systems operations / SRE as a risk management exercise, where the goal is to reduce the odds of a catastrophic failure. Total system outage is one level, unrecoverable total system outage is even worse. Having multiple redundant backups / storage systems, in different locations, with different vendor hardware / stacks, all helps reduce risk of a single-factor outage. Though complexity risk is its own issue.
  - kqr 4y ago
    > I've increasingly come to view systems operations / SRE as a risk management exercise, where the goal is to reduce the odds of a catastrophic failure. s/reduce/find an appropriate level for/ It's a common misconception that risk management and risk reduction are synonyms. Risk management is about finding the right level of risk given external factors. Sometimes that means maintaining the current level of risk or even increasing it in favour of other properties.

pca006132 4y ago

perhaps they are the same model. iirc people recommend not to use the same model of hardware to provide redundancy.

jeffreygoesto 4y ago

True. But it's more about the probability of things being "normal", isn't it? I had multiple Evo970 fail within a very short time. Turns out to be a systematic problem of drives produced in one specific month. Just how much difference is enough to be safe is the price question...

pmlnr 4y ago

Heh, no. We had a fleet of HPE Cloudline (CL3100) failing at the same time because the SSDs exhausted the writes.

somat 4y ago

The way I always put it is, you have identical drives, with identical usage, powered for identical times. and you are still surprised when a second drive fails under the high stress environment of rebuilding after the first drive fails.