6 ms·

SMART is a fantastic exercise in sensitivity and specificity. As backblaze is showing with this data, SMART stats have poor sensitivity, but what's much worse f

by mjb 10y ago

SMART is a fantastic exercise in sensitivity and specificity. As backblaze is showing with this data, SMART stats have poor sensitivity, but what's much worse for those who run big fleets of drives is their poor specificity. Lots of healthy drives are reported unhealthy by SMART. If I'm running a gold-plated database server, that doesn't matter. A couple of extra planned drive replacements is a small price to pay for avoiding unplanned failures. If I'm running a huge drive cluster, it's much, much more expensive. Take Backblaze's 0.01% for a group of four failures. That's replacing an extra 100 drives per million, at random, and only getting the benefit of correctly predicting failures 10.4% of the time. This is great data to have.

andy4blaze 10y ago
Nice analysis.
deleted 10y ago
[deleted]
Retric 10y ago
Thresholds are often useful with these kinds of stats. Aka a drive moving one sector might mean nothing, but moving 30 in a week could be great predictor. Further they only have 70k drives across a range of product lines so what predicts drive X failing very well might say little about drive Y. PS: Rememebr all RAM gets bit flit errors over time. Which is one of the reasons rebooting is often so useful, but also means one off errors are often meaningless.
- sn 10y ago
  ECC ram will either correct the error or can raise a non maskable interrupt if it can't be corrected.
  - Retric 10y ago
    ECC vastly lowers, but does not remove this problem. You can for example get bit flip errors while doing the checksum. As a personal user it's a non issue but scale things to ~70k devices and you get ~1.7 million device hours per day.
    - sn 10y ago
      Fair enough. Your original comment did not make this distinction. I dug and found https://www.fiala.me/pubs/papers/sc12-redmpi.pdf https://www.fiala.me/pubs/papers/sc12-redmpi.pdf the title of which is "Detection and Correction of Silent Data Corruption for Large-Scale High-Performance Computing." They state that for a cray there was a double bit flip about 1x/day for 75k modules. To (probably incorrectly) extrapolate, if you have a server with 16 modules that would be equivalent to a single double failure about once every 13 years.
devonkim 10y ago
SMART stats are only one source of data though that can be predictors of drive failures. Hard drives can experience latency spikes and erratic performance compared to peers before SMART records problems, for example. This is part of why predictive monitoring can be so difficult - the data you didn't know you needed is probably among the ones you didn't ingest into your metrics, and you can't dump literally everything in /proc every other second to your metrics system without sacrificing some CPU or network bandwidth either.
joosters 10y ago
It's unfair to blame SMART stats for this. On their own, SMART does not declare a drive 'healthy' or 'unhealthy', it's up to the user to decide. Each stat on its own seems very specific, e.g. Number of reallocated sectors. I'm not sure how you think this is 'poor specificity', the stats are measuring one thing exactly, you can't get more specific than that!
- fnj 10y ago
  Actually, it does declare. Direct quote from smartctl output: "SMART overall-health self-assessment test result: PASSED"

by mjb 10y ago

andy4blaze 10y ago

Nice analysis.

deleted 10y ago

[deleted]

Retric 10y ago

Thresholds are often useful with these kinds of stats. Aka a drive moving one sector might mean nothing, but moving 30 in a week could be great predictor. Further they only have 70k drives across a range of product lines so what predicts drive X failing very well might say little about drive Y. PS: Rememebr all RAM gets bit flit errors over time. Which is one of the reasons rebooting is often so useful, but also means one off errors are often meaningless.

sn 10y ago
ECC ram will either correct the error or can raise a non maskable interrupt if it can't be corrected.
- Retric 10y ago
  ECC vastly lowers, but does not remove this problem. You can for example get bit flip errors while doing the checksum. As a personal user it's a non issue but scale things to ~70k devices and you get ~1.7 million device hours per day.
  - sn 10y ago
    Fair enough. Your original comment did not make this distinction. I dug and found https://www.fiala.me/pubs/papers/sc12-redmpi.pdf https://www.fiala.me/pubs/papers/sc12-redmpi.pdf the title of which is "Detection and Correction of Silent Data Corruption for Large-Scale High-Performance Computing." They state that for a cray there was a double bit flip about 1x/day for 75k modules. To (probably incorrectly) extrapolate, if you have a server with 16 modules that would be equivalent to a single double failure about once every 13 years.

devonkim 10y ago

SMART stats are only one source of data though that can be predictors of drive failures. Hard drives can experience latency spikes and erratic performance compared to peers before SMART records problems, for example. This is part of why predictive monitoring can be so difficult - the data you didn't know you needed is probably among the ones you didn't ingest into your metrics, and you can't dump literally everything in /proc every other second to your metrics system without sacrificing some CPU or network bandwidth either.

joosters 10y ago

It's unfair to blame SMART stats for this. On their own, SMART does not declare a drive 'healthy' or 'unhealthy', it's up to the user to decide. Each stat on its own seems very specific, e.g. Number of reallocated sectors. I'm not sure how you think this is 'poor specificity', the stats are measuring one thing exactly, you can't get more specific than that!

fnj 10y ago
Actually, it does declare. Direct quote from smartctl output: "SMART overall-health self-assessment test result: PASSED"