8 ms·

Predicting Hard Drive Failure with Machine Learning

237 points by binwiederhier 6y ago

ploika 6y ago
Surprised that survival analysis wasn't mentioned anywhere. Cox regression would have been my go-to approach.
- wcrossbow 6y ago
  I'm currently using a non-linear version of it, DeepSURV [*], implemented in pycox for a predictive maintenance job. They are much more informative than a binary label and give space to make a business decicision as to how close to EOL you want to take care of the asset. [*] The underlying neural networks aren't deep at all.
- conjectures 6y ago
  I respect the fact that you can be surprised by that. But even rewinding a bit: where are the data plotted, e.g. time vs failures?
jaas 6y ago
It's great that they took the time to write this up even though the conclusion is that it doesn't work. Too much valuable information is lost because people don't value negative results enough.
bryanculver 6y ago
The article is fascinating but one thing I am finding very common on these engineering blogs is no easy path back to the company or service. Understandably the logo at the top goes back to the blog homepage and the copyright at the bottom is just static text. I could search or visit one of the social pages linked to in the footer but it seem like an obvious link one would want to have.
- whoisburbansky 6y ago
  Hitting the “Join The Team” button up top takes you back to the main site, which seems reasonable since in most cases a technical blog is meant to be a recruitment funnel more than a general public-facing PR document, no?
  - bryanculver 6y ago
    I see that now. I wouldn't disagree, but if you're a CTO for example you might have interest in the services more than joining the team.
- binwiederhier 6y ago
  Thanks for bringing this up. We'll add a link to the main Datto website shortly. :-)
whoisburbansky 6y ago
Wow, this is a surprisingly sober look at using ML for this particular problem. I really appreciate how the author walks through every step and the rationale behind it, spending as much time on explaining the dataset and cleaning it up as she does on, say, the model they end up using. It’s a bit of a breath of fresh air to see a post like this where someone clearly puts the work in to try and solve a given problem with ML and is able to articulate the failure modes of the approach so well.
- mlthoughts2018 6y ago
  This is very standard in industry (as opposed to academia) and highlights a lot of the reasons why you need ML specialists operating these models, and why you will get garbage in garbage out if you try having non-specialist engineers build, evaluate & operate these types of systems. For example, in this post the engineer does not adequately handle class imbalance at all, and reaches for a hugely complicated model framework (XGBoost) without even trying something like bias-corrected logistic regression first. I think this post is just an example of an infra engineer thinking “ML is easy” and when their quick attempt doesn’t work, then just bailing and saying “ML is overhyped and doesn’t work for my problem.”
  - causalmodels 6y ago
    This is not the correct modeling approach. All hard drives will fail given enough time, so labeling the failed hard drives as the positive class will bias your results. Stuff like this really should be handled using survival analysis.
    - mlthoughts2018 6y ago
      In fact it screams Bayesian survival analysis, since there is so much prior knowledge both of general hard drive failure rates and SMART stats.
      - astrophysician 6y ago
jeffbee 6y ago
This is consistent with prior work showing that SMART has no predictive value. Failure Trends in a Large Disk Drive Population, FAST '07 https://static.googleusercontent.com/media/research.google.com/en//archive/disk_failures.pdf https://static.googleusercontent.com/media/research.google.c...
- toast0 6y ago
  Didn't read your linked study, the linked blog post threw out the actual data though. Not surprising you get nothing of use when you start with: > Each attribute also has a feature named raw_value, but this is discarded due to inconsistent reporting standards between drive manufacturers. Sure, raw_value is inconsistent between manufacturers and sometimes models, but it's the most real data. Edit to add: reread the Google survey, which says more or less bad sector counts were indicative of failure in about half the cases; in 2007. One thing to note is drives made now may perform differently than those in the 2007 survey. Different firmware, different recording, better soldering (200x is full of poor soldering because of RoHS mandating lead free solder in most applications, and lack of experience with lead free solder leading to failing connections). I found, in 2013ish, with a couple thousand WD enterprise branded drives that just looking at the sum of all the sector counts and thresholding that was enough to catch drives that would soon fail. If your fleet is large enough, and you have things setup so failed disks is not a disaster (which you should!), it's pretty easy to run the collection and figure out your thresholds. IIRC, we would replace at 100+ damaged sectors, or if growth was rapid. Some drives in service before collecting smart data did seem to be working fine at around 1000 damaged sectors though, so you could make a case for a higher threshold. SSDs on the otherhand seemed to fail spectacularly (no longer visible from the OS), with no prefailure symptoms that I could see. Thankfully the failure rates were much lower than spinning disks, but a lot more of a pain.
  - heipei 6y ago
    Hmm, one could encode manufacturer and maybe even model as a categorical or one-hot encoded variable. Then again, the small number of failed drives in their dataset would probably make this hard.
- 6y ago
jmpman 6y ago
The author doesn’t explicitly call out the classic reliability engineer’s concept of a bathtub failure model. https://en.m.wikipedia.org/wiki/Bathtub_curve https://en.m.wikipedia.org/wiki/Bathtub_curve Thermal abuse (running at a high temp for many hours), duty cycle, iop rate (high head acceleration versus head dwell) all impact the lifetime. Early life media issues are predictive of failure and can be detected through stress testing with various seek ranges. It’s not as simple as a SMART metric, but did appear to have predictive value. The one experiment I ran discovered a set of drives which ultimately the manufacturer admitted to a since corrected process problem. The above factors are different on each drive family and can’t normally be generalized.
NikolaeVarius 6y ago
I'm not surprised by the conclusions, I've never seen anyone claim SMART has ever been useful, unless the HDD is already known to have issues that are already noticeable without SMART, SMART just makes it possible to diagnose the actual problem. Though, of course this is limited by my own anecdotal research and personal experience. What I wonder than, is if sensitive isolated microphones could be tried for this purpose? We already know that sounds (ie yelling at the drive) can vibrate the platter enough to cause performance degradation. If there were internal mics in each HDD recording sound as the HDD spins and correlate that to HDD activity, could that be correlated with failure rate?
- toast0 6y ago
  If you have spinning drives, monitor the (raw) sector count stats. Add together reallocated sectors, uncorrectable and offline uncorrectable. If you have good operations, replace when convenient after it hits 100. If you have poor operations (like in home use, with no backups and only ocassional SMART checks) replace if it hits 10 and try to run a full SMART surface scan before using a new drive. For SSDs, good luck, I haven't seen prefailure indicators.
  - kdeldycke 6y ago
    Which attributes IDs are you referring to? Current_Pending_Sector + Reallocated_Sector_Ct + Offline_Uncorrectable + ???
- tinus_hn 6y ago
  I have had drives tell me something was wrong before they developed bad sectors. It does take a lot of monitoring though and probably leads one to replace drives earlier than might be necessary.
- dehrmann 6y ago
  I've also never found anything meaningful out of SMART at home lab scale. I do know the way vendors report it is a shitshow, so I wouldn't suspect it'd be great training data. What I'd be really curious to try is run IO benchmarks on the drive and see if there are performance issues that indicate a drive is failing.
heipei 6y ago
Great post, glad to see something written from beginning to end. One thing that surprised me though is that there wasn't a single word lost on feature selection (by looking at feature importance for correct predictions) or feature engineering by combining features or somehow else applying domain knowledge. So for example, trying to figure out if you can put different attributes in relation to each other to get a better feature (e.g. number-of-spinups / operating-time-in-hours). Personally that would be the first thing I'd try, and coincidentally it's also the most fun aspect of machine learning for me ;)
- mlthoughts2018 6y ago
  I think it’s not actually a very good ML case study for this reason. Feature engineering would be a huge part of a problem like this. Additionally, jumping to start with XGBoost is a pretty amateurish thing to do, and the very first problem to attack is class imbalance. In the bio blurb they self-describe as an infra engineer who also enjoys data science. In some sense I really don’t like to see that. The quality of the data science / ML work in this is actually quite bad, but people use these blog posts as resume padders to try to jump into ML jobs without ever having any real experience or training. I think it’s a bad thing because it devalues the importance of real statistical computing skills, which are very hard to develop through many years of education and experience - absolutely not the sort of thing you can get by dabbling in some Python packages on the weekend to do a little project like this. The amount of waste I see from companies trying to avoid paying higher wages and avoid team structures that facilitate productivity of statistics experts is staggering - with all kinds of hacked up scripts and notebooks stitched together without proper backing statistical understanding, making ML engineers manage their own devops, and just ignoring base statistical questions. For this drive problem for example, I expect to see a progression from simple models, each of which should address class imbalance as a first order concern. I expect to see how Bayesian modeling can help and how simple lifetime survivorship models can help. I expect to see a lot of feature engineering. Instead I see an infra engineer playing around with data and trying one off the shelf framework, then claiming the whole premise can’t work in production.
  - woeirua 6y ago
gerdesj 6y ago
"Only one drive manufacturer had enough failed drives in the dataset to produce a large enough subset, which we’ll refer to as Manufacturer A. Because this data treatment was so effective, and because most of the drives that had been in our fleet long enough to begin failing were already from Manufacturer A, we decided to reduce the scope of smarterCTL to only their drives." I'd have dropped Manufacturer A from the supplier list and used the A only model for the remainder of their drives. Then I'd have another go with SmarterCTL for "not A".
- rasz 6y ago
  Seagate, just call it what it is. You would not drop Seagate because their drives are >20% cheaper than competition.
- dehrmann 6y ago
  There are really only 2.5 manufacturers, though.
- sokoloff 6y ago
  If Manufacturer A represented 80% of the fleet and 70% of the failures, dropping Manufacturer A doesn't make sense.
washedup 6y ago
Just curious: how come the total observations of the confusion matrices are so different? Shouldn't the same data set be used in each trial to fully evaluate the performance differences? Maybe I missed some details in the article...
- rachelwenzel 6y ago
  Hey there! Article author here- This was written as an after-the-fact look back after a year long project and the target testing set did change a little along the way. The big change was splitting by manufacturer (and eventually by model family), which changed the number of applicable samples to compare performance on. I cut some stuff from the article for length, so I totally get how that could have been unclear.
  - washedup 6y ago
    Appreciate the explanation, thanks! Enjoyed the article.
Abishek_Muthian 6y ago
Neat ML exercise, though it wasn't the right solution for the problem. What is the right solution at consumer level though? I currently run smartmontools at regular intervals, compare the TBW with manufacturer's TBW and send a MQTT message using a script.
MaxBarraclough 6y ago
These guys offer something similar as a service: https://www.senseye.io/how-it-works https://www.senseye.io/how-it-works
moonbug 6y ago
Ugh. Statistics, people.
ryeguy_24 6y ago
It would be interesting to empirically identify the state space that leads to increased probability of failure and then seek to understand the rationale for this failure through physics. Ultimately, the ML model is picking up on some combination of factors that exposes physical weaknesses of the device.
PaulHoule 6y ago
My understanding is that SMART is not effective because it only catches (some) mechanical problems and that about half of the time it is the logic board that went bad. You can filter out some of the bad drives ahead of time, but you will still get blindsided much of the time.
- 8fingerlouie 6y ago
  Assuming you don’t have an early fail drive, it has one usable stat, “Power On Hours”. When that begins to creep towards 40,000 you will have a failed drive sooner than later. As for the rest of the counter, raw read error rate, relocated sector count and friends are also good indications that your drive might not be in the best shape. They don’t say anything about when it will fail though. I’ve had a drive rack up 20 relocated sectors in a couple of weeks, only to stay at 20 for 4 years after that. And load cycle count is only good for telling if you’re within manufacturer parameters. I frequently see old drives with Load Cycle Count 10-25 times the rated value, and they work just “fine”.
  - Uberphallus 6y ago
    Yeah, the training data is basically useless: 1. Extreme imbalance and survivorship bias. The dataset is unwieldy because of the imbalance, no matter the algorithm; and too small if you balance it. 2. I would take a daily/weekly/monthly diff between different values and time to failure, and obviously only from drives that ACTUALLY failed. All drives fail, but data from healhty drives is useless as it lacks the target value. In practice you want to predict how close they are to failing and act preemptively, tagging them as OK and KO completely misses the target (hehe) IMO. 3. Drive maker and model should be vectorized and fed into the model. Mandatory. Different manufacturers and models show different SMART behaviors, and while some stats are red flags in certain drives, they mean nothing in others. 4. This is more of a personal preference, but I like random trees to test whether the dataset is useful. They're not perfect but they give reasonably good results for most tasks, and tend not to overfit, so you can use it as overfitting benchmark vs other algorithms. If they don't either the dataset is shit or it needs some transformation, or the model needs to be more complex (but that's easy to see). Obviously this applies to classifiers and regressions, and not to many other ML tasks.
CraigJPerry 6y ago
For enriching the model, a few ideas for discriminating failure drives: Some hard drives have a vibration sensor(s) in them - i wonder if a daemon could sample throughout the day and dump a summarisation of the data when the daily SMART data dump is being generated. The drives report temperature via SMART, i wonder if anything meaningful can be extracted from the temperature differential between the HDD’s sensor and a chassis temp sensor? I wonder if more discrimination by manufacturing detail (date of mfr, which factory etc) could help? I was going to suggest drive supply voltages but those are hard to compare across chassis - calibration of voltage measuring circuitry in PSUs isn’t too hot for the millivolt differences you’d want to measure - the differences would be so small to be lost in the inaccuracies between boards. Also many PSUs lack a method to query their measurements. Fascinating study and excellent writeup. I have zero ML background but i really enjoyed this.
oli5679 6y ago
Fitting a binary classifier on failure doesn't seem quite right to me. There are some quite natural questions you can't answer unless you can predict some stats on when failure will occur, and if you have any drift in the dataset, you might just find features correlated with age (and hence having more time to fail) that don't inherently increase risk. I would recommend: (a) using survival analysis https://lifelines.readthedocs.io/en/latest/ https://lifelines.readthedocs.io/en/latest/ https://xgboost.readthedocs.io/en/latest/tutorials/aft_survival_analysis.html https://xgboost.readthedocs.io/en/latest/tutorials/aft_survi... or (b) flattening the data to be one observation per time window, and predicting failure over specific time windows. This is a good talk also. https://dataorigami.net/blogs/napkin-folding/18308867-video-presentation-on-lifelines-survival-analysis-in-python-sept-23-2014 https://dataorigami.net/blogs/napkin-folding/18308867-video-... Although I'm not sure about this case, in general the approach in the article (tree-ensemble predicting binary classier) works well for many business problems!
mleonhard 6y ago
They sample the SMART metrics once a day. Would more frequent samples would provide a better signal? For example, perhaps the read errors number increases about once a week. But in the week before a failure, it increases once a day. Or in the day before a failure, it increases once an hour. OS-level metrics could also be predictive: write latency, i/o queue length, communication errors, and checksum errors in read data.
Faint 6y ago
Maybe training with actual bottom-line utility function would have shown some use case, that is, instead of MCC you would have predicted whether sending a person to change the drive now before it fails would cost less than letting it fail and cleaning up afterwards. But if the drives are part of redundant array, it would be almost always cheapest to let them fail.. and if large part of failures are asymptomatic, you need the array anyway for critical stuff, so I suppose it's a useless exercise.
jiveturkey 6y ago
love everything coming out of datto and this is no exception
the_optimist 6y ago
I think this write-up is interesting but structurally incorrect, and everyone here is taking away the wrong conclusions. The punchline here is: > Unfortunately, once they were scaled up to production levels of use, their tradeoffs turned out to be too much to bear. These models are good—especially given how imbalanced the data for this problem is—but with too many false positives from one and too little predictive power from the other, they’re not worth betting so much money on. A practical next-step analysis would be some back-of-the-envelope economic math about the necessary detection threshold for detecting and replacing failing drives. Overall, it doesn't even make sense to even think about applying ML without understanding the cost/benefit of solving the problem. Think about Backblaze -- the drives are packed in there tightly and it's super expensive to replace one, whereas there's basically zero cost to leaving one in place. Even if you could perfectly predict which drives would fail, there's no reason you'd act on that info.

237 points by binwiederhier 6y ago

ploika 6y ago

Surprised that survival analysis wasn't mentioned anywhere. Cox regression would have been my go-to approach.

wcrossbow 6y ago
I'm currently using a non-linear version of it, DeepSURV [*], implemented in pycox for a predictive maintenance job. They are much more informative than a binary label and give space to make a business decicision as to how close to EOL you want to take care of the asset. [*] The underlying neural networks aren't deep at all.
conjectures 6y ago
I respect the fact that you can be surprised by that. But even rewinding a bit: where are the data plotted, e.g. time vs failures?

jaas 6y ago

It's great that they took the time to write this up even though the conclusion is that it doesn't work. Too much valuable information is lost because people don't value negative results enough.

bryanculver 6y ago

The article is fascinating but one thing I am finding very common on these engineering blogs is no easy path back to the company or service. Understandably the logo at the top goes back to the blog homepage and the copyright at the bottom is just static text. I could search or visit one of the social pages linked to in the footer but it seem like an obvious link one would want to have.

whoisburbansky 6y ago
Hitting the “Join The Team” button up top takes you back to the main site, which seems reasonable since in most cases a technical blog is meant to be a recruitment funnel more than a general public-facing PR document, no?
- bryanculver 6y ago
  I see that now. I wouldn't disagree, but if you're a CTO for example you might have interest in the services more than joining the team.
binwiederhier 6y ago
Thanks for bringing this up. We'll add a link to the main Datto website shortly. :-)

whoisburbansky 6y ago

Wow, this is a surprisingly sober look at using ML for this particular problem. I really appreciate how the author walks through every step and the rationale behind it, spending as much time on explaining the dataset and cleaning it up as she does on, say, the model they end up using. It’s a bit of a breath of fresh air to see a post like this where someone clearly puts the work in to try and solve a given problem with ML and is able to articulate the failure modes of the approach so well.

mlthoughts2018 6y ago
This is very standard in industry (as opposed to academia) and highlights a lot of the reasons why you need ML specialists operating these models, and why you will get garbage in garbage out if you try having non-specialist engineers build, evaluate & operate these types of systems. For example, in this post the engineer does not adequately handle class imbalance at all, and reaches for a hugely complicated model framework (XGBoost) without even trying something like bias-corrected logistic regression first. I think this post is just an example of an infra engineer thinking “ML is easy” and when their quick attempt doesn’t work, then just bailing and saying “ML is overhyped and doesn’t work for my problem.”
- causalmodels 6y ago
  This is not the correct modeling approach. All hard drives will fail given enough time, so labeling the failed hard drives as the positive class will bias your results. Stuff like this really should be handled using survival analysis.
  - mlthoughts2018 6y ago
    In fact it screams Bayesian survival analysis, since there is so much prior knowledge both of general hard drive failure rates and SMART stats.
    - astrophysician 6y ago

jeffbee 6y ago

This is consistent with prior work showing that SMART has no predictive value. Failure Trends in a Large Disk Drive Population, FAST '07 https://static.googleusercontent.com/media/research.google.com/en//archive/disk_failures.pdf https://static.googleusercontent.com/media/research.google.c...

toast0 6y ago
Didn't read your linked study, the linked blog post threw out the actual data though. Not surprising you get nothing of use when you start with: > Each attribute also has a feature named raw_value, but this is discarded due to inconsistent reporting standards between drive manufacturers. Sure, raw_value is inconsistent between manufacturers and sometimes models, but it's the most real data. Edit to add: reread the Google survey, which says more or less bad sector counts were indicative of failure in about half the cases; in 2007. One thing to note is drives made now may perform differently than those in the 2007 survey. Different firmware, different recording, better soldering (200x is full of poor soldering because of RoHS mandating lead free solder in most applications, and lack of experience with lead free solder leading to failing connections). I found, in 2013ish, with a couple thousand WD enterprise branded drives that just looking at the sum of all the sector counts and thresholding that was enough to catch drives that would soon fail. If your fleet is large enough, and you have things setup so failed disks is not a disaster (which you should!), it's pretty easy to run the collection and figure out your thresholds. IIRC, we would replace at 100+ damaged sectors, or if growth was rapid. Some drives in service before collecting smart data did seem to be working fine at around 1000 damaged sectors though, so you could make a case for a higher threshold. SSDs on the otherhand seemed to fail spectacularly (no longer visible from the OS), with no prefailure symptoms that I could see. Thankfully the failure rates were much lower than spinning disks, but a lot more of a pain.
- heipei 6y ago
  Hmm, one could encode manufacturer and maybe even model as a categorical or one-hot encoded variable. Then again, the small number of failed drives in their dataset would probably make this hard.
6y ago

jmpman 6y ago

The author doesn’t explicitly call out the classic reliability engineer’s concept of a bathtub failure model. https://en.m.wikipedia.org/wiki/Bathtub_curve https://en.m.wikipedia.org/wiki/Bathtub_curve Thermal abuse (running at a high temp for many hours), duty cycle, iop rate (high head acceleration versus head dwell) all impact the lifetime. Early life media issues are predictive of failure and can be detected through stress testing with various seek ranges. It’s not as simple as a SMART metric, but did appear to have predictive value. The one experiment I ran discovered a set of drives which ultimately the manufacturer admitted to a since corrected process problem. The above factors are different on each drive family and can’t normally be generalized.

NikolaeVarius 6y ago

I'm not surprised by the conclusions, I've never seen anyone claim SMART has ever been useful, unless the HDD is already known to have issues that are already noticeable without SMART, SMART just makes it possible to diagnose the actual problem. Though, of course this is limited by my own anecdotal research and personal experience. What I wonder than, is if sensitive isolated microphones could be tried for this purpose? We already know that sounds (ie yelling at the drive) can vibrate the platter enough to cause performance degradation. If there were internal mics in each HDD recording sound as the HDD spins and correlate that to HDD activity, could that be correlated with failure rate?

toast0 6y ago
If you have spinning drives, monitor the (raw) sector count stats. Add together reallocated sectors, uncorrectable and offline uncorrectable. If you have good operations, replace when convenient after it hits 100. If you have poor operations (like in home use, with no backups and only ocassional SMART checks) replace if it hits 10 and try to run a full SMART surface scan before using a new drive. For SSDs, good luck, I haven't seen prefailure indicators.
- kdeldycke 6y ago
  Which attributes IDs are you referring to? Current_Pending_Sector + Reallocated_Sector_Ct + Offline_Uncorrectable + ???
tinus_hn 6y ago
I have had drives tell me something was wrong before they developed bad sectors. It does take a lot of monitoring though and probably leads one to replace drives earlier than might be necessary.
dehrmann 6y ago
I've also never found anything meaningful out of SMART at home lab scale. I do know the way vendors report it is a shitshow, so I wouldn't suspect it'd be great training data. What I'd be really curious to try is run IO benchmarks on the drive and see if there are performance issues that indicate a drive is failing.

heipei 6y ago

Great post, glad to see something written from beginning to end. One thing that surprised me though is that there wasn't a single word lost on feature selection (by looking at feature importance for correct predictions) or feature engineering by combining features or somehow else applying domain knowledge. So for example, trying to figure out if you can put different attributes in relation to each other to get a better feature (e.g. number-of-spinups / operating-time-in-hours). Personally that would be the first thing I'd try, and coincidentally it's also the most fun aspect of machine learning for me ;)

mlthoughts2018 6y ago
I think it’s not actually a very good ML case study for this reason. Feature engineering would be a huge part of a problem like this. Additionally, jumping to start with XGBoost is a pretty amateurish thing to do, and the very first problem to attack is class imbalance. In the bio blurb they self-describe as an infra engineer who also enjoys data science. In some sense I really don’t like to see that. The quality of the data science / ML work in this is actually quite bad, but people use these blog posts as resume padders to try to jump into ML jobs without ever having any real experience or training. I think it’s a bad thing because it devalues the importance of real statistical computing skills, which are very hard to develop through many years of education and experience - absolutely not the sort of thing you can get by dabbling in some Python packages on the weekend to do a little project like this. The amount of waste I see from companies trying to avoid paying higher wages and avoid team structures that facilitate productivity of statistics experts is staggering - with all kinds of hacked up scripts and notebooks stitched together without proper backing statistical understanding, making ML engineers manage their own devops, and just ignoring base statistical questions. For this drive problem for example, I expect to see a progression from simple models, each of which should address class imbalance as a first order concern. I expect to see how Bayesian modeling can help and how simple lifetime survivorship models can help. I expect to see a lot of feature engineering. Instead I see an infra engineer playing around with data and trying one off the shelf framework, then claiming the whole premise can’t work in production.
- woeirua 6y ago

gerdesj 6y ago

"Only one drive manufacturer had enough failed drives in the dataset to produce a large enough subset, which we’ll refer to as Manufacturer A. Because this data treatment was so effective, and because most of the drives that had been in our fleet long enough to begin failing were already from Manufacturer A, we decided to reduce the scope of smarterCTL to only their drives." I'd have dropped Manufacturer A from the supplier list and used the A only model for the remainder of their drives. Then I'd have another go with SmarterCTL for "not A".

rasz 6y ago
Seagate, just call it what it is. You would not drop Seagate because their drives are >20% cheaper than competition.
dehrmann 6y ago
There are really only 2.5 manufacturers, though.
sokoloff 6y ago
If Manufacturer A represented 80% of the fleet and 70% of the failures, dropping Manufacturer A doesn't make sense.

washedup 6y ago

Just curious: how come the total observations of the confusion matrices are so different? Shouldn't the same data set be used in each trial to fully evaluate the performance differences? Maybe I missed some details in the article...

rachelwenzel 6y ago
Hey there! Article author here- This was written as an after-the-fact look back after a year long project and the target testing set did change a little along the way. The big change was splitting by manufacturer (and eventually by model family), which changed the number of applicable samples to compare performance on. I cut some stuff from the article for length, so I totally get how that could have been unclear.
- washedup 6y ago
  Appreciate the explanation, thanks! Enjoyed the article.

Abishek_Muthian 6y ago

Neat ML exercise, though it wasn't the right solution for the problem. What is the right solution at consumer level though? I currently run smartmontools at regular intervals, compare the TBW with manufacturer's TBW and send a MQTT message using a script.

MaxBarraclough 6y ago

These guys offer something similar as a service: https://www.senseye.io/how-it-works https://www.senseye.io/how-it-works

moonbug 6y ago

Ugh. Statistics, people.

ryeguy_24 6y ago

It would be interesting to empirically identify the state space that leads to increased probability of failure and then seek to understand the rationale for this failure through physics. Ultimately, the ML model is picking up on some combination of factors that exposes physical weaknesses of the device.

PaulHoule 6y ago

My understanding is that SMART is not effective because it only catches (some) mechanical problems and that about half of the time it is the logic board that went bad. You can filter out some of the bad drives ahead of time, but you will still get blindsided much of the time.

8fingerlouie 6y ago
Assuming you don’t have an early fail drive, it has one usable stat, “Power On Hours”. When that begins to creep towards 40,000 you will have a failed drive sooner than later. As for the rest of the counter, raw read error rate, relocated sector count and friends are also good indications that your drive might not be in the best shape. They don’t say anything about when it will fail though. I’ve had a drive rack up 20 relocated sectors in a couple of weeks, only to stay at 20 for 4 years after that. And load cycle count is only good for telling if you’re within manufacturer parameters. I frequently see old drives with Load Cycle Count 10-25 times the rated value, and they work just “fine”.
- Uberphallus 6y ago
  Yeah, the training data is basically useless: 1. Extreme imbalance and survivorship bias. The dataset is unwieldy because of the imbalance, no matter the algorithm; and too small if you balance it. 2. I would take a daily/weekly/monthly diff between different values and time to failure, and obviously only from drives that ACTUALLY failed. All drives fail, but data from healhty drives is useless as it lacks the target value. In practice you want to predict how close they are to failing and act preemptively, tagging them as OK and KO completely misses the target (hehe) IMO. 3. Drive maker and model should be vectorized and fed into the model. Mandatory. Different manufacturers and models show different SMART behaviors, and while some stats are red flags in certain drives, they mean nothing in others. 4. This is more of a personal preference, but I like random trees to test whether the dataset is useful. They're not perfect but they give reasonably good results for most tasks, and tend not to overfit, so you can use it as overfitting benchmark vs other algorithms. If they don't either the dataset is shit or it needs some transformation, or the model needs to be more complex (but that's easy to see). Obviously this applies to classifiers and regressions, and not to many other ML tasks.

CraigJPerry 6y ago

For enriching the model, a few ideas for discriminating failure drives: Some hard drives have a vibration sensor(s) in them - i wonder if a daemon could sample throughout the day and dump a summarisation of the data when the daily SMART data dump is being generated. The drives report temperature via SMART, i wonder if anything meaningful can be extracted from the temperature differential between the HDD’s sensor and a chassis temp sensor? I wonder if more discrimination by manufacturing detail (date of mfr, which factory etc) could help? I was going to suggest drive supply voltages but those are hard to compare across chassis - calibration of voltage measuring circuitry in PSUs isn’t too hot for the millivolt differences you’d want to measure - the differences would be so small to be lost in the inaccuracies between boards. Also many PSUs lack a method to query their measurements. Fascinating study and excellent writeup. I have zero ML background but i really enjoyed this.

oli5679 6y ago

Fitting a binary classifier on failure doesn't seem quite right to me. There are some quite natural questions you can't answer unless you can predict some stats on when failure will occur, and if you have any drift in the dataset, you might just find features correlated with age (and hence having more time to fail) that don't inherently increase risk. I would recommend: (a) using survival analysis https://lifelines.readthedocs.io/en/latest/ https://lifelines.readthedocs.io/en/latest/ https://xgboost.readthedocs.io/en/latest/tutorials/aft_survival_analysis.html https://xgboost.readthedocs.io/en/latest/tutorials/aft_survi... or (b) flattening the data to be one observation per time window, and predicting failure over specific time windows. This is a good talk also. https://dataorigami.net/blogs/napkin-folding/18308867-video-presentation-on-lifelines-survival-analysis-in-python-sept-23-2014 https://dataorigami.net/blogs/napkin-folding/18308867-video-... Although I'm not sure about this case, in general the approach in the article (tree-ensemble predicting binary classier) works well for many business problems!

mleonhard 6y ago

They sample the SMART metrics once a day. Would more frequent samples would provide a better signal? For example, perhaps the read errors number increases about once a week. But in the week before a failure, it increases once a day. Or in the day before a failure, it increases once an hour. OS-level metrics could also be predictive: write latency, i/o queue length, communication errors, and checksum errors in read data.

Faint 6y ago

Maybe training with actual bottom-line utility function would have shown some use case, that is, instead of MCC you would have predicted whether sending a person to change the drive now before it fails would cost less than letting it fail and cleaning up afterwards. But if the drives are part of redundant array, it would be almost always cheapest to let them fail.. and if large part of failures are asymptomatic, you need the array anyway for critical stuff, so I suppose it's a useless exercise.

jiveturkey 6y ago

love everything coming out of datto and this is no exception

the_optimist 6y ago

I think this write-up is interesting but structurally incorrect, and everyone here is taking away the wrong conclusions. The punchline here is: > Unfortunately, once they were scaled up to production levels of use, their tradeoffs turned out to be too much to bear. These models are good—especially given how imbalanced the data for this problem is—but with too many false positives from one and too little predictive power from the other, they’re not worth betting so much money on. A practical next-step analysis would be some back-of-the-envelope economic math about the necessary detection threshold for detecting and replacing failing drives. Overall, it doesn't even make sense to even think about applying ML without understanding the cost/benefit of solving the problem. Think about Backblaze -- the drives are packed in there tightly and it's super expensive to replace one, whereas there's basically zero cost to leaving one in place. Even if you could perfectly predict which drives would fail, there's no reason you'd act on that info.