4 ms·
Wow, this is a surprisingly sober look at using ML for this particular problem. I really appreciate how the author walks through every step and the rationale be
by whoisburbansky 6y ago
Wow, this is a surprisingly sober look at using ML for this particular problem. I really appreciate how the author walks through every step and the rationale behind it, spending as much time on explaining the dataset and cleaning it up as she does on, say, the model they end up using.
It’s a bit of a breath of fresh air to see a post like this where someone clearly puts the work in to try and solve a given problem with ML and is able to articulate the failure modes of the approach so well.
- mlthoughts2018 6y agoThis is very standard in industry (as opposed to academia) and highlights a lot of the reasons why you need ML specialists operating these models, and why you will get garbage in garbage out if you try having non-specialist engineers build, evaluate & operate these types of systems. For example, in this post the engineer does not adequately handle class imbalance at all, and reaches for a hugely complicated model framework (XGBoost) without even trying something like bias-corrected logistic regression first. I think this post is just an example of an infra engineer thinking “ML is easy” and when their quick attempt doesn’t work, then just bailing and saying “ML is overhyped and doesn’t work for my problem.”
- causalmodels 6y agoThis is not the correct modeling approach. All hard drives will fail given enough time, so labeling the failed hard drives as the positive class will bias your results. Stuff like this really should be handled using survival analysis.
- mlthoughts2018 6y agoIn fact it screams Bayesian survival analysis, since there is so much prior knowledge both of general hard drive failure rates and SMART stats.
- astrophysician 6y agoI agree and have been doing some similar analysis on the backblaze dataset. I suppose you can use this for prediction but I'm personally just interested in post-hoc analysis and (1) getting better AFR estimates when failure counts are low + (2) exploring time-dependence of hazard functions with different priors (GP priors, etc.). This post and your comment have motivated me to make a post this weekend! Thanks!
- causalmodels 6y agoYou should check out this time-to-event neural network [1]. [1] https://github.com/ragulpr/wtte-rnn https://github.com/ragulpr/wtte-rnn
- jgalentine007 6y agoSMART is a liar sometimes. I have first hand experience with faulty Seagate firmware and Equallogic SANs - where errant statistics caused disks to be ejected from the volumes before you could finish a rebuild. Nothing like watching 40TB of data disappear on multiple installations over the course of a few weeks!
- R0b0t1 6y agoSMART seems to be extremely useless in practice. Manufacturers don't seem to expose actual failure statistics through it, likely for fear of making their product look bad.
- deleted 6y ago[deleted]
- Karrot_Kream 6y agoWhat was the author's rationale for choosing XGBoost? Failure analysis is an old field so I'm very surprised that there's not even a cursory nod to the exponentially distributed failure times that most failure models use. If anything, this post just spends a lot more time than usual talking about data cleanup.
- suifbwish 6y agoThis is an interesting experiment because if the model is running on the same hard drive it is detecting, then it is looking at itself dying in the mirror