2 ms·
Do we need actual prediction? Ill settle for Just in time warning. This is what SMART parameter C4 gives you. You want to stop using the drive at the first sign
by rasz 6y ago
Do we need actual prediction? Ill settle for Just in time warning. This is what SMART parameter C4 gives you. You want to stop using the drive at the first sign of trouble, thats what C4 is good for.
>Only one drive manufacturer had enough failed drives in the dataset to produce a large enough subset, which we’ll refer to as Manufacturer A
Seagate, we will refer to this manufacturer as SEAGATE.
- jeffbee 6y agoI agree that the article fails to make an argument for why this prediction would be useful. What would you do with it, even if it was perfect?
- rasz 6y agoIf prediction would be possible you could do unethical things, like selling off units bound to fail in near future without disclosing this information. I wonder if there are any existing examples of such practices. Supermarket chains sell products near their expiration date at steep discounts, of course the difference is you know what you are buying and can still eat a 20 days until expiry 50% price Nutella jar.
- toast0 6y agoWith perfect (or good enough) prediction, you can replace drives before they fail, in controlled conditions in a maintenance window.
- jeffbee 6y agoAssuming your prediction is so good that it completely eliminates surprise failures, OK. But if your software can handle surprise failures then it can handle them, and in that case I don't see the value of the prediction. The only valuable thing I can see is if a system could predict that you were going to have widespread, coordinated failure of so many drives that your ability to reconstruct a stripe was at elevated risk. But such a thing is science fiction (as the article demonstrates) and you can approximate it quite well by assuming that all drives of the same make and model with the same in-service date will fail at the same moment.
- toast0 6y agoThe difference is urgency, and knock on effects. If your system is designed for hot swap of failed disks, but 1% of the time, you need to reboot the system for the new disk to be detected; predictive replacement lets you move traffic off the system at a convenient time to do the swap (when traffic is low, and operations staff isn't super busy). Predictive replacement can be deferred more easily, because the system is still working; maybe you can batch it with some other maintenance. A surprise failure needs to be dealt with in a more timely fashion. In my experience (which I didn't write a blog post for), monitoring the right SMART values and doing predictive replacement eliminated most of the surprise failures for spinning drives. SSDs had less failures, but I wasn't able to find any predictive indicators; they generally just disappeared from the OS perspective.
- jeffbee 6y agoYou are trading off cost, though. According to that paper I referenced, the SMART reallocated sector count is just binary (the critical threshold is 1 sector). Drives with non-zero counts are much more likely to fail but 85% of them are still in service a year later. If you proactively replace them you may indeed have avoided some surprise failures but it's actually not a great deal more effective than replacing drives randomly, and it costs money. Also we haven't even discussed the online disk repair phenomenon, in which you take a "failed" disk, perform whatever lowest-level format routine the manufacturer has furnished, and get another 5 years of service out of it. This is done without ever touching it.
- toast0 6y agoReallocated sectors is pretty good, but I'd also look at pending and offline uncorrectables to get a potential earlier warning. I've seen some drives that were apparently working fine with about 1000 reallocated sectors; and the smart firmware usually indicates a maximum of quite a few more than that, so if you replace at 100 problem sectors, I would consider that 'predictive' replacement. It's debatable of course; something that predicted failure based on changes in data transfer (that presumably precede sectors being flagged), would be a bigger predictive step, but I never got that far in modeling; using flagged sector counts was enough to turn unscheduled drive failures into scheduled drive replacements for me.