5 ms·

Says the annual failure rate is 1.5%, but average time to failure is 2.5 years? Those numbers don't line up. Are most drives retired without failing?

by sdwr 2y ago

codemac 2y ago
Drives have warranties, after which the manufacturer doesn't make any claims about it's durability. This could put your fleet at wild and significant risk if things start hitting a wall and failing en masse. You may not be able to repair away if as you're repairing the data you're copying to yet another dying drive. So you have usually a lifetime of drive tput and start/stop values you want to stay under, and depending on how accurate your data is for each drive you may push beyond the drive warranties. But you will generally stop before the drive actually fails.
rsync 2y ago
"Are most drives retired without failing?" Yes, certainly. One can watch both SMART indicators as well as certain ZFS stats and catch a problem drive before it actually fails. I like to remove drives from zpools early because there is a common intermediate state they can fall into where they have not failed out but dramatically impact ZFS performance as they timeout/retry certain operations thousands and thousands of times.
- favorited 2y ago
  What's the best way to monitor those ZFS stats? I just rely on scheduled ZFS scrubs, and the occasional `zpool status -v`...
mnw21cam 2y ago
Yes, and because of that the numbers on the average time to failure are completely meaningless. The drives the don't ever fail skew the numbers completely. If a fantastically reliable drive were to have 5/5000 drives fail, but they all failed in the first month and then the rest carried on forever, then that would show here as having a lower "reliability" than a dire drive where 4000/5000 drives fail after a year. I'd like to see instead something like mean time until 2% of the drives fail. That'd actually be comparable between drives. And yes, it would also mean that some drive types haven't reached 2% failure yet, so they'd be shown as ">X months". This is what a Kaplan-Meier survival curve was meant for [0]. Please use it. Also, it'd be great to see the confidence intervals on the annualised failure rates. [0] https://en.wikipedia.org/wiki/Kaplan%E2%80%93Meier_estimator https://en.wikipedia.org/wiki/Kaplan%E2%80%93Meier_estimator
bombcar 2y ago
Obviously yes. At an AFR of 1.5% they'd have to have the drives run for (about) 67 years to have them all retire from failure. (in reality they'd probably have failure rates spike at some point, but the idea stands. And they explicitly said they retired a bunch of 4TBs)
scottlamb 2y ago
> Are most drives retired without failing? I'd expect so given that HDDs are still having significant density advancements. After a while old drives aren't worth the power and sled/rack space that could be used for a higher capacity drive. And, yeah, it makes these statistics make more sense together. Edit: plus they are just increasing drive count so most drives haven't hit the time when they would fail or be retired...
rovr138 2y ago
They just retired 4TB ones. While they seem to get retired, it's not as quick as we'd think.

Says the annual failure rate is 1.5%, but average time to failure is 2.5 years? Those numbers don't line up. Are most drives retired without failing?

by sdwr 2y ago

Says the annual failure rate is 1.5%, but average time to failure is 2.5 years? Those numbers don't line up. Are most drives retired without failing?

codemac 2y ago

Drives have warranties, after which the manufacturer doesn't make any claims about it's durability. This could put your fleet at wild and significant risk if things start hitting a wall and failing en masse. You may not be able to repair away if as you're repairing the data you're copying to yet another dying drive. So you have usually a lifetime of drive tput and start/stop values you want to stay under, and depending on how accurate your data is for each drive you may push beyond the drive warranties. But you will generally stop before the drive actually fails.

rsync 2y ago

"Are most drives retired without failing?" Yes, certainly. One can watch both SMART indicators as well as certain ZFS stats and catch a problem drive before it actually fails. I like to remove drives from zpools early because there is a common intermediate state they can fall into where they have not failed out but dramatically impact ZFS performance as they timeout/retry certain operations thousands and thousands of times.

favorited 2y ago
What's the best way to monitor those ZFS stats? I just rely on scheduled ZFS scrubs, and the occasional `zpool status -v`...

mnw21cam 2y ago

Yes, and because of that the numbers on the average time to failure are completely meaningless. The drives the don't ever fail skew the numbers completely. If a fantastically reliable drive were to have 5/5000 drives fail, but they all failed in the first month and then the rest carried on forever, then that would show here as having a lower "reliability" than a dire drive where 4000/5000 drives fail after a year. I'd like to see instead something like mean time until 2% of the drives fail. That'd actually be comparable between drives. And yes, it would also mean that some drive types haven't reached 2% failure yet, so they'd be shown as ">X months". This is what a Kaplan-Meier survival curve was meant for [0]. Please use it. Also, it'd be great to see the confidence intervals on the annualised failure rates. [0] https://en.wikipedia.org/wiki/Kaplan%E2%80%93Meier_estimator https://en.wikipedia.org/wiki/Kaplan%E2%80%93Meier_estimator

bombcar 2y ago

Obviously yes. At an AFR of 1.5% they'd have to have the drives run for (about) 67 years to have them all retire from failure. (in reality they'd probably have failure rates spike at some point, but the idea stands. And they explicitly said they retired a bunch of 4TBs)

scottlamb 2y ago

> Are most drives retired without failing? I'd expect so given that HDDs are still having significant density advancements. After a while old drives aren't worth the power and sled/rack space that could be used for a higher capacity drive. And, yeah, it makes these statistics make more sense together. Edit: plus they are just increasing drive count so most drives haven't hit the time when they would fail or be retired...

rovr138 2y ago

They just retired 4TB ones. While they seem to get retired, it's not as quick as we'd think.