6 ms·

Backblaze Drive Stats for Q1 2022

183 points by caution 4y ago

ww520 4y ago
Is there a failure list for write heavy drives? The drives for my security system have failed twice already. They are NAS grades from Seagate and WD.
- brianwski 4y ago
  Disclaimer: I work at Backblaze but more on the backup client side that runs on laptops, not in the datacenter storage side. > Is there a failure list for write heavy drives? To be clear about what these drive failure stats are and what they are not: Backblaze runs a data storage service with about 214,000 hard drives in it right now. We don't run any specific tests or induce issues on purpose FOR the drive failure stats, we just report what occurred in our datacenter. Sometimes readers think we're carefully running a "study", but it's more just what we have experienced as honestly as we can offer it up. If the reads and writes and seeks our drives experience matches your particular application, great! Or maybe it is just interesting to read. Now, we do save (and publish) all the raw data, and some other awesome people out there have done various analysis on it, which always makes us happy also. You can find the raw data here: https://www.backblaze.com/b2/hard-drive-test-data.html https://www.backblaze.com/b2/hard-drive-test-data.html At this point it goes back almost a full decade.
  - isomorphic 4y ago
    It looks like the data does contain SMART 241, total LBAs written, for at least some drives. Someone with some time could correlate that to failure rate. My hypothesis is of course it's correlated--but by how much?
    - brianwski 4y ago
      > Someone with some time could correlate SMART data to failure rate. We looked into it a little, some notes written up here: https://www.backblaze.com/blog/what-smart-stats-indicate-hard-drive-failures/ https://www.backblaze.com/blog/what-smart-stats-indicate-har... Short summary is there are a few SMART stats that seem to predict failure way more than others, which is probably obvious. But we aren't PhDs in statistics and it isn't our area of focus, so.... This guy wrote a paper based on the Backblaze SMART data: https://etd.ohiolink.edu/apexprod/rws_etd/send_file/send?accession=bgsu1594957948648404&disposition=inline https://etd.ohiolink.edu/apexprod/rws_etd/send_file/send?acc... These 5 guys wrote another paper based on the Backblaze SMART data to train up a Bayesian network to predict failures: https://ieeexplore.ieee.org/document/8489097 https://ieeexplore.ieee.org/document/8489097 This is another article of predicting hard drive failures using the Backblaze SMART data: https://karthikna.github.io/Prediction-of-Hard-Drive-Failure/ https://karthikna.github.io/Prediction-of-Hard-Drive-Failure... I can't comment on their findings, but it's DEFINITELY an interesting thing to study now that we have almost 10 years of these drive stats across a pretty big drive farm.
  - stonecharioteer 4y ago
    Is there something you published regarding regular consumer drives? I ask this because I cannot find / afford high capacity enterprise drives in India, and I am forced to make multiple backups on cheap external hard drives. I'm building https://github.com/stonecharioteer/renfield https://github.com/stonecharioteer/renfield for this purpose. Before I get around to it, I'm trying out git-annex, but I must say I don't like the git approach to files.
mark-r 4y ago
I was wondering why there were so many Star Wars quotes, then I realized today is May the fourth. Nicely done!
joncrane 4y ago
I like how the article was posted on May 4th and it has lots of Star Wars references.
azalemeth 4y ago
Lovely and wonderfully punny write up. I wonder if they've done any proper Cox proportional hazards lifetime analysis across the different models? The stats for this sort of problem are well developed. Of course, the hazard is not going to be proportional, I suspect, bit I'd love to see that come out of the regression...
ksec 4y ago
The whole reason why the market still demands HGST. There is a whole segment / sector / section of industry that refuse to call it anything other than HGST. And they are willing to pay for it.
malwarebytess 4y ago
I don't even look at reviews when it's time to add more storage. I go straight to these updates. Hasn't steered me wrong.
- comboy 4y ago
  These have been popular for some time now and I've been wondering what are the chances of manufacturers sending better batches to Backblaze. I imagine you run some batch and some percentage of disks don't pass tests, you may adjust something or maybe it's the last one before routine machine maintenance, etc. No idea if there really even is such thing as batches which manufacturers know have higher failure rates. But it would be interesting. Backblaze shouldn't complain though ;)
  - Sakos 4y ago
    https://www.backblaze.com/blog/how-backblaze-buys-hard-drives/ https://www.backblaze.com/blog/how-backblaze-buys-hard-drive... Some interesting answers to this and other questions.
    - jaycar 4y ago
      ... "Due to the quantities we need and the visibility of the posts, drive manufacturers are motivated to give us their best possible product." ....
dsr_ 4y ago
I had osmosed that the bathtub curve isn't real, but the new time-trail quadrant graph certainly looks like evidence for at least the infant-mortality section.
jenny91 4y ago
The thing people care about is the AFR over time: like actuarial life tables. These updates always make it a bit hard to figure that one out: the tables show "average life" and AFR; it makes no sense to compare drives of different ages. The "Age vs AFR" plot is getting there but it should be flipped and is hard to read; error bars would also help. Also percentiles would be useful: people want to minimize the probability of failure.
alex3305 4y ago
Based on these reports and pricing, I've currently filled my NAS with only Toshiba drives. 1 4TB, 1 5TB and 2 14TB drives and all of them have been great. These enterprise drives are quite noisy, but my oldest already runs for over 4 years without any reallocated sectors or other issues. Apart from that, Toshiba provides free 5 year warranty on these drives. And the best thing? These bad boys are only 250 euro's for 14TB or 300 euro's for 18TB of raw storage capacity.
Melatonic 4y ago
I love these stats Generally I only buy enterprise drives
- throwaway894345 4y ago
  According to https://www.backblaze.com/blog/how-backblaze-buys-hard-drives/ https://www.backblaze.com/blog/how-backblaze-buys-hard-drive..., consumer drives give better quality, and the main reason BB buys enterprise drives is that consumer drives aren't available at their scale.
  - dijit 4y ago
    Just to add: Consumer drives are not necessarily sub-par quality-wise just because they're consumer drives. Sometimes what it comes down to is firmware; WD NAS drives used to be almost identical to conumer drives except their firmware would report errors back to the SATA controller directly instead of continually trying to read the block. Continually reading the block seems doomed but it works a surprisingly large number of times, and when it's successful the block is often put somewhere else on the drive transparently (and the old block is marked as "bad"). This drastically improves the lifetime of the drives since they're slowly wearing down vs "going bad" because of a single block being unrecoverable. This is not the only difference, but it's one thing to consider. It's similar to how the linux kernel can be compiled with low latency in mind; which causes lower throughput but makes the desktop experience much nicer (ime). It's just trade-offs. Nothing is just "objectively better" it's a different trade off.
    - throwaway894345 4y ago
      > Consumer drives are not necessarily sub-par quality-wise just because they're consumer drives. According to the link I shared, they aren't lower quality, they're higher quality than enterprise drives. But that's an interesting note about the drive firmware!
raybb 4y ago
I wonder if they consider doing something like putting data more likely to be accessed on the more reliable drives so that that ones that wear out easier are used less.

183 points by caution 4y ago

Is there a failure list for write heavy drives? The drives for my security system have failed twice already. They are NAS grades from Seagate and WD.

brianwski 4y ago
Disclaimer: I work at Backblaze but more on the backup client side that runs on laptops, not in the datacenter storage side. > Is there a failure list for write heavy drives? To be clear about what these drive failure stats are and what they are not: Backblaze runs a data storage service with about 214,000 hard drives in it right now. We don't run any specific tests or induce issues on purpose FOR the drive failure stats, we just report what occurred in our datacenter. Sometimes readers think we're carefully running a "study", but it's more just what we have experienced as honestly as we can offer it up. If the reads and writes and seeks our drives experience matches your particular application, great! Or maybe it is just interesting to read. Now, we do save (and publish) all the raw data, and some other awesome people out there have done various analysis on it, which always makes us happy also. You can find the raw data here: https://www.backblaze.com/b2/hard-drive-test-data.html https://www.backblaze.com/b2/hard-drive-test-data.html At this point it goes back almost a full decade.
- isomorphic 4y ago
  It looks like the data does contain SMART 241, total LBAs written, for at least some drives. Someone with some time could correlate that to failure rate. My hypothesis is of course it's correlated--but by how much?
  - brianwski 4y ago
    > Someone with some time could correlate SMART data to failure rate. We looked into it a little, some notes written up here: https://www.backblaze.com/blog/what-smart-stats-indicate-hard-drive-failures/ https://www.backblaze.com/blog/what-smart-stats-indicate-har... Short summary is there are a few SMART stats that seem to predict failure way more than others, which is probably obvious. But we aren't PhDs in statistics and it isn't our area of focus, so.... This guy wrote a paper based on the Backblaze SMART data: https://etd.ohiolink.edu/apexprod/rws_etd/send_file/send?accession=bgsu1594957948648404&disposition=inline https://etd.ohiolink.edu/apexprod/rws_etd/send_file/send?acc... These 5 guys wrote another paper based on the Backblaze SMART data to train up a Bayesian network to predict failures: https://ieeexplore.ieee.org/document/8489097 https://ieeexplore.ieee.org/document/8489097 This is another article of predicting hard drive failures using the Backblaze SMART data: https://karthikna.github.io/Prediction-of-Hard-Drive-Failure/ https://karthikna.github.io/Prediction-of-Hard-Drive-Failure... I can't comment on their findings, but it's DEFINITELY an interesting thing to study now that we have almost 10 years of these drive stats across a pretty big drive farm.
- stonecharioteer 4y ago
  Is there something you published regarding regular consumer drives? I ask this because I cannot find / afford high capacity enterprise drives in India, and I am forced to make multiple backups on cheap external hard drives. I'm building https://github.com/stonecharioteer/renfield https://github.com/stonecharioteer/renfield for this purpose. Before I get around to it, I'm trying out git-annex, but I must say I don't like the git approach to files.

mark-r 4y ago

I was wondering why there were so many Star Wars quotes, then I realized today is May the fourth. Nicely done!

joncrane 4y ago

I like how the article was posted on May 4th and it has lots of Star Wars references.

azalemeth 4y ago

Lovely and wonderfully punny write up. I wonder if they've done any proper Cox proportional hazards lifetime analysis across the different models? The stats for this sort of problem are well developed. Of course, the hazard is not going to be proportional, I suspect, bit I'd love to see that come out of the regression...

ksec 4y ago

The whole reason why the market still demands HGST. There is a whole segment / sector / section of industry that refuse to call it anything other than HGST. And they are willing to pay for it.

malwarebytess 4y ago

I don't even look at reviews when it's time to add more storage. I go straight to these updates. Hasn't steered me wrong.

comboy 4y ago
These have been popular for some time now and I've been wondering what are the chances of manufacturers sending better batches to Backblaze. I imagine you run some batch and some percentage of disks don't pass tests, you may adjust something or maybe it's the last one before routine machine maintenance, etc. No idea if there really even is such thing as batches which manufacturers know have higher failure rates. But it would be interesting. Backblaze shouldn't complain though ;)
- Sakos 4y ago
  https://www.backblaze.com/blog/how-backblaze-buys-hard-drives/ https://www.backblaze.com/blog/how-backblaze-buys-hard-drive... Some interesting answers to this and other questions.
  - jaycar 4y ago
    ... "Due to the quantities we need and the visibility of the posts, drive manufacturers are motivated to give us their best possible product." ....

dsr_ 4y ago

I had osmosed that the bathtub curve isn't real, but the new time-trail quadrant graph certainly looks like evidence for at least the infant-mortality section.

jenny91 4y ago

The thing people care about is the AFR over time: like actuarial life tables. These updates always make it a bit hard to figure that one out: the tables show "average life" and AFR; it makes no sense to compare drives of different ages. The "Age vs AFR" plot is getting there but it should be flipped and is hard to read; error bars would also help. Also percentiles would be useful: people want to minimize the probability of failure.

alex3305 4y ago

Based on these reports and pricing, I've currently filled my NAS with only Toshiba drives. 1 4TB, 1 5TB and 2 14TB drives and all of them have been great. These enterprise drives are quite noisy, but my oldest already runs for over 4 years without any reallocated sectors or other issues. Apart from that, Toshiba provides free 5 year warranty on these drives. And the best thing? These bad boys are only 250 euro's for 14TB or 300 euro's for 18TB of raw storage capacity.

Melatonic 4y ago

I love these stats Generally I only buy enterprise drives

throwaway894345 4y ago
According to https://www.backblaze.com/blog/how-backblaze-buys-hard-drives/ https://www.backblaze.com/blog/how-backblaze-buys-hard-drive..., consumer drives give better quality, and the main reason BB buys enterprise drives is that consumer drives aren't available at their scale.
- dijit 4y ago
  Just to add: Consumer drives are not necessarily sub-par quality-wise just because they're consumer drives. Sometimes what it comes down to is firmware; WD NAS drives used to be almost identical to conumer drives except their firmware would report errors back to the SATA controller directly instead of continually trying to read the block. Continually reading the block seems doomed but it works a surprisingly large number of times, and when it's successful the block is often put somewhere else on the drive transparently (and the old block is marked as "bad"). This drastically improves the lifetime of the drives since they're slowly wearing down vs "going bad" because of a single block being unrecoverable. This is not the only difference, but it's one thing to consider. It's similar to how the linux kernel can be compiled with low latency in mind; which causes lower throughput but makes the desktop experience much nicer (ime). It's just trade-offs. Nothing is just "objectively better" it's a different trade off.
  - throwaway894345 4y ago
    > Consumer drives are not necessarily sub-par quality-wise just because they're consumer drives. According to the link I shared, they aren't lower quality, they're higher quality than enterprise drives. But that's an interesting note about the drive firmware!

raybb 4y ago

I wonder if they consider doing something like putting data more likely to be accessed on the more reliable drives so that that ones that wear out easier are used less.