6 ms·
Backblaze Drive Stats for Q1 2021
- nikisweeting 5y agoI feel like SSD failure has little correlation with hours running, and more to do with TBW. Would be nice to see some read/write totals on these stats going forward.
- toast0 5y agoDepends on the SSDs and the cause of failure. There have been high profile cases where a firmware bug meant an absolute cap on hours running. I've been involved in server farms with thousands of (mostly Intel) SSDs and (mostly WD) spinning drives; the spinning drives tended to have pre-failure indicators, but we couldn't figure out any indicators before SSD failure and generally they would just completely disappear from the bus when they did fail. The failure rate was signficantly less though. Our write rate wasn't very high and tended to be small writes; for busy disks, more than what we could do with a spinning disk, but usually not near the capability of the drives.
- sekh60 5y agoJust a homelabber, but in my 40+ drive homelab I've noticed the same. HDDs normally toss a couple SMART errors when dying. For SSDs I've gone through maybe 10 of them so far, and they just suddenly kick the bucket without warning.
- nikisweeting 5y agoMy experience with WD SSDs is that they just slow down more and more as they get close to the TBW limits, without failing fully / dropping off the bus (as another user describes here: https://news.ycombinator.com/item?id=27040491 https://news.ycombinator.com/item?id=27040491).
- tux3 5y agoSomeday I'm going to take those numbers and try to see how the AFR varies with the age of the drive. I expect 8 year old drives to have a higher AFR than 3yo, and 6 month olds to be somewhere in between, for example. And then, we could compute what the life expectancy of a given model is given how long you've had it (just bought, few years old, etc) It'd be fun to compare those across vendors and drive models. There's maybe even enough data at this point that some of the numbers might be meaningful! =)
- andy4blaze 5y agoAndy at Backblaze here. We do look at drive model failure over time. We did a post on this topic several years ago, 2015? At the time, most drive followed the bathtub curve of failure, but I'm not sure that is still the case. I think its time to update that report.
- ksec 5y agoMay be also comment on price of HDD? Doesn't seems to move at all.
- drexlspivey 5y agoI bought a toshiba NAS drive mainly due to backblaze’s stats and it came bricked out of the box :(
- tinus_hn 5y agoOn the bright side it didn’t lose you your data!
- deleted 5y ago[deleted]
- agumonkey 5y agoblack swan spotted
- dragontamer 5y agoThat very well could be on the delivery person / packing of the hard drive. Hard drives are quite fragile: if they're dropped hard by the delivery person (ex: dropped onto your concrete steps), they could break.
- louwrentius 5y agoI once had a 20-drive NAS with 1 TB Samsung Spinpoints F1s. The NAS is replaced but I still have the drives just for labbing / testing purposes. I never had a drive failure during the lifetime of the NAS. Probably because it was off most of the time and only powered-on with wakeonlan when needed. So those drives don’t have many hours on them. But recently they started dying. I lost 3 of them this year during some tests. Imagine that these drives are probably 10+ years old. Age does seem to matter. Obviously this is a small uncontrolled sample but it seems that you really should keep this in mind when you run a NAS at home. Keep an eye on the SMART parameters as suggested by Backblaze and really consider replacing drives at some point. I would be afraid that drives do start dying at the same time due to age.
- guerby 5y agoWhat would be interesting for SSD is percentage of advertized TBW when (or just before) the SSD failed, ie 100% if the SSD fails at exactly its advertized TBW, 50% if it fails at half the TBW, and 200% if it lasts two times the advertized TBW. In case someone from backblaze read this :)
- londons_explore 5y agoA well written SSD firmware simply slows down with age. It will never get to failure because the slowdown gets so extreme it becomes unusable. The drive also gets slightly smaller (by passing write failures to the OS to mark sectors bad). Thats because a "worn out" flash sector is never fully worn out - it can still store some data, just less than the error correction can correct. It is possible to combine two sectors to have extra error correction data to still recover a sector. Now you have less than half the performance. Worn out flash also doesn't hold data long - perhaps only a few hours before too many bits have flipped and it is unreadable. To fix that, you need to rewrite the data, which slows everything down more. And now that you have a bunch of unreliable sectors, you also need "super-sectors" which can do sector-based hierarchical erasure coding to recover data from sectors where even the methods above have caused data to be lost. This slows down writes even more. In the worst case, reading a single sector requires reading every sector on the drive to reconstruct. Clearly thats going to be slow enough the drive will have stopped being used long before that. Sadly some drive firmware doesn't implement some or all of the above, so they appear to have "failed" and become unreadable, which IMO is inexcusable when it's very easy to design so that worn out drives become slow instead.
- bl4ckneon 5y agoWow this was a really in depth comment!
- capableweb 5y agoSuper interesting. Do you have any examples of drives that have the proper firmware in your experience? Sounds much better to own than something that suddenly fails.
- jonplackett 5y agoBackblaze is awesome. Anyone not using it should give it a try. But I have a question - why do they share this info? Is it to show they’re reliable or just for curiosity? Or some other reason?
- radicality 5y agoIs it, have you had good experiences loading the backups? I was on the lookout recently for some offsite backup solution and came across this blog post that didn’t inspire confidence in backblaze. https://messengergeek.wordpress.com/2018/03/09/backblaze-review-data-loss/ https://messengergeek.wordpress.com/2018/03/09/backblaze-rev...
- treesknees 5y agoThey've added a separate backup client that is supposed to be better than the web interface.
- jw1224 5y agoIt's marketing (and effective, too). I became a Backblaze customer after reading one of these reports (many years ago!).
- brainzap 5y agoIf it only would also backup attached USB sticks.
- wing-_-nuts 5y agoOne thing I've always wondered, are these drives what people would recommend that you stick in a desktop or nas or are these 'datacenter' drives that are overkill for consumer use?
- OldTimeCoffee 5y agoI can only speak about the Exos drives, but the Ironwolf (Pro) drives are basically just the Exos drives relabeled for the consumer market. I'm actually about to buy 6 14TB Exos x16 drives to replace 7 6TB Exos 7E8 drives that are 5 years old. Frequently you can get the Exos drives cheaper than Ironwolf anyway.
- lostlogin 5y ago> Frequently you can get the Exos drives cheaper than Ironwolf anyway. You can. I’ve been getting the 16s and they are great but with caveats. The size makes volume creation and expansion insanely long (like a week per added drive). Not Seagate’s fault I know. The noise. They are loud. They are either chirping away or grinding away.
- robotmay 5y agoI would definitely recommend skipping any current large WD drives for home use. < 6TB are SMR and problematic in a NAS, and 6TB+ have a very irritating noise that no-one seems to be able to diagnose (sounds like a scanner when idle). I bought some 8TB WD Golds, and I was expecting them to be louder, but they do something very weird when idling and it's an extremely penetrating sound. It seems to be present on most large models: https://community.wd.com/t/strange-noise-coming-from-10tb-drives-wd100efax/226467 https://community.wd.com/t/strange-noise-coming-from-10tb-dr...
- elorant 5y agoBackblaze has said in the past that they don't buy enterprise drives because they didn't notice any difference from consumer ones. I don't know if that's still their policy.
- 5y ago
- londons_explore 5y agoThink of the classic "bathtub" curve (which says that young drives fail more frequently, old drives fail more frequently, and mid-age drives are most reliable). That curve doesn't seem to match the data here. Or if it does, it says the "old" increase in failure rate happens at over 5 years. I would guess backblaze will replace these old drives because they are too small/too slow/use too much power before they replace them for being too unreliable.
- andy4blaze 5y agoAndy for Backblaze here: A while back we did an analysis of drive failure over time, i.e. the bathtub curve. It is probably a good idea to update that, as I believe we are seeing lower failure rates upfront these days.
- londons_explore 5y agoDoes backblaze ever power cycle the drives (either on a schedule or due to planned/unforeseen circumstances)? If so, it would be interesting to know how many drives failed on the day of a power cycle vs days with no power cycle. I know other providers have found that "power cycle days" can be 100x more deadly for drives than "non-power-cycle days". It can have a massive impact when estimating data loss probabilities - since unforseen power cycle days tend to impact more than one drive at a time...
- andy4blaze 5y agoAndy for Backblaze here: I looked at that 3-4 years ago. It looks like power cycling increased failure rates, but we don't power cycle our systems very often, maybe 1-2 a year, so not the best use case. This is on my list for a relook one of these days, if we find anything interesting we'll let folks know.
- dang 5y agoFor those who need more: Backblaze Hard Drive Stats - https://news.ycombinator.com/item?id=26696876 https://news.ycombinator.com/item?id=26696876 - April 2021 (108 comments) Backblaze Hard Drive Stats for 2020 - https://news.ycombinator.com/item?id=25917466 https://news.ycombinator.com/item?id=25917466 - Jan 2021 (175 comments) Backblaze Hard Drive Stats Q3 2020 - https://news.ycombinator.com/item?id=24838106 https://news.ycombinator.com/item?id=24838106 - Oct 2020 (112 comments) Backblaze Hard Drive Stats Q2 2020 - https://news.ycombinator.com/item?id=24200048 https://news.ycombinator.com/item?id=24200048 - Aug 2020 (119 comments) Backblaze Hard Drive Stats Q1 2020 - https://news.ycombinator.com/item?id=23156815 https://news.ycombinator.com/item?id=23156815 - May 2020 (95 comments) Hard Drive Stats for 2019 - https://news.ycombinator.com/item?id=22299251 https://news.ycombinator.com/item?id=22299251 - Feb 2020 (163 comments) Backblaze Hard Drive Stats Q3 2019 - https://news.ycombinator.com/item?id=21515084 https://news.ycombinator.com/item?id=21515084 - Nov 2019 (122 comments) What S.M.A.R.T Stats Can Tell You About a Business - https://news.ycombinator.com/item?id=21331995 https://news.ycombinator.com/item?id=21331995 - Oct 2019 (3 comments) Backblaze Hard Drive Stats Q2 2019 - https://news.ycombinator.com/item?id=20624464 https://news.ycombinator.com/item?id=20624464 - Aug 2019 (136 comments) Backblaze Hard Drive Stats Q1 2019 - https://news.ycombinator.com/item?id=19788157 https://news.ycombinator.com/item?id=19788157 - April 2019 (21 comments) Backblaze Hard Drive Stats for 2018 - https://news.ycombinator.com/item?id=18969264 https://news.ycombinator.com/item?id=18969264 - Jan 2019 (158 comments) Backblaze Drive Stats: 2018 Q3 Hard Drive Failure Rates - https://news.ycombinator.com/item?id=18229776 https://news.ycombinator.com/item?id=18229776 - Oct 2018 (49 comments) Hard Drive Stats for Q2 2018 - https://news.ycombinator.com/item?id=17601687 https://news.ycombinator.com/item?id=17601687 - July 2018 (16 comments) Backblaze's Hard Drive Stats for Q1 2018 - https://news.ycombinator.com/item?id=16967146 https://news.ycombinator.com/item?id=16967146 - May 2018 (30 comments) Backblaze Hard Drive Stats for 2017 - https://news.ycombinator.com/item?id=16283844 https://news.ycombinator.com/item?id=16283844 - Feb 2018 (31 comments) Hard Drive Stats for Q3 2017 - https://news.ycombinator.com/item?id=15559679 https://news.ycombinator.com/item?id=15559679 - Oct 2017 (47 comments) Applying medical statistics to Backblaze Hard Drive stats - https://news.ycombinator.com/item?id=15438052 https://news.ycombinator.com/item?id=15438052 - Oct 2017 (3 comments) Backblaze's Hard Drive Stats for Q2 2017 - https://news.ycombinator.com/item?id=15125340 https://news.ycombinator.com/item?id=15125340 - Aug 2017 (143 comments) Hard Drive Stats for Q1 2017 - https://news.ycombinator.com/item?id=14300634 https://news.ycombinator.com/item?id=14300634 - May 2017 (4 comments) Backblaze Hard Drive Stats for 2016 - https://news.ycombinator.com/item?id=13531768 https://news.ycombinator.com/item?id=13531768 - Jan 2017 (132 comments) Backblaze hard drive reliability stats for Q3 2016 - https://news.ycombinator.com/item?id=12959457 https://news.ycombinator.com/item?id=12959457 - Nov 2016 (110 comments) What SMART Stats Tell Us About Hard Drives - https://news.ycombinator.com/item?id=12655445 https://news.ycombinator.com/item?id=12655445 - Oct 2016 (68 comments) Hard Drive Stats for Q2 2016 - https://news.ycombinator.com/item?id=12210509 https://news.ycombinator.com/item?id=12210509 - Aug 2016 (57 comments) One Billion Drive Hours and Counting: Q1 2016 Hard Drive Stats - https://news.ycombinator.com/item?id=11712989 https://news.ycombinator.com/item?id=11712989 - May 2016 (135 comments) Backblaze Hard Drive Reliability Stats for Q3 2015 - https://news.ycombinator.com/item?id=10387962 https://news.ycombinator.com/item?id=10387962 - Oct 2015 (19 comments) Hard Drive Reliability Stats for Q1 2015 - https://news.ycombinator.com/item?id=9589234 https://news.ycombinator.com/item?id=9589234 - May 2015 (26 comments) Predicting Hard Drive Failures with SMART Stats - https://news.ycombinator.com/item?id=8745062 https://news.ycombinator.com/item?id=8745062 - Dec 2014 (20 comments) I actually thought there had been more of these.
- zachruss92 5y agoJust echoing the sentiment of loving Backblaze. The HDD stats really aligns with the open source roots of the company. I love that they also open source the designs of their storage pods. This "show your work" strategy helps me trust them at the end of the day. With this kind of storage being a commodity, a high level of openness could be a competitive advantage.