5 ms·
> The AFR for 2020 dropped below 1% down to 0.93%. In 2019, it stood at 1.89%. That’s over a 50% drop year over year... In other words, whether a drive was old
by jedberg 6y ago
> The AFR for 2020 dropped below 1% down to 0.93%. In 2019, it stood at 1.89%. That’s over a 50% drop year over year... In other words, whether a drive was old or new, or big or small, they performed well in our environment in 2020.
If every drive type, new and old, big and small, did better this year, maybe they changed something in their environment this year? Better cooling, different access patterns, etc.
If this change doesn't have an obvious root cause, I'd be interested in finding out what it is if I were Backblaze. It could be something they could optimize around even more.
- codezero 6y agoI was wondering if the state of the world in 2020 might have dramatically changed their business / throughput / access patterns in a meaningful enough way to cause this dip. I'm not sure if they have a measure of the disk utilization or read/write load along with the failure rate.
- brianwski 6y agoDisclaimer: I work at Backblaze, but mostly on the client that runs on desktops and laptops. > I'm not sure if Backblaze has a measure of the disk utilization or read/write load along with the failure rate. We publish the complete hard drive SMART stats for anybody to attempt these analysis. Most of us in Backblaze engineering get giddy with excitement when a new article comes out that looks at correlating SMART stats and failures. :-) For example, this article circulated widely at Backblaze a few days ago: https://datto.engineering/post/predicting-hard-drive-failure-with-machine-learning https://datto.engineering/post/predicting-hard-drive-failure...
- codezero 6y agoI anticipate these reports every year and have strong trust in the data - I want to make that clear - Backblaze has done a massive service to the entire industry by collecting and aggregating this kind of data. I'm really super curious about the dip in errors over the past year :)
- rootusrootus 6y agoWhether intentional or not, it's also great word-of-mouth advertising. My preexisting experience with Backblaze's hard drive stats reporting definitely worked positively in their favor when I was looking for a new backup service.
- brianwski 6y ago> Whether intentional or not, it's also great word-of-mouth advertising. Oh, this has really worked out for Backblaze and we know it. The first time we published our drive failure rates (I think January of 2014?) a few people said, "Uh oh, now Backblaze will get sued by the drive manufacturers." And we cringed and waited. :-) But the lawsuit never came, in fact there were NO repercussions, only increased visibility. People who have never heard of our company before find the data interesting, and then they ask "hey, what does this company do to own this many drives?" And a few (like yourself) sign up for the service. Existing customers seem to stick with us for a long time, and even recommend us to other friends and family from time to time. So one tech person who stumbles across these stats might ACTUALLY bring us 3 or 4 more customers over the next 5 years. That's real money to us. Not only have the drive manufacturers not sued us, they are actually NICE to us beyond the scale of our actual drive purchases! In one amusing example, our drive stats were used in a lawsuit as evidence. To be clear Backblaze was not the plaintiff or the defendant in the court case, we had no skin in the game at all and didn't want to be involved, but our drive data (and internal emails) were subpoenaed to be entered into evidence. Before we were served, the drive manufacturer called us and apologized for the inconvenience and made it clear they had no beef with us. Yes, a multi-national company that makes BILLIONS of dollars per year called a 40 person company (at the time) that could barely make payroll each month to apologize for the inconvenience. :-) We thought it was very considerate of them, and a little amusing. I'm proud to be the one that "signed" the papers indicating Backblaze had been "served".
- rootusrootus 6y agoClicked on your profile and noticed that you are a fellow Beaver. Go Beavs! ;-). I got my CS degree from OSU about 10 years after you graduated.
- matmatmatmat 6y agoHi Brian, just a note of thanks to you and Backblaze for publishing these data. I always refer to them before a purchase and they're really helpful.
- willis936 6y agoThere are other interesting factors to look for as well. Temperature, moisture, electrical noise on the power rails, infrasound, etc.
- rachelwenzel 6y agoWow- crazy to see people at Backblaze actually read that article!! Thank you guys for putting out so much data and writing so much about your findings- it was HUGE in helping me come to conclusions about what's realistic to assume from SMART stats. Y'all are doing some really really cool stuff.
- atYevP 6y agoYev here -> of course we read it! It's a data-rich article about data failure! And the kudos is mutual - loved the post! :D
- cm2187 6y agoOr perhaps as time passes, a greater portion of their storage is rarely accessed archives so more disks in % are sitting doing nothing.
- codezero 6y agoI’m just assuming that folks doing archival storage aren’t using these kinds of spiny disks as it would be super expensive compared to other mediums, right? I do think access patterns in general should contribute to the numbers so that kind of thing can be determined.
- cm2187 6y agoBut even if it is hot storage, do you touch all your files every day? It is bound that you accumulate over time more files that you never access.
- codezero 6y agoYeah hard to say, I assume the diversity of their customers and normal distribution should statistic those patterns out but I have no clue :) I also wonder if failures are relating to physical location on disk vs other things like controller or mechanical failures. You may not be reading old files but you’re reading files, or not, so still depends also on over all utilization.
- freeone3000 6y agoCompared to what, exactly? Tape is cheaper per GB, but the drives and libraries tip that over the other way. Blu-Ray discs are now more expensive per GB than hard drives, thanks to SMR and He offerings. Also note that Backblaze does backups -- by definition, these are infrequently accessed, usually write-once-read-never. I've personally been a customer for three years and performed a restore exactly once.
- guenthert 6y agoDespite claims to the contrary, tape isn't dead just yet. They are still considerably cheaper than drives. An LTO-8 tape (12TB uncompressed capacity) can be had for about $100, while a 12TB HDD goes for some $300. Tape drives/libraries are quite expensive though, but that just shifts the break-even point out. For the largest sites, its still economical. Not sure, if backblaze is big enough (I'm sure they did their numbers). backglacier anyone?
- andruby 6y agoI guess these Hard Drive Stats post cover disks used for their B2 service as well? Maybe the service mix is changing (a larger percentage being used for B2 versus their traditional backup service). I'm not sure how more B2 access pattern would improve the stat though.
- brianwski 6y agoDisclaimer: I work at Backblaze. > I guess these Hard Drive Stats post cover disks used for their B2 service as well? Yes. The storage layer is storing both Backblaze Personal Backup files and B2 files. It's COMPLETELY interleaved, every other file might be one or the other. Same storage. And we are reporting the failure rates of drives in that storage layer. We THINK (but don't know for certain) that the access patterns are relatively similar. For example, many of the 3rd party integrations that store files in B2 are backup programs and those will definitely have similar access patterns. However, B2 is used in some profoundly different applications, like the origin store for a Cloudflare fronted website. So that implies more "reads" than the average backup, and that could be changing the profile over time as that part of our business grows.
- numpad0 6y agoWhat about air quality? There are actually air filters in a form of a patch or a package similar to miniature mustard packets through which drives breathe. Supposedly those are super fine filters but toxic gas molecules might still pass through them.
- brianwski 6y ago> There are actually air filters ... through which drives breathe Although the helium drives are more sealed up, which also might be a factor?
- jeffbee 6y agoThat seems like a misleading aggregation. Their total AFR can have been affected just by mix shift from early death to mid-life. It looks that way to me from their tables.
- einpoklum 6y ago> If every drive type, new and old, big and small, did better this year, maybe they changed something in their environment this year? It can also be the case that newer drives this year are better than newer drives last year, while older drives are over a "hill" in the failure statistics, e.g. it could be the case that there are more 1st-year failures than 2nd-year failures (for a fixed number of drives starting the year).
- alinspired 6y agoperhaps margin of error should be raised to accommodate this change of about 1%, although the set of drives under test is likely not the same between years
- deeblering4 6y agoI wonder if datacenter covid restrictions had an impact? Less movement, less vibration, less breathing, datacenter doors staying closed longer, generally less planned maintenance, etc etc.