19 ms·
Backblaze Drive Stats for Q3 2023
- hiatus 3y agoSurprised by the number of drives with 0 failures, though it seems not all of the drives were run for the required time to qualify for the rating. > In Q3, six different drive models managed to have zero drive failures during the quarter. But only the 6TB Seagate, noted above, had over 50,000 drive days, our minimum standard for ensuring we have enough data to make the AFR plausible.
- kqr 3y agoI have always wondered why they don't use techniques from survival analysis to be able to draw conclusions even from sets with lower failure rates. Or for that matter to avoid slight bias even for drives they do report.
- andy4blaze 3y agoAndy Klein from Backblaze here. We have done some survival analysis (kaplan-meier curves). In our case, we need to have a reasonable number of failures over the observation period to get decent results. You can take a look at some of our work here: https://www.backblaze.com/blog/hard-drive-life-expectancy/ https://www.backblaze.com/blog/hard-drive-life-expectancy/ and see if that is what you were expecting.
- kqr 3y agoYes! Thank you. I've been thinking about scratching this itch myself but it turns out you did already. In particular the shape of the survival curve interests me -- you hear so many things about exponential here, bathtub there, but ver little data. I will read once I have a spare moment.
- nicolaslem 3y agoThe article discusses the impact of high absolute temperature on the longevity of drives, however from my amateur knowledge the range of temperature during a day is also an important factor. I always assumed that having a stable 40°C is better than a drive constantly swinging between 20°C and 40°C, so I am surprised that the article only mentions alerts on reaching a high threshold.
- andy4blaze 3y agoAndy Klein from Backblaze here. Your point is a good one in that temperature fluctuation can be an important factor. We actually sample smart stats, which contain the temperature attribute, multiple times a day looking for such changes. The Drives Stats data is captured once a day, so it looks static, but behind the scenes the monitoring is more dynamic.
- rft 3y agoDo you have plans to look at whether higher fluctuations translate into higher failure rates? Not sure whether you have historical data on this, but I would be really interested in this aspect, even if you can only run the stats on a smaller number of drives or shorter time periods. Maybe dividing the drives roughly in "higher than average variability" and "low variability" and then looking at the AFR for this subset can show some relation. Of course as the AFR for many drives is already quite low, the effect might be too small to distinguish from noise. On the topic of temperatures: Have you run an analysis whether a drive increasing in temperature (or maybe even decreasing) compared to its base line and "neighbors" results in a higher chance of failure?
- defrost 3y agoCount me in as a second to the question from rft. I'd be interested if you have data to compare similar drives with stable tempretures against diurnal tempreture cycling. I'd imagine you have fairly constant data centre type environments though which would confound analysis for such questions.
- hardware2win 3y agoKinda low sample sizes for 0 and highest AFRs
- dist-epoch 3y agoIs there any backup software which continuously uploads your files to an AWS/GCP/Azure long term storage account that you control and pay for? Something like CrashPlan, which from time to time performs automatic maintenance and deletes old versions of files.
- briffle 3y agoBackblaze software is pretty reasonable. But I have a linux machine, so restic and their B2 storage is a buck or two a month to backup a few computers in my house. (around 185GB of photos, etc)
- encom 3y ago>Backblaze software is pretty reasonable. Hard disagree. The jankiness of their software was what made me cancel my sub after several years. Firstly it doesn't follow the OS date and number formatting. It's a minor thing, but it's so annoying having to parse the dumb M/D/Y format and comma thousands separator etc (being da-DK). It's not a deal breaker, but on the other hand, it's such a low-hanging thing that most other software gets right immediately. But far more importantly, the BB client would some times just decide to re-upload several hundred gigabytes of data that I know for sure didn't change, which makes me wonder if it's just the client being retarded or if the data got lost server-side. And it takes absolutely forever to detect USB harddrives being plugged in. And its log files will grow to absurd sizes, and you're not allowed to purge them or the client will become brain damaged. And one time I needed to do a restore, it took literal days for BB to prepare it and I had to get support involved. I feel I just can't trust the BB stack, the client being the weakest link by far, and backup I can't trust is worthless.
- briffle 3y agoYou are probably correct. But there is a reason I put Backblaze client on my mom's laptop, rather than a scheduled powershell task to run restic to a cloud storage. And a scheduled task to run the prune weekly.
- 3y ago
- dehrmann 3y ago> The average age of the retired drives was just over eight years Didn't realize they can last so long.
- margalabargala 3y agoThey can last much longer. I have operational IDE drives from the early aughts (not with anything important on them)
- thebiss 3y agoUnder my desk right now is a too-underutilized-to-upgrade-NAS that has been spinning 1TB Western Digitals since 2010. Between RAID-Z2 and cloud backups, there's almost no reason to get rid of them except for performance, which doesn't matter here.
- toast0 3y agoIf all of your disks are about the same age, from the same vendor, it might be reasonable to replace them over time to mitigate the risk that a firmware or manufacturing issue results in them all failing around the same time. Many vendors have had firmware errors where counters rolled over and the drive becomes inaccessible (generally much sooner than 13 years though).
- pathartl 3y agoI just retired some drives out of my home array. 3TB WD Reds with 10.5 years of power on time, no logged errors. Ran them through a full block check and had no errors.
- nucleardog 3y agoI have a WD Black 1TB drive that did service as my main desktop drive for about 6 or 7 years. After that it went into my server at home. It was used for various things. At this point it’s spent the last 5 years as the disk my DVR records to. (Because 5 years ago I was expecting it to die any day and didn’t want anything more important on it…) So it’s being continuously written and rewritten 24 hours a day, 7 days a week. It’s now about 17 years old and has spent almost the entirety of that time powered on. It’s been packed up and sent on two cross country moves. Still kicking. I have a number of WD Green 1.5TB drives that are nearly as old and still in daily use in the same server. Maybe I’ve just had great luck, but I’d be more surprised by a drive dying sooner than eight years.
- Lorin 3y agoI wonder if they have special hardware recycling arrangements with their vendors for decommissioned drives to reduce their footprint. How many magnets are laying around the office? :)
- 1-6 3y agoMy simple rule still stands even after 20 years: Avoid Seagate.
- Mistletoe 3y agoI always found it hilarious when they were sponsoring the datahoarder subreddit. I’d like to meet that marketer and shake his or her hand.
- 1-6 3y agoI'm surprised that Seagate has consistently kept its 'slightly less reliable' crown after all these years. It's like the student who copies the 'A' student's exam. They'll purposefully get a couple of the answers wrong to avoid suspicion.
- kstrauser 3y agoConversely, I’ve had nothing but bad luck with WD the last few years, and my Seagates have been flawless. My simple rule is that all drives suck, and always have good backups.
- pcurve 3y agoYup. All drives kick the bucket at some point. That's why I use BB. I don't trust myself with NAS. The ones that never failed me were any drives made by Quantum using SCSI interface. 5 drives, zero failure over 10+ years. But those were slower and cooler running units.
- icelancer 3y agoThe best blog series going. Great technical writeups that I wish more companies would do - we've been doing it at our small business and customers really get a lot out of it. Also helps with marketing.
- vouaobrasil 3y agoBackblaze is interesting but it's not very easy to use. Its interface is rather basic and it was difficult to select which drives to back up and which not to. It kept trying to back up directories on my computer that I specifically told it not to, and there was no way to efficiently update program's behaviouir from the the "what to backup list". It might be nice if you just want to backup your computer and all your drives but the moment you want only parts of your computer backed up, it's frustrating.
- cvccvroomvroom 3y agoI run a bunch of WUH721414ALE6L4 and WUH721414ALE604 in numerous RAID10 volumes. Haven't had a failure yet. L = without power disable, 0 = with
- BOOSTERHIDROGEN 3y ago> this chart is the confidence interval, which is the difference between the low and high AFR confidence levels calculated at 95%. How to calculate low and high AFR ?
- Titan2189 3y ago> ambient temperature within a data center often increases during the summer months What? Isn't your Data Center supposed to be temperature controlled, where the A/C has a setpoint which it's keeping the entire environment to within a degree? Being able to tell what season it is based on your HDD SMART temperature (armchair expert here) sounds bad.
- calamari4065 3y agoClose to 10 years ago, I bought a "new" hard drive on Amazon. When I got it, I happened to notice some suspicious signs of wear, like corrosion on the PCB from someone's greasy fingerprint. I dive into the SMART stats and it's completely zeroed. Not quite what I'd expect from a new drive. So then I start scanning the drive. The partition table was deleted, but it was full of data, most of it encrypted. It was also badly fragmented, so it was likely in some type of array. What I could recover in cleartext implied that it had been spun up for thousands of hours, in a Backblaze datacenter. It wasn't conclusive enough to go to Backblaze about it, so I just returned it to Amazon. I probably did a zero pass on it first, can't remember. Encrypted or no, if that was a backblaze drive, they were disposing of drives with customer data still on them. I'm not surprised someone tried to pass it off as new on Amazon, that scam is old hat. I was very shocked to see the data still intact though.
- gruez 3y ago>it was full of data, most of it encrypted. [...] Encrypted or no, if that was a backblaze drive, they were disposing of drives with customer data still on them Who cares if there was "customer data still on them" if it was encrypted? One of the nice things about encryption is that you don't have to worry about wiping drives.
- jbverschoor 3y agoSure.. until it gets decrypted
- gruez 3y agoIs that a realistic threat with AES-256 and randomly generated keys?
- Dalewyn 3y agoThere's a reason that the industry standard for proper disposal of storage media, including HDDs, is nothing short of physical destruction. It doesn't matter if the data is encrypted or not, the point of the matter is the data is still there when presumably that data should not exist outside <X> premises. Encryption serves as a mitigation against theft or accidental leakage of data, its purpose is not to facilitate data disposal. Put another way, you have to answer Yes to this question for liability purposes: "Is the data gone?" The only way to say Yes with reasonable certainty is physically destroying the storage medium the data resides on.
- bimguy 3y agoCool article! Looks like HGST (formally Hitachi) perform very well overall. Toshiba 4GB ran 101 months with no failures, holy moly! I used to like Samsung hard drives personally, Lacie used them in their rugged series and I found them to have pretty low failure rates. Seagate bought out Samsung's disk drive business in 2011 apparently. I guess Samsung saw the future of SSD's?
- IYasha 3y agoAs much as I appreciate these articles... Guys... Please don't save up on ink! Eyes are more expensive than black pixel paint these days.