13 ms·

Hard Drive Reliability Update – Sep 2014

385 points by nuriaion 12y ago

michaelbuckbee 12y ago
Biggest takeaway was at the end, with the "enterprise" drives being slightly less reliable than the consumer ones at half the cost.
- masklinn 12y ago
  Second takeaway: if you want working HDDs, go with Hitachi.
  - moe 12y ago
    You are wrong. The second takeaway is: Check the last Backblaze report right before you buy new drives.
  - ars 12y ago
    But Hitachi is owned by Western Digital, and Western Digital hard disks are not doing quite as well.
    - linker3000 12y ago
      It's now called "HGST (a Western Digital company)" and the two product lines (WD and HGST HDDs) are totally separate in both parts and manufacturing. /HGST employee
      - gukov 12y ago
        What makes your drives so good?
        atYevP 12y ago
        Our guess is that they kiss every one of them before they leave the factory. For good luck...
        linker3000 12y ago
        Please do not publicize our trade secrets.
        atYevP 12y ago
        Shit.
      - Jetlag 12y ago
        Can you explain what "assets" Toshiba got to start their 3.5" drive business? Are HGST and Toshiba making very similar 3.5" drives?
        linker3000 12y ago
        Can't help with that one at all.
      - Someone1234 12y ago
        You don't work in the HDD division, right? Someone seriously needs to look into ADM (advanced power management) on your 2.5" drives. It is currently set too low and the drives power down instantly after each instruction, resulting in simply astonishingly terrible IO (and seriously lag spikes). It is so bad even running HD video off of some 2014 HGST laptop hard drives causes distortion. There's also no way to increase the ADM value indefinitely that I am aware of (it reverts after every sleep).
        linker3000 12y ago
        Nope, I do other stuff with Flash Platforms. Go through the support link on the web site and send in your comments.
    - deleted 12y ago
      [deleted]
    - atYevP 12y ago
      Yev from Backblaze here -> HGST is run by a completely different division w/ a different manufacturing line. So there are some difference!
      - atYevP 12y ago
        ^ *is some difference. #sigh
- VLM 12y ago
  And thats whats weird, who is the audience for the article? I know what enterprise drive means and I know the author is smart and knows what it means, so why the weird implications in the article that have nothing to do with "enterprise"? For those not "in the know" the hardware is the same, but desktop firmware drives will sit there for 10 seconds or whatever it is beating the drive when there's a read (or write) fail on the assumption that if your machine only has one drive you're better off trying as hard as possible to keep retrying until it works, and possibly the slowness will motivate them to replace (god forbid an end user have backups lol) Enterprise firmware, when it has a soft fail, just croaks as fast as possible. That lets the raid array hurry up and do its thing, or maybe even higher level replication do its thing. (edited to add the old startup adage of "fail quickly". Thats what enterprise drives do to keep overall array latency low, which is counter productive for consumer non-array drives) Aside from the firmware load the prices are different because usually enterprise has better guarantee and better service and unlike consumer drives which statistically are never replaced under guarantee so you can claim anything on paper for marketing purposes it won't cost anything, enterprise drives WILL get replaced and there will be a papertrail etc. So the guarantee for an enterprise drive actually costs something. Sometimes the firmware has some other subtle differences like how it handles recalibrates and scrubs (consumer home drives are like "too bad you get to wait on my schedule" and again, enterprise will go to some effort to eliminate array latency) My guess is the article is subtle astroturf by the winning drive mfgr?
  - gknoy 12y ago
    I'm not sure who the audience is, but thank you for your explanation of the differences between consumer and enterprise drive firmwares. That runs counter to my naive assumptions, which was that "enterprise" targeted things tend to be of higher reliability and quality than the consumer-targeted alternatives. I imagine someone in the drive / enterprise storage business would know better, but many of us might not have known that.
    - VLM 12y ago
      "tend to be of higher reliability and quality" Oh I should preface my remarks with "in family, same technology level" etc. I'm talking about the same package with the same storage where different firmwares are loaded. Thats the business model the OP is in, he makes posts about stripping consumer drives out of external cases and I don't want to speak for him but he radiates the impression of their whole secret sauce being reliability at the system level not the hardware level. It is true in my experience that "exotic hardware" marketed as enterprise grade is different physical hardware and is going to be a different level of reliability than consumer hardware. So the difference between an office NAS SATA 1 TB enterprise and consumer grade is a firmware load, and I think thats what the article is talking about. There is a big difference between a 15K SAS enterprise server drive and a 7200 SATA consumer drive at the hardware level, but that's not the article author's "thing".
- sliverstorm 12y ago
  I don't believe reliability has ever necessarily been an "enterprise" feature. The concrete features I understand "enterprise" drives as having are: 1) firmware is built with RAID in mind. This might sound weird, but consumer drives are more likely to have problems with RAID, just because they aren't designed for it. See, for example, the first WD Green drives crashing RAID clusters because some timeout was too high. 2) Longer warranty period 3) This may not be true anymore, but I believe they typically feature larger cache 4) ECC [1]: http://www.pantz.org/hardware/disks/what_makes_a_hard_drive_enterprise_class.html http://www.pantz.org/hardware/disks/what_makes_a_hard_drive_...
  - acdha 12y ago
    At every sales-pitch I've ever attended, the enterprise vendors mention “reliability” about every other word – the message that your boss will fire you for taking risks unless you buy their hardware is communicated with the subtlety of a solar eclipse. As for the RAID point, you'd very much want to find real data for that. I've heard similar folklore but have also heard plenty of lost data stories from enterprise disks in enterprise RAID. If I were building out a large storage farm, I'd strongly consider forgoing hardware RAID altogether and using something like ZFS so it'd be observable.
    - wmf 12y ago
      Being optimized for RAID basically means TLER and possibly never spinning down.
    - sliverstorm 12y ago
      heard plenty of lost data stories from enterprise disks in enterprise RAID If you've ever worked with enterprise RAID, that should not surprise you! RAID is powerful, but also so easy to botch...
saosebastiao 12y ago
Tangential: When are you going to offer a linux client?
- tachion 12y ago
  Even better, when will be FreeBSD client available?! :)
  - spindritf 12y ago
    You're pretty much morally obligated to use Tarsnap when running FreeBSD.
    - rakoo 12y ago
      You don't get the same space/price ratio, though...
      - mappu 12y ago
        Try attic-backup[1] to one of these[2] super-cheap VPSes. 2.5c/GB/month for a 50GB block. 1. https://attic-backup.org/ https://attic-backup.org/ 2. https://crissic.net/openvz_vps https://crissic.net/openvz_vps
        deleted 12y ago
        [deleted]
        icebraining 12y ago
        That's only cheap because it's a small block. If you need larger ones, you can get much cheaper. I pay 0.64c/GB/month on my 2TB storage with TransIP: https://www.transip.eu/vps/big-storage/ https://www.transip.eu/vps/big-storage/
- tambourine_man 12y ago
  Another potencial custumer here. I use CrashPlan only because I can run it on my RaspberryPi, even though they throttle my upload at 1MB/sec.
mercurialshark 12y ago
All my WD and Seagate drives have failed within two years of use. Call me the luckiest.
- B5geek 12y ago
  My small datacenter results mimic BackBlaze too. Dead/dying seagates all over the place. So I notified management that we will only be purchasing Hitachi drives from now on. I have a BackBlaze server that I recently converted to FreeBSD & ZFS. I love the drive-density that Backblaze offers but I HATE the lack of physical notification when a drive dies. Most ofther file-servers have a front-facing drive caddy, that usually has LEDs on the front to indicate disk access or errors. This is great because you can walk into the datacenter, and SEE which disk has failed. With the backBlaze system you can get /dev/DriveID but not know where in the array that particular disk is.
  - sounds 12y ago
    You definitely don't want to use drives from the same manufacturing run (batch) on the same array, since they are the most likely to fail all at the same time. Second to that, you probably don't want to go single-source for your drives -- maybe use Hitachi with a mix of WD.
    - rane 12y ago
      > Second to that, you probably don't want to go single-source for your drives -- maybe use Hitachi with a mix of WD. Can you explain why not?
      - bronson 12y ago
        Because then a single firmware bug could wipe you out. Once the drives all hit 4500 hours or 700 bad blocks or some other trigger, they die. It's happened before. Could also be caused by bad grease, or shoddy bearings, or pretty much anything. There is security in diversity.
archgrove 12y ago
Not that I use even 0.001% of the disks that BackBlaze go through, but my anecdata suggests the same. The only dead hard disks I have on my desk at the moment are Seagate, and they dominate the disks I've sent back in the last few years. However, they are cheap, and they do honour their warranties. Would just be nice if they didn't have to quite so much.
- atYevP 12y ago
  I think my new favorite phrase is going to be "anecdata". Love that.
cgore 12y ago
My main Linux box has quite a few hard drives in it from a large range of time. About 4 weeks ago the oldest of them all died: it is from 2007, so about 7 years old, which I think is pretty good for a consumer drive that's on 24/7. It was a Western Digital Caviar SE WD3200JB, 320GB. I replaced it with a 2TB drive. [No lost data, I do daily backups.]
- luckyno13 12y ago
  I just replaced a WD Black 640 3 years into its life. Blacks have a 5 year so they RMA'd it and sent me a 750 in its place. I do however have a Seagate drive laying around somewhere that has almost 10 years on it and it still functions flawlessly. But it is admittedly smaller given its age and that may contribute to lifespan. Either that or Seagate has slipped in the last decade.
  - sounds 12y ago
    Seagate apparently has slipped in the last 2-3 years. (Could this be related to the hard disk shortage due to flooding in Malaysia?) Hard drive failures tend to happen more for drives that are power cycled a lot, and for drives that undergo big swings in temperature (even when the temps are all within the rated temp range).
- agumonkey 12y ago
  I wonder what media people use to backup xTB NAS. Tapes ?
  - pessimizer 12y ago
    I back up my RAIDs to JBODs - so hard drive backups for hard drives, just fewer. I only use tape for vital and frequently backed up things - not entire drives, and generally not large.
cake 12y ago
I wish there was something similar for SSDs.
- deleted 12y ago
  [deleted]
- nl 12y ago
  There is, roughly. This is a lab test as opposed to production monitoring, but it is still interesting. Obviously SSDs and HDDs are quite different technically, so the appropriate test is quite different. They write data continuously to see how long they last. We've now written over a petabyte, and only half of the SSDs remain. Three drives failed at different points—and in different ways—before reaching the 1PB milestone[1] http://techreport.com/review/26523/the-ssd-endurance-experiment-casualties-on-the-way-to-a-petabyte http://techreport.com/review/26523/the-ssd-endurance-experim...
  - XzetaU8 12y ago
    Latest results: http://techreport.com/review/27062/the-ssd-endurance-experiment-only-two-remain-after-1-5pb http://techreport.com/review/27062/the-ssd-endurance-experim...
    - techrat 12y ago
      Also worth looking at: http://us.hardware.info/reviews/4178/10/hardwareinfo-tests-lifespan-of-samsung-ssd-840-250gb-tlc-ssd-updated-with-final-conclusion-final-update-20-6-2013 http://us.hardware.info/reviews/4178/10/hardwareinfo-tests-l... Hardware.info did a "Write until it dies" test for a couple of 250GB Samsung SSDs last year. They found that the drive consistently exceeds the 1000 writes per cell spec.
- cjensen 12y ago
  After a few hundred drives, our anecdata is that failure over time on SSDs is generally related to drive endurance. Make sure you use a SMART utility which can read (and translate to English) the current net usage of the drive. Throw them away when you get to 100% usage. I recently examined a set of Crucial m4s which were at 130% of usage. There was no lost data, but write bandwidth was hilariously bad (around 10-20MB/s).
shiftpgdn 12y ago
I manage a computation cluster for an oil and gas exploration company. We have a 50% failure (and rising!) of Seagate Constellation drives in 250GB, 1TB and 2TB configurations. My sample size is fairly small at a few hundred drives but man does it keep me busy.
- iSloth 12y ago
  Were they all purchased at the same time? Sounds more like a faulty batch or issue with the environment for a rate that high.
  - shiftpgdn 12y ago
    They've been purchased over a span of a few years so I doubt that's the case. They did have a single exposure to 110F ambient temperature for a few hours when the A/C to the server room went out which may be a contributing factor.
    - pixl97 12y ago
      I have found that to be very hard on disks. Lost a AC unit in summer and over the next month had a much higher rate of disk loss.
- tnorthcutt 12y ago
  50% annual failure rate, or 50% total, over several years?
  - shiftpgdn 12y ago
    50% total over 2-5 years.
zaroth 12y ago
Since annual failure rate is a function mostly of age, it would be interesting to see a line chart of cumulative failure rate vs age. But since new drives are continually being added to the population, there would be fewer drives in the data set as you moved up each curve. I guess you could calculate confidence intervals at quarterly intervals, and so the error bars would get larger as age increases and 'n' decreases. How would you calculate the CI for failure rate? It's not binomial or poisson, since failure rate goes to 1 over time... A little searching turns up http://rmod.ee.duke.edu/statistics.htm http://rmod.ee.duke.edu/statistics.htm which I'm sure completely explains how to do this... (rolls eyes). I hate that this is how statistics is commonly taught. Knowing which distribution to use and applying it correctly can actually be intuitive if taught properly. It doesn't always need to be an exercise in alphabet soup / deriving from base principles.
- dugmartin 12y ago
  It's been 20 years since I took a reliability engineering class but I believe the go to curve is the negative exponential. Here is a link from a quick Googling: http://www.quanterion.com/FAQ/Exponential_Dist.htm http://www.quanterion.com/FAQ/Exponential_Dist.htm
  - darkmighty 12y ago
    The exponential lifetime distribution is the model for ideal memoryless failures: the probability of failure in a dt interval is independent of current lifetime and have value lambda*dt. Those are as "random" failures as they get. I suppose hard drives are better modelled by a variable failure rate lamba(t), which should have a peak for the first few hours/days, settle down and then start growing quickly after a few months.
- jrapdx3 12y ago
  I'm not a statistician, but what you are describing is a situation in which the survival of the "subjects" (SSDs) is the crucial variable. How quickly the population declines is the question. This is similar to studies in medical research, e.g., how long do subjects live after an experimental cancer treatment. What I've seen used is the Kaplan-Meier non-parametric method (Kaplan-Meier plot) for the purpose. Maybe I'm just used to seeing these plots, but I find the method conveys information very effectively. More info can be found here: https://en.wikipedia.org/wiki/Kaplan%E2%80%93Meier_estimator https://en.wikipedia.org/wiki/Kaplan%E2%80%93Meier_estimator I agree that presenting the SSD life data in that way would be a good idea.
BradRuderman 12y ago
Let's get a blog post describing how they handled reimbursements for the drive farming. I imagine it to be incredibly complicated to cross reference a receipt with a product at that scale , especially since all the products were the same.
- andy4blaze 12y ago
  Andy from Backblaze here: Here's the post we did about that if you're interested: https://www.backblaze.com/blog/crowdsourcing-hard-drives/ https://www.backblaze.com/blog/crowdsourcing-hard-drives/
emodendroket 12y ago
How times have changed; Seagate used to be (or at least have the reputation of being) the most reliable and Hitachi the least.
- happycube 12y ago
  They bought Maxtor and kept the crappy Chinese plants. Interestingly they don't sell the Chinese 3.5" drives at Costco (where they can easily get returned) right now...
- jcampbell1 12y ago
  At some point, Hitachi Diskstars were referred to as "Death Stars", and that was all I knew about disk reliability. It is great to have some real information. Dell, Google, and Amazon can never write reports like this because the vendor relationship is important. Because these guys have no relationship and are buying consumer disks, the world finally gets brand level reliability reports. Kudos to Backblaze.
  - jonknee 12y ago
    > Dell, Google, and Amazon can never write reports like this because the vendor relationship is important. Because these guys have no relationship and are buying consumer disks, the world finally gets brand level reliability reports. Kudos to Backblaze. Dell maybe, but Google and Amazon consider what goes on in their data centers to be part of their secret sauce. It doesn't have to do with vendor relationships, it has to do with a competitive edge. Google and Amazon will drop a vendor without a second thought if it's even slightly more advantageous.
  - caycep 12y ago
    I think that was one IBM model way before Hitachi acquired that division from Big Blue; they've otherwise been pretty good.
  - hga 12y ago
    If I remember correctly they didn't name names, but Google did publish a very useful paper on their experiences with disks. The two big takeaways were that in only half the disks the self testing signaled a future failure, and manufactures seemed to have solved heat problems.
GreenGiant 12y ago
At a glance, it appears that the Hitachi's are far more reliable, but looking at the average age, we really see that the graph is worthless (as is most of the data provided). Average Age: 13.4 Seagate 7.3 Hitachi I'll skip the next "update" and save myself the 5 minutes of wasted time.
deleted 12y ago
[deleted]
justcommenting 12y ago
There are well established methods for time-to-failure and time-to-event data not used here. The author makes no effort to control for the multiple, obvious biases created by the analytical approach employed. A few simple graphs would give a much more telling view of these data.
- sillysaurus3 12y ago
  Would you mind listing some of those time-to-failure and time-to-event methods and how the author might control for them, and which graphs in particular the author should have included?
  - justcommenting 12y ago
    first, i should've mentioned from the outset that these are really interesting and useful data, and that i'm glad you took the time to generate them. i really wish consumers could systematically report these data in a way that was reliable/trustworthy...! cox proportional hazards models and KM survival curves are the big kahuna with a data set like this. basically, my impression is that you'd want to pretend that you're doing a cohort study and essentially analyze it like a big clinical trial. and re: graphing, since you have sub-samples of the different drive models and relatively different but still small numbers of models for each brand, the big take-away graph comparing manufacturers leaves out a lot of useful nuances that could be pulled out of the table, e.g. that most of the hitachi drives are newer and you have fewer of them. it also doesn't portray how consistent failure rates are across models produced by the same brand might be, and it looks like there's a significant range of failure rates across different seagate models. so even just seeing IQRs of pooled failure rates for each manufacturer in box plots might be eye opening.. immortal time bias may be something to consider here as well...when you have some sample groups that mostly include newer individual drives that you have not yet had for a year on average, subtle differences in how the failure event is described can make a big difference in the conclusions you draw...especially in terms of uncertainty. if i have 10,000 hitachi drives and 5 of them fail in the first 6 months, the robustness of conclusions i can draw from those data are different in some important ways from similar insights drawn from a sample of 1,000 hitachi drives i've used for 5 years. it's also not clear to me how you've dealt with replacement drives. based on what i gather from the post (and i could be totally wrong), if you have a bunch of one model of a drive failing and then get replacements for them... some might argue that refurbed drives should be analyzed almost like a separate drive model, since they often have physical differences compared to those bought via retail channels. i'd be quite curious to dig into these data a bit further if you're willing to post the original data set... thanks again for posting information that's quite useful and interesting
shadeless 12y ago
I recently bought 3TB Western Digital Red, following their advice from [1], but now I see that it has yearly failure rate of 8.8%, bummer. Off-topic, but It's a shame that BackBlaze isn't available in some countries, I'd love to use it. What would be the best alternative to it, Tarsnap? [1] https://www.backblaze.com/blog/what-hard-drive-should-i-buy/ https://www.backblaze.com/blog/what-hard-drive-should-i-buy/
rancor 12y ago
Used in a small file server, my net failure rate on Seagate's consumer 3TB drives has been over 50% thus far. The pair of their SAS drives I currently have in use have been fine, although both of them are still below a year of service life... Edit: Just checked my drive status, and yet another one has dropped. If I'm doing my math correctly, that's 75% of the drives that weren't DOA...
tambourine_man 12y ago
I love reading these posts from Backblaze, but what I never understand is that they are getting a cost of U$ ~0.05/GB with their storage pods: https://www.backblaze.com/blog/why-now-is-the-time-for-backblaze-to-build-a-270-tb-storage-pod/ https://www.backblaze.com/blog/why-now-is-the-time-for-backb... At these rates, why not use S3? What am I missing?
- bstar77 12y ago
  Perhaps being in control of your own destiny?
- frankchn 12y ago
  Amazon S3 is US$0.0275/GB per month at high volumes. I assume Backblaze's operational cost of keeping each Backblaze pod hooked up and online is much much less than US$0.05/GB/month.
  - tambourine_man 12y ago
    You're right, sorry. I'm so used to seeing U$/month I didn't realize it was the per GB cost for building the pod.
- brianwski 12y ago
  Disclaimer- I work at Backblaze. Here is a chart of the cost of a Petabyte that shows why we cannot use Amazon S3: https://www.backblaze.com/blog/wp-content/uploads/2009/08/cost-of-a-petabyte-chart.jpg https://www.backblaze.com/blog/wp-content/uploads/2009/08/co... It is from our original blog post about our storage pods (we assemble 45 hard drives in some sheet metal with off the shelf parts): https://www.backblaze.com/blog/petabytes-on-a-budget-how-to-build-cheap-cloud-storage-2/ https://www.backblaze.com/blog/petabytes-on-a-budget-how-to-...
larrys 12y ago
Question for the OP here (or for anyone else). Do you burn in new drives before using? I typically will take any new drive and do some type of stress test [1] on it for 18 to 24 hours to see if it fails with that initial constant use. [1] Constant reformatting for example writing 0's to the entire disk 7 times etc.
- rancor 12y ago
  Yes, these days I'll make sure that it sees a decent amount of power on time and several full drive writes before ever trusting data to it.
- atYevP 12y ago
  Yev from Backblaze here -> Yes, we do burn in the drives before deploying them. So the drives in the study are ones that have at least made it past that state.
  - larrys 12y ago
    Would be interesting to see a post on what your burn-in procedure is.
  - pbhjpbhj 12y ago
    Doesn't that negate a lot of the usefulness of the data? If most Hitachis fail in the burn-in but all Seagates pass it then the expected failure of a drive would be wildly different to that suggested by the charts. Do you have the figures for raw losses including at burn-in? I mean, sure, you burn-in and do warranty returns so buying Hitachi would still seem better - but if one needs a drive to just work then it's key to know the overall failure rate.
    - brianwski 12y ago
      Brian from Backblaze here. Let's call that "DOA Failure rate", where the drive is either dead on arrival, or does not survive the 2 day initial burn in test. I'll ask the boots on the ground for more data, but my impression is the 2 day burn in doesn't catch many drive failures, let's say less than 1/2 of 1 percent. We have even proposed skipping it, but we gain confidence the REST of the system is wired up correctly and other components like power supplies also have infant mortality.
ChuckMcM 12y ago
One of the challenges I have with this analysis is that a 'failure' isn't just that your drive is no longer working, it is that your drive isn't working and you have to go replace it. The operational costs of replacing a drive have three parts, loss of production while the drive is offline, operator time to physically replace the drive and prep it for re-entry into the system, and transactional costs of doing a warranty replacement (filling out the RMA form, getting a valid RMA, shipping the and receiving replacements). We minimize the latter by doing RMAs in batches of 20 but its still a cost across those 20 drives. (and the population of 40 drives which exist as spares are effectively not available for production). It isn't as simple as 'sure drives fail a bit more often but we don't expect to use them that long.'
- acdha 12y ago
  That's a good point but also requires a more complicated analysis since it requires you to account for your ops capacity (e.g. the marginal cost increase of doing 10 drives instead of 5 is probably a lot less than 2 unless your ops team is almost completely booked) and also requires some guessing about whether there actually is a better option. They addressed that second point with the discussion of enterprise drives, which certainly matched my own experience.
disordr 12y ago
I really want to applaud backblaze for publishing these reports and stats. Too many companies closely guard this information that really helps the larger community. Based on the previous blogs from backblaze, when I built out our new hadoop cluster, I purchased 1450 Hitachi drives. I plan to gather our failure rates and publish them as backblaze does. Thanks for blazing the path!
- atYevP 12y ago
  Yev from Backblaze -> Thanks! That really is one of our goals with these updates, is for others to join us and start sharing this data. It makes a lot more sense if everyone is doing it, then we can start comparing environments, and all sorts of fun stuff!
  - Serow225 12y ago
    For the Seagate 7200.14, have you checked to see if you have the latest firmware on all of them? http://knowledge.seagate.com/articles/en_US/FAQ/223651en http://knowledge.seagate.com/articles/en_US/FAQ/223651en Same for the 7200.11: http://knowledge.seagate.com/articles/en_US/FAQ/207951en http://knowledge.seagate.com/articles/en_US/FAQ/207951en
    - atYevP 12y ago
      Yup, we have top-men that are in charge of keeping an eye on firmware, if we see a need in an update, we'll do it.
      - Serow225 12y ago
        Have you been able to reach out to anyone at Seagate in their drive FA group to see if they'd be interested in you sending back samples of the failed drives?
        mey 12y ago
        I assume (mentioned in the article) they get sent back for warranty RMA.
sauere 12y ago
Hard drive age a bad parameter to use. It should be the hours the drive was actually powered on.
- jamroom 12y ago
  I have a feeling that with Backblaze that ALL of their drives are mostly powered on continuously until they fail (i.e. they might sit for a bit at their data center waiting to go into one of their pods, but after that they pretty much run until they fail).
  - sauere 12y ago
    If thats the case, okay. Obviously still not ideal - you would want to have even IO load distribution on all drives - but hey, better than nothing.
  - brianwski 12y ago
    Backblaze Employee here - this is correct. Pods get powered up, then pummeled with customer data until full. Then they stay spinning forever, but the rate of writes drops to "churn" rates for the rest of the drive's life. This use pattern is all we have experience with, so if your application is different (maybe a database that gets heavy load forever) then your outcomes may vary.
makmanalp 12y ago
Anyone have any reliability information on hitachi's new NAS drive series? They're supposed to build on the 7k3000 etc, but specifically tailored for NAS / raid situations, like WD reds. One major difference is that they're 7200 rpm instead of 5400 which is most non-high-end NAS drives.
- linker3000 12y ago
  http://www.hgst.com/hard-drives/internal-drive-kits/nas-desktop-drive-kit http://www.hgst.com/hard-drives/internal-drive-kits/nas-desk... Specs and data sheets
arb99 12y ago
Very off topic, but their html is wrong: "<a href='https://www.backblaze.com/blog/hard-drive-reliability-update-september-2014'><img https://www.backblaze.com/blog/hard-drive-reliability-update... src='https://www.backblaze.com/blog/wp-content/uploads/2014/09/blog-fail-drives-manufacture-report2.jpg' https://www.backblaze.com/blog/wp-content/uploads/2014/09/bl... alt='Hard Drive Failure Rates by Model' width='560px' border='0' /></a>" should be "width='560'" not "width='560px'"
- atYevP 12y ago
  Yev from Backblaze here -> It should work with the 560px, if you're referring to the "share this" html code!
ars 12y ago
No Toshiba hard disks apparently. HGST and Wester Digital are the same company, but it seems they have separate product lines? It's confusing.
- ShinyObject 12y ago
  The Hitachi drives are Toshiba drives. The 3.5" desktop and server Hitachi drives are all manufactured by Toshiba. WD wanted to buy Hitachi's drive division so there would be a WD/Seagate drive duopoly, European regulators said they had to sell Hitachi's 3.5" to someone else before they would approve the merger. Toshiba stepped up and got it. http://www.anandtech.com/show/5635/western-digital-to-sell-hitachis-35-hard-drive-business-to-toshiba-complete-hitachi-buyout http://www.anandtech.com/show/5635/western-digital-to-sell-h... So yes it is somewhat confusing. You buy one of these drives and it will say "HGST a Western Digital company" on the box, when in reality it's all made by Toshiba, probably in an old Hitachi factory.
  - hga 12y ago
    There's a much less sinister reason for WD buying HGST: SCSI/SAS support. Way back when in the '90s WD tried to enter that market, and a company I was at even got some of those drives, but from what I heard while their read rate was OK their write rates weren't competitive with Seagate and IBM (which sold their disk drive business to Hitachi after the Deathstar screwup). And they exited that segment of the business, and hadn't, last time I checked, returned to it.
  - blahblah234 12y ago
    Toshiba drives are not exactly Hitachi drives. First, Toshiba had their own 3.5" drives long before the merger with Hitachi. For example, Toshiba MK2002TSKB 2TB 3.5" hard drive was on sale since 2011. Toshiba's own design and factories wouldn't magically disappear after acquiring Hitachi's assets. So, after the merger, Toshiba sells some ex-Hitachi drives, for example toshiba DT01ACA300 3TB is obviously a relabeled Hitachi drive. But I believe that Toshiba MD03ACA300 3TB and Toshiba MD04ACA300 3TB drives are based on Toshiba's own design because they don't look like Hitachi or DT01ACA300. I suppose reliability and performance will be different between these three Toshiba 3TB drives. And it would be interesting to get some info on this. Also, HGST is wholly owned by Western Digital. Despite regulators requirement, they didn't sell all 3.5" assets to Toshiba, only some of them. Most of the good stuff (that is, everything currently sold under HGST brand) still went to WD. So, even ex-Hitachi drives sold under Toshiba brand (by Toshiba) and under HGST brand (by WD) might have different reliability.
robomartin 12y ago
Thanks for sharing such useful data. I just had a Seagate drive fail. Was able to recover data since the last local backup with various tools. It took hours of repair work. I've been procrastinating about getting off-site backup. This post on HN reminded me that I've been meaning to get an account going with your company for a while. I just signed up and will test on my machine before deploying to other machines in my business. Thank you.
TheLoneWolfling 12y ago
I'd be interested to see what a graph of percentage remaining versus time since installation looks like for these. Might give a better picture of what's going on.
BuckRogers 12y ago
I keep seeing this stuff over the years, so I go buy something other than Seagate like WD... and they fail within a year. So I replace it with a Seagate and no problems for years. See another report that says Seagate is terrible- repeat process. I'm just going to keep using Seagate until my anecdata refutes the reality I live in.
- atYevP 12y ago
  Yev from Backblaze here -> all of the drives in our datacenter will fail. We just report on the timing of those failures. We still buy Seagates in droves because they are a great value. As long as you have backups of the data, any drive is great!
GravityWell 12y ago
This is extraordinarily useful and unique. My compliments to Backblaze for making this available. This is the type of empirical data I would love to have for as many things as possible: SSDs, monitors, TVs, kitchen appliances, tablets, cars, etc.

385 points by nuriaion 12y ago

Biggest takeaway was at the end, with the "enterprise" drives being slightly less reliable than the consumer ones at half the cost.

masklinn 12y ago
Second takeaway: if you want working HDDs, go with Hitachi.
- moe 12y ago
  You are wrong. The second takeaway is: Check the last Backblaze report right before you buy new drives.
- ars 12y ago
  But Hitachi is owned by Western Digital, and Western Digital hard disks are not doing quite as well.
  - linker3000 12y ago
    It's now called "HGST (a Western Digital company)" and the two product lines (WD and HGST HDDs) are totally separate in both parts and manufacturing. /HGST employee
    - gukov 12y ago
      What makes your drives so good?
      - atYevP 12y ago
        Our guess is that they kiss every one of them before they leave the factory. For good luck...
        linker3000 12y ago
        Please do not publicize our trade secrets.
        atYevP 12y ago
        Shit.
    - Jetlag 12y ago
      Can you explain what "assets" Toshiba got to start their 3.5" drive business? Are HGST and Toshiba making very similar 3.5" drives?
      - linker3000 12y ago
        Can't help with that one at all.
    - Someone1234 12y ago
      You don't work in the HDD division, right? Someone seriously needs to look into ADM (advanced power management) on your 2.5" drives. It is currently set too low and the drives power down instantly after each instruction, resulting in simply astonishingly terrible IO (and seriously lag spikes). It is so bad even running HD video off of some 2014 HGST laptop hard drives causes distortion. There's also no way to increase the ADM value indefinitely that I am aware of (it reverts after every sleep).
      - linker3000 12y ago
        Nope, I do other stuff with Flash Platforms. Go through the support link on the web site and send in your comments.
  - deleted 12y ago
    [deleted]
  - atYevP 12y ago
    Yev from Backblaze here -> HGST is run by a completely different division w/ a different manufacturing line. So there are some difference!
    - atYevP 12y ago
      ^ *is some difference. #sigh
VLM 12y ago
And thats whats weird, who is the audience for the article? I know what enterprise drive means and I know the author is smart and knows what it means, so why the weird implications in the article that have nothing to do with "enterprise"? For those not "in the know" the hardware is the same, but desktop firmware drives will sit there for 10 seconds or whatever it is beating the drive when there's a read (or write) fail on the assumption that if your machine only has one drive you're better off trying as hard as possible to keep retrying until it works, and possibly the slowness will motivate them to replace (god forbid an end user have backups lol) Enterprise firmware, when it has a soft fail, just croaks as fast as possible. That lets the raid array hurry up and do its thing, or maybe even higher level replication do its thing. (edited to add the old startup adage of "fail quickly". Thats what enterprise drives do to keep overall array latency low, which is counter productive for consumer non-array drives) Aside from the firmware load the prices are different because usually enterprise has better guarantee and better service and unlike consumer drives which statistically are never replaced under guarantee so you can claim anything on paper for marketing purposes it won't cost anything, enterprise drives WILL get replaced and there will be a papertrail etc. So the guarantee for an enterprise drive actually costs something. Sometimes the firmware has some other subtle differences like how it handles recalibrates and scrubs (consumer home drives are like "too bad you get to wait on my schedule" and again, enterprise will go to some effort to eliminate array latency) My guess is the article is subtle astroturf by the winning drive mfgr?
- gknoy 12y ago
  I'm not sure who the audience is, but thank you for your explanation of the differences between consumer and enterprise drive firmwares. That runs counter to my naive assumptions, which was that "enterprise" targeted things tend to be of higher reliability and quality than the consumer-targeted alternatives. I imagine someone in the drive / enterprise storage business would know better, but many of us might not have known that.
  - VLM 12y ago
    "tend to be of higher reliability and quality" Oh I should preface my remarks with "in family, same technology level" etc. I'm talking about the same package with the same storage where different firmwares are loaded. Thats the business model the OP is in, he makes posts about stripping consumer drives out of external cases and I don't want to speak for him but he radiates the impression of their whole secret sauce being reliability at the system level not the hardware level. It is true in my experience that "exotic hardware" marketed as enterprise grade is different physical hardware and is going to be a different level of reliability than consumer hardware. So the difference between an office NAS SATA 1 TB enterprise and consumer grade is a firmware load, and I think thats what the article is talking about. There is a big difference between a 15K SAS enterprise server drive and a 7200 SATA consumer drive at the hardware level, but that's not the article author's "thing".
sliverstorm 12y ago
I don't believe reliability has ever necessarily been an "enterprise" feature. The concrete features I understand "enterprise" drives as having are: 1) firmware is built with RAID in mind. This might sound weird, but consumer drives are more likely to have problems with RAID, just because they aren't designed for it. See, for example, the first WD Green drives crashing RAID clusters because some timeout was too high. 2) Longer warranty period 3) This may not be true anymore, but I believe they typically feature larger cache 4) ECC [1]: http://www.pantz.org/hardware/disks/what_makes_a_hard_drive_enterprise_class.html http://www.pantz.org/hardware/disks/what_makes_a_hard_drive_...
- acdha 12y ago
  At every sales-pitch I've ever attended, the enterprise vendors mention “reliability” about every other word – the message that your boss will fire you for taking risks unless you buy their hardware is communicated with the subtlety of a solar eclipse. As for the RAID point, you'd very much want to find real data for that. I've heard similar folklore but have also heard plenty of lost data stories from enterprise disks in enterprise RAID. If I were building out a large storage farm, I'd strongly consider forgoing hardware RAID altogether and using something like ZFS so it'd be observable.
  - wmf 12y ago
    Being optimized for RAID basically means TLER and possibly never spinning down.
  - sliverstorm 12y ago
    heard plenty of lost data stories from enterprise disks in enterprise RAID If you've ever worked with enterprise RAID, that should not surprise you! RAID is powerful, but also so easy to botch...

saosebastiao 12y ago

Tangential: When are you going to offer a linux client?

tachion 12y ago
Even better, when will be FreeBSD client available?! :)
- spindritf 12y ago
  You're pretty much morally obligated to use Tarsnap when running FreeBSD.
  - rakoo 12y ago
    You don't get the same space/price ratio, though...
    - mappu 12y ago
      Try attic-backup[1] to one of these[2] super-cheap VPSes. 2.5c/GB/month for a 50GB block. 1. https://attic-backup.org/ https://attic-backup.org/ 2. https://crissic.net/openvz_vps https://crissic.net/openvz_vps
      - deleted 12y ago
        [deleted]
      - icebraining 12y ago
        That's only cheap because it's a small block. If you need larger ones, you can get much cheaper. I pay 0.64c/GB/month on my 2TB storage with TransIP: https://www.transip.eu/vps/big-storage/ https://www.transip.eu/vps/big-storage/
tambourine_man 12y ago
Another potencial custumer here. I use CrashPlan only because I can run it on my RaspberryPi, even though they throttle my upload at 1MB/sec.

mercurialshark 12y ago

All my WD and Seagate drives have failed within two years of use. Call me the luckiest.

B5geek 12y ago
My small datacenter results mimic BackBlaze too. Dead/dying seagates all over the place. So I notified management that we will only be purchasing Hitachi drives from now on. I have a BackBlaze server that I recently converted to FreeBSD & ZFS. I love the drive-density that Backblaze offers but I HATE the lack of physical notification when a drive dies. Most ofther file-servers have a front-facing drive caddy, that usually has LEDs on the front to indicate disk access or errors. This is great because you can walk into the datacenter, and SEE which disk has failed. With the backBlaze system you can get /dev/DriveID but not know where in the array that particular disk is.
- sounds 12y ago
  You definitely don't want to use drives from the same manufacturing run (batch) on the same array, since they are the most likely to fail all at the same time. Second to that, you probably don't want to go single-source for your drives -- maybe use Hitachi with a mix of WD.
  - rane 12y ago
    > Second to that, you probably don't want to go single-source for your drives -- maybe use Hitachi with a mix of WD. Can you explain why not?
    - bronson 12y ago
      Because then a single firmware bug could wipe you out. Once the drives all hit 4500 hours or 700 bad blocks or some other trigger, they die. It's happened before. Could also be caused by bad grease, or shoddy bearings, or pretty much anything. There is security in diversity.

archgrove 12y ago

Not that I use even 0.001% of the disks that BackBlaze go through, but my anecdata suggests the same. The only dead hard disks I have on my desk at the moment are Seagate, and they dominate the disks I've sent back in the last few years. However, they are cheap, and they do honour their warranties. Would just be nice if they didn't have to quite so much.

atYevP 12y ago
I think my new favorite phrase is going to be "anecdata". Love that.

cgore 12y ago

My main Linux box has quite a few hard drives in it from a large range of time. About 4 weeks ago the oldest of them all died: it is from 2007, so about 7 years old, which I think is pretty good for a consumer drive that's on 24/7. It was a Western Digital Caviar SE WD3200JB, 320GB. I replaced it with a 2TB drive. [No lost data, I do daily backups.]

luckyno13 12y ago
I just replaced a WD Black 640 3 years into its life. Blacks have a 5 year so they RMA'd it and sent me a 750 in its place. I do however have a Seagate drive laying around somewhere that has almost 10 years on it and it still functions flawlessly. But it is admittedly smaller given its age and that may contribute to lifespan. Either that or Seagate has slipped in the last decade.
- sounds 12y ago
  Seagate apparently has slipped in the last 2-3 years. (Could this be related to the hard disk shortage due to flooding in Malaysia?) Hard drive failures tend to happen more for drives that are power cycled a lot, and for drives that undergo big swings in temperature (even when the temps are all within the rated temp range).
agumonkey 12y ago
I wonder what media people use to backup xTB NAS. Tapes ?
- pessimizer 12y ago
  I back up my RAIDs to JBODs - so hard drive backups for hard drives, just fewer. I only use tape for vital and frequently backed up things - not entire drives, and generally not large.

cake 12y ago

I wish there was something similar for SSDs.

deleted 12y ago
[deleted]
nl 12y ago
There is, roughly. This is a lab test as opposed to production monitoring, but it is still interesting. Obviously SSDs and HDDs are quite different technically, so the appropriate test is quite different. They write data continuously to see how long they last. We've now written over a petabyte, and only half of the SSDs remain. Three drives failed at different points—and in different ways—before reaching the 1PB milestone[1] http://techreport.com/review/26523/the-ssd-endurance-experiment-casualties-on-the-way-to-a-petabyte http://techreport.com/review/26523/the-ssd-endurance-experim...
- XzetaU8 12y ago
  Latest results: http://techreport.com/review/27062/the-ssd-endurance-experiment-only-two-remain-after-1-5pb http://techreport.com/review/27062/the-ssd-endurance-experim...
  - techrat 12y ago
    Also worth looking at: http://us.hardware.info/reviews/4178/10/hardwareinfo-tests-lifespan-of-samsung-ssd-840-250gb-tlc-ssd-updated-with-final-conclusion-final-update-20-6-2013 http://us.hardware.info/reviews/4178/10/hardwareinfo-tests-l... Hardware.info did a "Write until it dies" test for a couple of 250GB Samsung SSDs last year. They found that the drive consistently exceeds the 1000 writes per cell spec.
cjensen 12y ago
After a few hundred drives, our anecdata is that failure over time on SSDs is generally related to drive endurance. Make sure you use a SMART utility which can read (and translate to English) the current net usage of the drive. Throw them away when you get to 100% usage. I recently examined a set of Crucial m4s which were at 130% of usage. There was no lost data, but write bandwidth was hilariously bad (around 10-20MB/s).

shiftpgdn 12y ago

I manage a computation cluster for an oil and gas exploration company. We have a 50% failure (and rising!) of Seagate Constellation drives in 250GB, 1TB and 2TB configurations. My sample size is fairly small at a few hundred drives but man does it keep me busy.

iSloth 12y ago
Were they all purchased at the same time? Sounds more like a faulty batch or issue with the environment for a rate that high.
- shiftpgdn 12y ago
  They've been purchased over a span of a few years so I doubt that's the case. They did have a single exposure to 110F ambient temperature for a few hours when the A/C to the server room went out which may be a contributing factor.
  - pixl97 12y ago
    I have found that to be very hard on disks. Lost a AC unit in summer and over the next month had a much higher rate of disk loss.
tnorthcutt 12y ago
50% annual failure rate, or 50% total, over several years?
- shiftpgdn 12y ago
  50% total over 2-5 years.

zaroth 12y ago

Since annual failure rate is a function mostly of age, it would be interesting to see a line chart of cumulative failure rate vs age. But since new drives are continually being added to the population, there would be fewer drives in the data set as you moved up each curve. I guess you could calculate confidence intervals at quarterly intervals, and so the error bars would get larger as age increases and 'n' decreases. How would you calculate the CI for failure rate? It's not binomial or poisson, since failure rate goes to 1 over time... A little searching turns up http://rmod.ee.duke.edu/statistics.htm http://rmod.ee.duke.edu/statistics.htm which I'm sure completely explains how to do this... (rolls eyes). I hate that this is how statistics is commonly taught. Knowing which distribution to use and applying it correctly can actually be intuitive if taught properly. It doesn't always need to be an exercise in alphabet soup / deriving from base principles.

dugmartin 12y ago
It's been 20 years since I took a reliability engineering class but I believe the go to curve is the negative exponential. Here is a link from a quick Googling: http://www.quanterion.com/FAQ/Exponential_Dist.htm http://www.quanterion.com/FAQ/Exponential_Dist.htm
- darkmighty 12y ago
  The exponential lifetime distribution is the model for ideal memoryless failures: the probability of failure in a dt interval is independent of current lifetime and have value lambda*dt. Those are as "random" failures as they get. I suppose hard drives are better modelled by a variable failure rate lamba(t), which should have a peak for the first few hours/days, settle down and then start growing quickly after a few months.
jrapdx3 12y ago
I'm not a statistician, but what you are describing is a situation in which the survival of the "subjects" (SSDs) is the crucial variable. How quickly the population declines is the question. This is similar to studies in medical research, e.g., how long do subjects live after an experimental cancer treatment. What I've seen used is the Kaplan-Meier non-parametric method (Kaplan-Meier plot) for the purpose. Maybe I'm just used to seeing these plots, but I find the method conveys information very effectively. More info can be found here: https://en.wikipedia.org/wiki/Kaplan%E2%80%93Meier_estimator https://en.wikipedia.org/wiki/Kaplan%E2%80%93Meier_estimator I agree that presenting the SSD life data in that way would be a good idea.

BradRuderman 12y ago

Let's get a blog post describing how they handled reimbursements for the drive farming. I imagine it to be incredibly complicated to cross reference a receipt with a product at that scale , especially since all the products were the same.

andy4blaze 12y ago
Andy from Backblaze here: Here's the post we did about that if you're interested: https://www.backblaze.com/blog/crowdsourcing-hard-drives/ https://www.backblaze.com/blog/crowdsourcing-hard-drives/

emodendroket 12y ago

How times have changed; Seagate used to be (or at least have the reputation of being) the most reliable and Hitachi the least.

happycube 12y ago
They bought Maxtor and kept the crappy Chinese plants. Interestingly they don't sell the Chinese 3.5" drives at Costco (where they can easily get returned) right now...
jcampbell1 12y ago
At some point, Hitachi Diskstars were referred to as "Death Stars", and that was all I knew about disk reliability. It is great to have some real information. Dell, Google, and Amazon can never write reports like this because the vendor relationship is important. Because these guys have no relationship and are buying consumer disks, the world finally gets brand level reliability reports. Kudos to Backblaze.
- jonknee 12y ago
  > Dell, Google, and Amazon can never write reports like this because the vendor relationship is important. Because these guys have no relationship and are buying consumer disks, the world finally gets brand level reliability reports. Kudos to Backblaze. Dell maybe, but Google and Amazon consider what goes on in their data centers to be part of their secret sauce. It doesn't have to do with vendor relationships, it has to do with a competitive edge. Google and Amazon will drop a vendor without a second thought if it's even slightly more advantageous.
- caycep 12y ago
  I think that was one IBM model way before Hitachi acquired that division from Big Blue; they've otherwise been pretty good.
- hga 12y ago
  If I remember correctly they didn't name names, but Google did publish a very useful paper on their experiences with disks. The two big takeaways were that in only half the disks the self testing signaled a future failure, and manufactures seemed to have solved heat problems.

GreenGiant 12y ago

At a glance, it appears that the Hitachi's are far more reliable, but looking at the average age, we really see that the graph is worthless (as is most of the data provided). Average Age: 13.4 Seagate 7.3 Hitachi I'll skip the next "update" and save myself the 5 minutes of wasted time.

deleted 12y ago

[deleted]

justcommenting 12y ago

There are well established methods for time-to-failure and time-to-event data not used here. The author makes no effort to control for the multiple, obvious biases created by the analytical approach employed. A few simple graphs would give a much more telling view of these data.

sillysaurus3 12y ago
Would you mind listing some of those time-to-failure and time-to-event methods and how the author might control for them, and which graphs in particular the author should have included?
- justcommenting 12y ago
  first, i should've mentioned from the outset that these are really interesting and useful data, and that i'm glad you took the time to generate them. i really wish consumers could systematically report these data in a way that was reliable/trustworthy...! cox proportional hazards models and KM survival curves are the big kahuna with a data set like this. basically, my impression is that you'd want to pretend that you're doing a cohort study and essentially analyze it like a big clinical trial. and re: graphing, since you have sub-samples of the different drive models and relatively different but still small numbers of models for each brand, the big take-away graph comparing manufacturers leaves out a lot of useful nuances that could be pulled out of the table, e.g. that most of the hitachi drives are newer and you have fewer of them. it also doesn't portray how consistent failure rates are across models produced by the same brand might be, and it looks like there's a significant range of failure rates across different seagate models. so even just seeing IQRs of pooled failure rates for each manufacturer in box plots might be eye opening.. immortal time bias may be something to consider here as well...when you have some sample groups that mostly include newer individual drives that you have not yet had for a year on average, subtle differences in how the failure event is described can make a big difference in the conclusions you draw...especially in terms of uncertainty. if i have 10,000 hitachi drives and 5 of them fail in the first 6 months, the robustness of conclusions i can draw from those data are different in some important ways from similar insights drawn from a sample of 1,000 hitachi drives i've used for 5 years. it's also not clear to me how you've dealt with replacement drives. based on what i gather from the post (and i could be totally wrong), if you have a bunch of one model of a drive failing and then get replacements for them... some might argue that refurbed drives should be analyzed almost like a separate drive model, since they often have physical differences compared to those bought via retail channels. i'd be quite curious to dig into these data a bit further if you're willing to post the original data set... thanks again for posting information that's quite useful and interesting

shadeless 12y ago

I recently bought 3TB Western Digital Red, following their advice from [1], but now I see that it has yearly failure rate of 8.8%, bummer. Off-topic, but It's a shame that BackBlaze isn't available in some countries, I'd love to use it. What would be the best alternative to it, Tarsnap? [1] https://www.backblaze.com/blog/what-hard-drive-should-i-buy/ https://www.backblaze.com/blog/what-hard-drive-should-i-buy/

rancor 12y ago

Used in a small file server, my net failure rate on Seagate's consumer 3TB drives has been over 50% thus far. The pair of their SAS drives I currently have in use have been fine, although both of them are still below a year of service life... Edit: Just checked my drive status, and yet another one has dropped. If I'm doing my math correctly, that's 75% of the drives that weren't DOA...

tambourine_man 12y ago

I love reading these posts from Backblaze, but what I never understand is that they are getting a cost of U$ ~0.05/GB with their storage pods: https://www.backblaze.com/blog/why-now-is-the-time-for-backblaze-to-build-a-270-tb-storage-pod/ https://www.backblaze.com/blog/why-now-is-the-time-for-backb... At these rates, why not use S3? What am I missing?

bstar77 12y ago
Perhaps being in control of your own destiny?
frankchn 12y ago
Amazon S3 is US$0.0275/GB per month at high volumes. I assume Backblaze's operational cost of keeping each Backblaze pod hooked up and online is much much less than US$0.05/GB/month.
- tambourine_man 12y ago
  You're right, sorry. I'm so used to seeing U$/month I didn't realize it was the per GB cost for building the pod.
brianwski 12y ago
Disclaimer- I work at Backblaze. Here is a chart of the cost of a Petabyte that shows why we cannot use Amazon S3: https://www.backblaze.com/blog/wp-content/uploads/2009/08/cost-of-a-petabyte-chart.jpg https://www.backblaze.com/blog/wp-content/uploads/2009/08/co... It is from our original blog post about our storage pods (we assemble 45 hard drives in some sheet metal with off the shelf parts): https://www.backblaze.com/blog/petabytes-on-a-budget-how-to-build-cheap-cloud-storage-2/ https://www.backblaze.com/blog/petabytes-on-a-budget-how-to-...

larrys 12y ago

Question for the OP here (or for anyone else). Do you burn in new drives before using? I typically will take any new drive and do some type of stress test [1] on it for 18 to 24 hours to see if it fails with that initial constant use. [1] Constant reformatting for example writing 0's to the entire disk 7 times etc.

rancor 12y ago
Yes, these days I'll make sure that it sees a decent amount of power on time and several full drive writes before ever trusting data to it.
atYevP 12y ago
Yev from Backblaze here -> Yes, we do burn in the drives before deploying them. So the drives in the study are ones that have at least made it past that state.
- larrys 12y ago
  Would be interesting to see a post on what your burn-in procedure is.
- pbhjpbhj 12y ago
  Doesn't that negate a lot of the usefulness of the data? If most Hitachis fail in the burn-in but all Seagates pass it then the expected failure of a drive would be wildly different to that suggested by the charts. Do you have the figures for raw losses including at burn-in? I mean, sure, you burn-in and do warranty returns so buying Hitachi would still seem better - but if one needs a drive to just work then it's key to know the overall failure rate.
  - brianwski 12y ago
    Brian from Backblaze here. Let's call that "DOA Failure rate", where the drive is either dead on arrival, or does not survive the 2 day initial burn in test. I'll ask the boots on the ground for more data, but my impression is the 2 day burn in doesn't catch many drive failures, let's say less than 1/2 of 1 percent. We have even proposed skipping it, but we gain confidence the REST of the system is wired up correctly and other components like power supplies also have infant mortality.

ChuckMcM 12y ago

One of the challenges I have with this analysis is that a 'failure' isn't just that your drive is no longer working, it is that your drive isn't working and you have to go replace it. The operational costs of replacing a drive have three parts, loss of production while the drive is offline, operator time to physically replace the drive and prep it for re-entry into the system, and transactional costs of doing a warranty replacement (filling out the RMA form, getting a valid RMA, shipping the and receiving replacements). We minimize the latter by doing RMAs in batches of 20 but its still a cost across those 20 drives. (and the population of 40 drives which exist as spares are effectively not available for production). It isn't as simple as 'sure drives fail a bit more often but we don't expect to use them that long.'

acdha 12y ago
That's a good point but also requires a more complicated analysis since it requires you to account for your ops capacity (e.g. the marginal cost increase of doing 10 drives instead of 5 is probably a lot less than 2 unless your ops team is almost completely booked) and also requires some guessing about whether there actually is a better option. They addressed that second point with the discussion of enterprise drives, which certainly matched my own experience.

disordr 12y ago

I really want to applaud backblaze for publishing these reports and stats. Too many companies closely guard this information that really helps the larger community. Based on the previous blogs from backblaze, when I built out our new hadoop cluster, I purchased 1450 Hitachi drives. I plan to gather our failure rates and publish them as backblaze does. Thanks for blazing the path!

atYevP 12y ago
Yev from Backblaze -> Thanks! That really is one of our goals with these updates, is for others to join us and start sharing this data. It makes a lot more sense if everyone is doing it, then we can start comparing environments, and all sorts of fun stuff!
- Serow225 12y ago
  For the Seagate 7200.14, have you checked to see if you have the latest firmware on all of them? http://knowledge.seagate.com/articles/en_US/FAQ/223651en http://knowledge.seagate.com/articles/en_US/FAQ/223651en Same for the 7200.11: http://knowledge.seagate.com/articles/en_US/FAQ/207951en http://knowledge.seagate.com/articles/en_US/FAQ/207951en
  - atYevP 12y ago
    Yup, we have top-men that are in charge of keeping an eye on firmware, if we see a need in an update, we'll do it.
    - Serow225 12y ago
      Have you been able to reach out to anyone at Seagate in their drive FA group to see if they'd be interested in you sending back samples of the failed drives?
      - mey 12y ago
        I assume (mentioned in the article) they get sent back for warranty RMA.

sauere 12y ago

Hard drive age a bad parameter to use. It should be the hours the drive was actually powered on.

jamroom 12y ago
I have a feeling that with Backblaze that ALL of their drives are mostly powered on continuously until they fail (i.e. they might sit for a bit at their data center waiting to go into one of their pods, but after that they pretty much run until they fail).
- sauere 12y ago
  If thats the case, okay. Obviously still not ideal - you would want to have even IO load distribution on all drives - but hey, better than nothing.
- brianwski 12y ago
  Backblaze Employee here - this is correct. Pods get powered up, then pummeled with customer data until full. Then they stay spinning forever, but the rate of writes drops to "churn" rates for the rest of the drive's life. This use pattern is all we have experience with, so if your application is different (maybe a database that gets heavy load forever) then your outcomes may vary.

makmanalp 12y ago

Anyone have any reliability information on hitachi's new NAS drive series? They're supposed to build on the 7k3000 etc, but specifically tailored for NAS / raid situations, like WD reds. One major difference is that they're 7200 rpm instead of 5400 which is most non-high-end NAS drives.

linker3000 12y ago
http://www.hgst.com/hard-drives/internal-drive-kits/nas-desktop-drive-kit http://www.hgst.com/hard-drives/internal-drive-kits/nas-desk... Specs and data sheets

arb99 12y ago

Very off topic, but their html is wrong: "<a href='https://www.backblaze.com/blog/hard-drive-reliability-update-september-2014'><img https://www.backblaze.com/blog/hard-drive-reliability-update... src='https://www.backblaze.com/blog/wp-content/uploads/2014/09/blog-fail-drives-manufacture-report2.jpg' https://www.backblaze.com/blog/wp-content/uploads/2014/09/bl... alt='Hard Drive Failure Rates by Model' width='560px' border='0' /></a>" should be "width='560'" not "width='560px'"

atYevP 12y ago
Yev from Backblaze here -> It should work with the 560px, if you're referring to the "share this" html code!

ars 12y ago

No Toshiba hard disks apparently. HGST and Wester Digital are the same company, but it seems they have separate product lines? It's confusing.

ShinyObject 12y ago
The Hitachi drives are Toshiba drives. The 3.5" desktop and server Hitachi drives are all manufactured by Toshiba. WD wanted to buy Hitachi's drive division so there would be a WD/Seagate drive duopoly, European regulators said they had to sell Hitachi's 3.5" to someone else before they would approve the merger. Toshiba stepped up and got it. http://www.anandtech.com/show/5635/western-digital-to-sell-hitachis-35-hard-drive-business-to-toshiba-complete-hitachi-buyout http://www.anandtech.com/show/5635/western-digital-to-sell-h... So yes it is somewhat confusing. You buy one of these drives and it will say "HGST a Western Digital company" on the box, when in reality it's all made by Toshiba, probably in an old Hitachi factory.
- hga 12y ago
  There's a much less sinister reason for WD buying HGST: SCSI/SAS support. Way back when in the '90s WD tried to enter that market, and a company I was at even got some of those drives, but from what I heard while their read rate was OK their write rates weren't competitive with Seagate and IBM (which sold their disk drive business to Hitachi after the Deathstar screwup). And they exited that segment of the business, and hadn't, last time I checked, returned to it.
- blahblah234 12y ago
  Toshiba drives are not exactly Hitachi drives. First, Toshiba had their own 3.5" drives long before the merger with Hitachi. For example, Toshiba MK2002TSKB 2TB 3.5" hard drive was on sale since 2011. Toshiba's own design and factories wouldn't magically disappear after acquiring Hitachi's assets. So, after the merger, Toshiba sells some ex-Hitachi drives, for example toshiba DT01ACA300 3TB is obviously a relabeled Hitachi drive. But I believe that Toshiba MD03ACA300 3TB and Toshiba MD04ACA300 3TB drives are based on Toshiba's own design because they don't look like Hitachi or DT01ACA300. I suppose reliability and performance will be different between these three Toshiba 3TB drives. And it would be interesting to get some info on this. Also, HGST is wholly owned by Western Digital. Despite regulators requirement, they didn't sell all 3.5" assets to Toshiba, only some of them. Most of the good stuff (that is, everything currently sold under HGST brand) still went to WD. So, even ex-Hitachi drives sold under Toshiba brand (by Toshiba) and under HGST brand (by WD) might have different reliability.

robomartin 12y ago

Thanks for sharing such useful data. I just had a Seagate drive fail. Was able to recover data since the last local backup with various tools. It took hours of repair work. I've been procrastinating about getting off-site backup. This post on HN reminded me that I've been meaning to get an account going with your company for a while. I just signed up and will test on my machine before deploying to other machines in my business. Thank you.

TheLoneWolfling 12y ago

I'd be interested to see what a graph of percentage remaining versus time since installation looks like for these. Might give a better picture of what's going on.

BuckRogers 12y ago

I keep seeing this stuff over the years, so I go buy something other than Seagate like WD... and they fail within a year. So I replace it with a Seagate and no problems for years. See another report that says Seagate is terrible- repeat process. I'm just going to keep using Seagate until my anecdata refutes the reality I live in.

atYevP 12y ago
Yev from Backblaze here -> all of the drives in our datacenter will fail. We just report on the timing of those failures. We still buy Seagates in droves because they are a great value. As long as you have backups of the data, any drive is great!

GravityWell 12y ago

This is extraordinarily useful and unique. My compliments to Backblaze for making this available. This is the type of empirical data I would love to have for as many things as possible: SSDs, monitors, TVs, kitchen appliances, tablets, cars, etc.