14 ms·
One Billion Drive Hours and Counting: Q1 2016 Hard Drive Stats
- taspeotis 10y ago> For Q1 2016 we are reporting on 61,590 operational hard drives ... [t]hese days we need to purchase drives in reasonably large quantities, 5,000 to 10,000 at a time. To me, that's an astoundingly large number of hard drives. But I realise there are probably much bigger deployments. Does anybody happen to know just how many hard drives Amazon or Microsoft have for AWS/Azure?
- atYevP 10y agoYev from Backblaze here -> They have a lot more :D
- chiph 10y agoAnd yet, your quantities aren't enough to attract the attention of the WDC direct sales department?
- atYevP 10y ago> And yet, your quantities aren't enough to attract the attention of the WDC direct sales department? We've just started heading down the direct paths, but yea it seems like the minimum order number to work direct with the manufacturers is about 10,000 hard drives per order. We aren't quite at that capacity and we don't like to keep inventory since we try to run pretty lean. Also the price for hard drives tends to drop by a small percentage monthly, so for every month we have excess inventory we purchased earlier, that could potentially be money left on the table. So our orders to be smaller than their minimums. We're inching towards it though :)
- alecbaldwinlol 10y agoThat seems like a silly minimum for WDC to have, unless the discounting is phenomenal (50%+) compared to other vendors. I'm asking myself, even if I'm a WDC rep that is selling hundreds of thousands of PC hard drives and having an excellent quarter/year, why would I turn away the business of a growing company? Tape cartridges can be purchased in packs of 20 from any corporate vendor and no order is too small to attract a rep's attention. I've seen deals (25th to 75th percentile) from $10,000 to $200,000, and the min/max deal range is from $2,000 to $500,000
- atYevP 10y agoI imagine it has something to do with demand or their partnerships with their distributors. This is conjecture, but I'd think that in order to keep distribution deals/channels up they might have deals in place that stipulate any order over 10,000 goes direct and any order under goes to preferred distributors. No idea though, but that would make a smidge of sense.
- alecbaldwinlol 10y agoI imagine maybe 10x more than this. The numbers actually aren't too shocking if you think of the mirroring of data that's required for maximum uptime (not even archival), and the occasional disk failure. 1,000 TB (1PB) can be easily handled across ~150 (6-7TB)HDDs for one copy, but 300-450 HDDs would be required for additional mirroring. Largest tape cartridges out there are between 6 and 8.5 TBs, and cost around $22 per TB. That's only $22,000 per PB, and this is for high throughput cartridges like LTO7 or StorageTek Titanium. LTO5 is much cheaper. Considering that the largest tech companies and major organizations routinely cut POs for several $100ks and are dealing with 100s of PBs of data across disk, tape, DVDs etc, it isn't outside the realm of possibility to have 300,000+ individual disks and tapes floating out there :)
- justinsaccount 10y ago> 1,000 TB (1PB) can be easily handled across ~150 (6-7TB)HDDs for one copy, but 300-450 HDDs would be required for additional mirroring. Mirroring is not used at that scale https://www.backblaze.com/blog/reed-solomon/ https://www.backblaze.com/blog/reed-solomon/ https://blogs.msdn.microsoft.com/windowsazurestorage/2012/06/13/usenix-best-paper-award-erasure-coding-in-windows-azure-storage/ https://blogs.msdn.microsoft.com/windowsazurestorage/2012/06...
- alecbaldwinlol 10y agoThat's all good, it would just increase the number of disks required to be purchased and the amount of electricity/cooling/floorspace to maintain them. It would add to the disk count, but not really affect cost per TB all that much. How do cloud storage vendors guarantee triple-mirroring and uptime then? Lots of 2TB drives? Lies? :)
- rz2k 10y agoDoes it say triple mirroring or triple redundancy? Take a look at the article from Backblaze for an overview of the math. In their case it might be called quadruple redundancy. 20 shards hold 17 shards of data with triple parity. Events destroying hard drives containing information about your data could happen three times, and you still wouldn't lose anything.
- bluedino 10y ago>> how many hard drives Amazon or Microsoft have for AWS/Azure? I'd like to know YouTube's totals with 500 hours of video being uploaded every minute...
- alecbaldwinlol 10y agoTough to say, but if this link is right: http://video.stackexchange.com/questions/8850/typical-size-of-one-hour-long-mp4-video http://video.stackexchange.com/questions/8850/typical-size-o... Then being conservative, we can say an hour of video is 1-2GB of data, so between 500 and 1,000 GB of data uploaded (just to YT) every minute. Accounting for mirroring and compression, 525,949 minutes per year * 1 TB per minute = ~526 PB per year Ironically this would be about 65,743 8TB HDDs, which is close to the number in this article :)
- rdl 10y agoFrom reading this, HGST >>> Seagate, right? I wasn't sure how the "weird" sourcing they had (cracking external drives open during the HDD drought) would affect things, but that was over in 2015, and the trend still seems to be HGST > Seagate.
- oarsinsync 10y agoYeah, that was my takeaway. In particular, this graph makes for sobering viewing. https://www.backblaze.com/blog/wp-content/uploads/2016/05/drive-stats-2016-q1-failure-by-mfg.jpg https://www.backblaze.com/blog/wp-content/uploads/2016/05/dr... I'm regretting the last batch of Seagates I bought to replace HGSTs now.
- atYevP 10y agoYev from Backblaze -> you're probably fine. Everything will fail eventually, just have a backup! We recommend Backblaze :P
- rdl 10y agoI wish you had a way to "seed" backups with removable media. I'm a backblaze user but initial backup (I have photo collections) is super slow. Probably not in your pricing model, though.
- atYevP 10y agoYea, that's a bit tough for us to do. We'd have to have a way to tie that to your account from inside our datacenter and that's a bit complicated. Plus we'd have to find a way to charge for that to make up for the additional labor. That might be something we consider for B2, but it's a tougher sell for our personal/business product.
- rdl 10y agoFor B2, making an open source version of the amazon data transfer appliance would be cool.
- pazra 10y agoBackblaze is such a great company. Not only do they offer almost unlimited backup for just $5 a month, they also publish extremely useful articles such as this one. Good to see Seagate has improved the quality of its product significantly, but a 3.5% failure rate still seems rather high. How old are the drives in question?
- saryant 10y agoHow's the restore process? I've actually been backing up to Backblaze for about a year now but (knock on wood) haven't had to restore any data. That said, one of my drives has been acting up in the last few days so that moment may be at hand (though I also have it backed up to a second local drive).
- rbritton 10y agoI had a drive fail last fall and used the restore portion for the first time. I actually opted to use it over my local time machine backup and am glad I did. It was just a matter of selecting the files online, waiting for it to create the download, and then doing so. Much more painless than my other options.
- roasm 10y agoI back up to a local external drive, and also to BackBlaze. External drives are so cheap these days that the ease of handling your restore yourself is worth it to me. BackBlaze is just an offsite back up so when the house is burning down, I can grab the kids instead of pictures of the kids.
- theandrewbailey 10y agoMy alternative: buy 2+ external drives and a fire/water proof box. Keep one drive unplugged and in the box when not backing up. Keep other n-1 drives elsewhere and swap with the local occasionally. I have one at work, and another at my parent's house, more than 100 miles away. In case of a literal blaze and/or high water, even the local drive should be fine. If all drives are gone from a single catastrophe, I figure I (and everyone else) have more serious things to worry about, like what's for dinner.
- osolo 10y agoI really appreciate these blog posts and have used the data they present when buying HDDs for personal use. One thing that bothers me is that the data presented doesn't really take into account the age of the HDDs. For example, if a batch of HDDs of a particular model is 6 years old and has a failure rate of 12%, that really doesn't tell me much except that it's an old HDD. What I'd like to know, for a given model, what the blended failure rate is after 3mo, 6mo, 12mo, 24mo etc of operational time. That would be a real apples-to-apples comparison.
- atYevP 10y ago> What I'd like to know, for a given model, what the blended failure rate is after 3mo, 6mo, 12mo, 24mo etc of operational time. That would be a real apples-to-apples comparison. You might actually be able to pull that from the raw data. We post it all here -> https://www.backblaze.com/b2/hard-drive-test-data.html https://www.backblaze.com/b2/hard-drive-test-data.html in case you wanted to play around with it!
- polarix 10y agohttp://bioinformare.blogspot.com.au/2016/02/survival-analysis-of-hard-disk-drive.html http://bioinformare.blogspot.com.au/2016/02/survival-analysi... has graphs of survival rate over time broken down by manufacturer and model. Far more informative, you're right. I don't think Backblaze uses 6 year old drives. :-)
- atYevP 10y agoThat graph is pretty dope! When it first came out we had fun diving in to it! Glad someone is using our data-set for fun stuff!
- osolo 10y agoThanks for sharing this. It's exactly what I was looking for. tldr; It clearly shows that HGST has an overall superior survival rate over time. WD is a distant second and Seagate in third (although the Seagate ST4000DM000 model is exceptional and fairs very well).
- jokoon 10y agoMy 5 year old hard drive has been vibrating like crazy this year, also making the metal case panes vibrate, which makes a ton of noise. So I searched online, and found a solution where the HDD is suspended with cable ties, a little like this http://i.imgur.com/4MXB1IG.jpg http://i.imgur.com/4MXB1IG.jpg I guess I should just buy another one. I'm going to lose all my data anyday now.
- atYevP 10y ago> So I searched online, and found a solution where the HDD is suspended with cable ties, a little like this http://i.imgur.com/4MXB1IG.jpg http://i.imgur.com/4MXB1IG.jpg Yev from Backblaze here -> good hustle! That sounds a lot like our "anti-vibration sleeves" from our earlier pod versions: https://www.backblaze.com/blog/backblaze-storage-pod-vendors-tips-and-tricks/ https://www.backblaze.com/blog/backblaze-storage-pod-vendors... (https://www.backblaze.com/blog/backblaze-storage-pod-vendors-tips-and-tricks/ https://www.backblaze.com/blog/backblaze-storage-pod-vendors...).
- jerf 10y agoYou're going to lose your drive any day now. Not necessarily because your drive is vibrating... just because you're always going to lose your drive any day now. Anything on only one drive can be on only zero drives in the blink of an eye. If that's a problem for you, best deal with it before it's a problem. It is true that hard drives often give a surprising amount of warning. I don't think I've ever had a drive totally spontaneously fail on me. But it's still best not to count on that.
- xhrpost 10y agoWow, did I read that right? They are filling 3+ vaults per month? A vault is 20 pods. Each pod is 480TB. They've mentioned in the past that they use 17+2 redundancy. So each pod should be about 429TB of added storage. At 60+ pods per month, they're adding 25,740TB per month?!
- atYevP 10y agoYev from Backblaze -> That's about right ;)
- deleted 10y ago[deleted]
- forgotpwtomain 10y agoThere seems to be an incomparably tiny sample size for about half of these... 1 out of 47 drives gives a 8.63% annual failure rate.
- pgrote 10y agoYes, they mention this in the article. "Failure rates with a small number of failures can be misleading. For example, the 8.65% failure rate of the Toshiba 3TB drives is based on one failure. That’s not enough data to make a decision."
- tibbon 10y agoI wish some other companies that use massive amounts of hardware (Dropbox, Amazon, Google, Facebook, NSA) would release data like this. I love that Backblaze does it, but having an even larger sampling of data would be great.
- bluedino 10y agoGoogle has released some similar information in the past http://static.googleusercontent.com/media/research.google.com/en//archive/disk_failures.pdf http://static.googleusercontent.com/media/research.google.co...
- brianwski 10y agoBrian from Backblaze here. This Google study is EXCELLENT and extremely useful, but it leaves off the drive model numbers and manufacturers which means you cannot use it to help guide purchasing decisions. It's still an awesome study and well worth reading and useful to a ton of people (like us at Backblaze when we were starting out).
- brianwski 10y agoBrian from Backblaze here. It completely, utterly blows my mind that we are the first and still only company to do this. First of all, it is a great way for people to hear about our company, which sometimes leads to people buying our products and services. Second, even just for the good of all humanity, why wouldn't you do this to promote the best drive manufacturers and heck, even help them learn how their drives are working in the field? It boggles my mind that Facebook, Yahoo, Google, and Amazon (Amazon Web Services include S3) don't publish their drive failure stats. What could possibly be the reason they keep silent?
- hinkley 10y agoYou guys are in the Do Well by Doing Good category on this, and unfortunately few companies in any business sector maintain this philosophy as they grow. The boardroom is so divorced from the showroom that it's not even funny how big he disconnects can get.
- anexprogrammer 10y agoInteresting that HGST do so well, yet are now part of WD, who do so badly. I would have expected more absorbtion+rationalisation from the takeover, yet they still clearly look of the IBM family than anything in WD's ranges. IBM always had a pretty good rep for failure rates, aside from a couple of horror drives in the 90s. I wonder how they've managed to keep their rates markedly better than other makes, even under changes of ownership. More to the point I wonder why WD haven't been able to improve rates as a result of taking over IBM/HGST.
- atYevP 10y agoYev here -> From what we can tell and have been told they are run as completely different entities. Hopefully that quality spillover will indeed happen in the future, but for what it's worth WD isn't doing all too poorly, they just tend to be slightly more expensive for us than the Seagates at the moment.
- joefkelley 10y agoThe Chinese regulatory system had been preventing any actual integration of the two companies for some time. They finally agreed to allow it to proceed in late 2015, but sales and marketing have to remain separate for another 2 years.
- snowwindwaves 10y agoAfter seeing that HGST are the best drives from back blaze reports and then seeing that WD owns the HGST and IP I bought WD shares. A reasonable dividend doesn't hurt either
- rsync 10y agoSince we're on the subject of hard drives, I hope this will be helpful or interesting: It has been our experience that it is no longer possible to buy new, non-fraudulant drives of a sort of recent, but not brand new drive model from Amazon. So, for instance, in early 2016 if you want to buy 4TB enterprise drives from Amazon, you will find them, they will be classified as brand new, and they will be sold by some big amazon parts seller. When you receive them they will be nice and shiny brand new wrapped - perfectly sealed - and when you spin them up, SMART stats will show 4000-6000 hours of use and that they are 2-3 years old. This is almost universal and has been happening for at least 3-4 years. These sellers are selling the drives as brand new and they are anything but. When you complain, they will immediately exchange or refund - there's never a hassle there - and once in a while the seller will spout some bullshit about the drives being "new pulls" ... that is, drives they stripped from unsold servers/desktops. Hope that helps.
- beachstartup 10y agohave you seen similar issues with brand new drives bought from legit VARs? i.e. the big new HSGT drives. we've had good luck but something about your username tells me you buy more than we do... the worst luck we've had is the large 2.5" drives. those are basically just disposable.
- rsync 10y agoNo, the only issues we have seen is buying "new" drives on amazon that aren't the very latest, greatest 6-8TB drives. If they are very new, they haven't had time to percolate through the supply chain and come out the other end as "drives we can pretend are new". So again, as of May 2016, it is 4TB drives that we are seeing at great prices, sold as brand new (no confusion about this at all) on Amazon, and ... turns out they are not new. I keep meaning to write a blog post about this with specific vendor names ...
- digi_owl 10y agoAmazon has become quite the cesspool of late. Everyone and their dog can register as a reseller there. And as long as the stink do not reach Amazon spreadsheets, nothing happens about it.
- Ologn 10y agoHaving had many years of experience as a sysadmin, I have one word for those concerned with hard drive reliability - temperature. Heat is the enemy of hard drives (and other system components). Sometimes heat causes a machine room meltdown, where machines start failing left and right. This is not uncommon. What usually happens is this - the company knows enough to put their web server etc. at a controlled colocation facility (which isn't always ideally controlled, but that's another story). But the developers would like a local file server, development source control server etc. There is a spare, windowless room in the office, and without much planning, a machine or two goes in. Then a machine goes in attached to a tape drive, which backs up those machines and the local desktops. Then more machines go in, then more. All these machines generate heat, so a room cooler is bought. But it drops condensation in a cup, and shuts off when the cup is full of water. So someone's job becomes to empty that cup every morning. But then summer comes, and on a particularly hot Sunday afternoon the condensation cup fills, the cooler shuts down, the outside temperature combines with the temperature of a closed, windowless room full of machines for hour after hour. Finally one machine has a component fail. Then another. Then e-mails and phone calls and panic starts flying. I have seen this happen more than once, and have heard about it more. It always starts as an ad-hoc, unplanned, "temporary" solution for "unimportant" machines. But as time goes on, and machines are added, and business dependencies are formed, you have to start supporting a machine room that was never planned as a machine room.
- redcalx 10y agoWondering if data centres typically power down a drive after some period of no accesses? Seems like you could save a lot of power and gain some HD life that way - so long as a drive isn't frequently spinning up and down. Seems like the data access patterns could be monitored and less frequently data moved to a 'low access frequency' HD bank(?)
- atYevP 10y agoYev from Backblaze here -> For us the drives are constantly spinning, we don't power them down. One of the reasons is we never know when a customer will want a restore, so we have the data available 24/7. That said with our Vaults (https://www.backblaze.com/blog/vault-cloud-storage-architecture/ https://www.backblaze.com/blog/vault-cloud-storage-architect...) it's theoretically possible to power down entire cabinets and still have the data available, but we don't currently see a need to do that.
- robotresearcher 10y agoMaybe energy is too cheap. I use BB and I'm kind of sad to hear that juice is wasted on my stuff I haven't restored for years.
- redcalx 10y agoI would suggest that you consider 'voting with your feet' so to speak. You are, after all, ultimately paying for that wasted energy with you backblaze fee.
- redcalx 10y agoSee the other comment: there is a lot of data that isn't accessed for years at a time, so you're essentially spinning disks for years to save a few seconds of spin up time. I think he's right - energy is too cheap for this to be a commercially relevant factor to you, but if the true cost of energy was factored in (climate change, the human cost s of the relatively dangerous mining and oil drilling industries, local pollution, etc) then I think that would tip the balance. Oh well. OK, final suggestion, maybe do it and spin it as a public relations win? :)
- ck2 10y agoThat lack of failure on the Seagate 6TB is impossibly impressive for Seagate. Someone else must be making the drive for them or they have a new factory doing something different.
- Veratyr 10y agoSo my math is probably messed up here but: - Price of buying a drive and a pod to house it: ~$0.036/GB [0] - 1.84% of drives need replacing each year (warranty aside) - They use 17:3 parity for redundancy (15% of storage) So the hardware price of a GB should be something like $0.036 * (1/0.85) * 1.0184^(years). For 10 years, the hardware would cost $0.05/GB, or $0.0004/GB/month. Power costs and DC space of course need to be taken into account but I still find it interesting that the hardware itself costs only ~10% of what they charge for B2. [0]: https://www.backblaze.com/blog/open-source-data-storage-server/ https://www.backblaze.com/blog/open-source-data-storage-serv...
- askldfhjkasfhd 10y agoHow did you choose 10 years?
- Veratyr 10y agoArbitrarily. I suspect after that point another storage technology will supersede hard drives.
- arca_vorago 10y agoI love the work Backblaze does, and especially how much data they publish. Not only that, but their openness of their storage pod design helped me more fully understand the current limitations of backplanes as we move to more and more data. I have dreams of m.2 backplanes so I can skip the limitations of sata... but I digress. Thanks Yev and everyone else there, a shining example of how to build a company and a reputation at the same time. Keep it up.
- 35bge57dtjku 10y ago> For WDC, we sometimes get offered a good price for the quantities we need, but before the deal gets done something goes sideways and the deal doesn’t happen. Sound pretty interesting. What's really going on?
- ausjke 10y agoJava is responsible for 91%* of security attacks. Backblaze's code is native to Mac and PC and doesn't use Java. -- is this real? Anyway I'm going to be a backblaze customer for $5 a month!
- krzyk 10y agoIsn't it for the applets?
- urza 10y agoI wonder, what filesystem are they using. Or more general the whole software stack (what OS, etc) would be interesting to know.
- urza 10y agoAh I found it on their blog: https://www.backblaze.com/blog/vault-cloud-storage-architecture/ https://www.backblaze.com/blog/vault-cloud-storage-architect... They use ext4: Each of the drives in a Vault has a standard Linux file system, ext4, on it. This is where the shards are stored. There are fancier file systems out there, but we don’t need them for Vaults. All that is needed is a way to write files to disk, and read them back. Ext4 is good at handling power failure on a single drive cleanly, without losing any files. It’s also good at storing lots of files on a single drive, and providing efficient access to them. Compared to a conventional RAID, we have swapped the layers here by putting the file systems under the replication. Usually, RAID puts the file system on top of the replication, which means that a file system corruption can lose data. With the file system below the replication, a Vault can recover from a file system corruption, because it can lose at most one shard of each file.
- heywire 10y agoI'd really like to try Backblaze, but I use a Windows Home Server 2011 Atom-based system as a makeshift NAS to store all of our family photos. The Backblaze Personal Backup installer fails with an error because it is considered a "Server Operating System". I'm currently using CrashPlan with this system.