10 ms·
Backblaze Hard Drive Stats Q3 2020
- busterarm 6y agoBackblaze doesn't provide data on DOA drives that need to be sent back, do they? I went through three rounds of 4x 12TB Seagate Exos X16 drives last month through different safe vendors. 100% of the drives arrived DOA (tested in a bunch of different machines). Switched to 16TB Toshiba MG08s and smooth sailing since. I know the dead ones tend to cluster because of their manufacturing, but I ended up having serious questions about their QC process. Talking to some of my peers, Seagate seems to be a bit of an embarrassment for a long time.
- atYevP 6y agoYev from Backblaze here -> no, any drives that come DOA wouldn't have made it past burn-in and into our fleet, so they aren't represented.
- busterarm 6y agoI really wish you tracked this data, because my colleagues and I in various parts of this industry have been noticing some trends... With failure rates of in-service drives mostly being similar, I start to become much more interested in data about a brand/product's quality control process.
- atYevP 6y agoYev -> that's really interesting. Do you want to publish your data and maybe we can compare notes ;-)
- busterarm 6y agoI just have anecdotes from about 4-5 people buying maybe in the 150TB range of disks per year. Mostly only Seagate or HGST. Not nearly as clean (and representative) a datsset as y'all would have.
- Already__Taken 6y agoIs the takeaway there that you need to be doing burn-in runs too?
- busterarm 6y agoYou can't burn in drives that can't even be used in the first place. They literally did not function straight out of the packaging.
- londons_explore 6y agoSounds like it could have been some kind of incompatibility or firmware/software bug, either on the machines you were using or the drive itself....
- fomine3 6y agoPossibly 3.3V issue? (I don't know Exos affects this) https://www.instructables.com/How-to-Fix-the-33V-Pin-Issue-in-White-Label-Disks-/ https://www.instructables.com/How-to-Fix-the-33V-Pin-Issue-i...
- busterarm 6y agoThat only affects WD drives, these are Seagates.
- busterarm 6y agoNegative.
- Exmoor 6y agoInteresting that they appear to not be deploying WD drives?
- busterarm 6y agoHGST is WD and their Enterprise tier now.
- Exmoor 6y agoIIRC, Backblaze used to install consumer grade drives because they found that whatever they saved in increased reliability and warranty more was more than offset by the price savings with consumer drives. I didn't go through the SKU's, but I'm assuming some part of the equation changed?
- 3np 6y agoHGST is effectively a brand name of WD at this point https://en.wikipedia.org/wiki/HGST https://en.wikipedia.org/wiki/HGST
- photon-torpedo 6y agoTable 1 (Lifetime Annualized Failure Rates) and Table 4 (Annualized Failure Rates for Q3) should be swapped, I believe.
- PatrickTBB 6y agoThanks for the heads up, fixed!
- Havoc 6y agoI don’t get why they keep buying seagates when every report they do shows they have higher fail rates? Are they just cheaper?
- theandrewbailey 6y agoI think they buy whatever their suppliers can get a hold of. They've said that they would buy HGST only, but they are too expensive.
- busterarm 6y agoYes, by around half. HGSTs are in short supply states-side right now and in high demand. Toshibas are also in limited supply here (at least the desirable helium-filled ones) and are about 2/3 between the Seagates and HGSTs. Also in their product, drive failure is okay!
- NortySpock 6y agoI am definitely finding Seagates the cheapest option at Microcenter, generally $5-$10 cheaper. Can't say I'm proud of cheaping out on storage, but that's my current price-point and haven't yet made the leap to trying to set up a properly-RAIDed NAS.
- busterarm 6y agoThe price gap of high-capacity Enterprise drives is in the hundreds (as in multiple) of dollars between the two brands.
- nucleardog 6y agoJust some recent numbers because I just replaced some drives in my home server... I grabbed 2x14TB Seagate EXOS drives for $250USD each. A 14TB WD Ultrastar looks to be a bit over $400USD each. For 25% more you can get 200% the capacity with the Seagate drives. So for 25% more I can run RAID1 and have redundancy versus a single WD Ultrastar.
- maz1b 6y agoI really appreciate how Backblaze is always so consistent in these reports. Quality and disciplined reporting like this is appreciated, and I always find them very interesting. Side note, does Backblaze not run any enterprise or datacenter SSDs?
- deleted 6y ago[deleted]
- atYevP 6y agoYev from Backblaze here -> we do have SSDs in the fleet but they are primarily boot and data drives, so they're not really an apples:apples use-case. We're constantly looking for interesting news from our fleet though, so if there's something that pops out, our default approach would be to talk about it :D
- mmm_grayons 6y agoOh cool, thanks for commenting here. This is off-topic but I had a question from a post a few days back about google drive dropping unlimited storage. Someone mentioned that the economics just don't work long-term due to people who abuse it. How does this apply to Backblaze's unlimited backup service?
- atYevP 6y agoYev -> Great question. The problem does lay with the abuse of the unlimited services. We've been offering our unlimited backup for 10+ years now and our only price increase was from $5 -> $6/month after 10 years of keeping them steady. It's possible to provide the service, but we have some natural "gates" that were built into the product to help mitigate abuse. Primarily our unlimited backup service is for Mac and PC computers – we don't back up NAS/Servers/Linus machines. That helps quite a bit because it reduces the outer-bound of how much data can store with us from typical Mac/PC devices. Granted we do have people with 80+TB backed up with that service, but that VAST majority store A LOT less. The average is low enough that we can continue to provide the unlimited service - kind of like a buffet model. Some will eat more, most will eat less and buy a drink (Extended Version History). Keeping it to Mac/PC only does make Linux, NAS, Server folks unhappy, but thus far no one has been able to crack the code on an unlimited, fixed-rate service that can back up those devices economically, but if we figure it out we'll let folks know :D
- metalliqaz 6y agoAh yes it's that time of year for me to check up on how my lame photos are stored. Looking good chaps
- atYevP 6y agoFeelin' good!
- bufferoverflow 6y agoHGST drives are doing significantly worse than before. WD ruined the solid brand that fast? Q2: https://www.backblaze.com/blog/wp-content/uploads/2020/08/Q2-2020-AFR.png https://www.backblaze.com/blog/wp-content/uploads/2020/08/Q2... Q1: https://www.backblaze.com/blog/wp-content/uploads/2020/05/image1.jpg https://www.backblaze.com/blog/wp-content/uploads/2020/05/im...
- deleted 6y ago[deleted]
- HeadsUpHigh 6y agoAre you surprised?
- pwinnski 6y agoI treasure these reports, and yet every time I go to buy more hard disks, it is very difficult resist the low, low prices of those Seagates. I know they'll fail faster than the WD/HGSTs, but they're so much cheaper! Clearly, given these numbers, Backblaze thinks the same way.
- busterarm 6y agoAfter getting burned on a consecutive $3800 worth of dead-on-arrival Seagate drives, well, never again.
- atYevP 6y agoYev from Backblaze here -> price plays a large factor in our purchasing decisions, but you'll notice if you look at the failure rates that they're not THAT much different across the manufacturers. For home usage, you're likely going to be fine regardless of what drives you buy. Though those sweet sweet 4TB HGST drives are getting harder and harder to find!
- AnIdiotOnTheNet 6y agoTo me, cheaper disks will always win because I expect hard disks to fail and take precautions against data loss anyway. If they fail within the warranty period it doesn't really matter where they came from. If they fail outside of it I probably still got my money's worth.
- java-man 6y agoThe net effect, paradoxically, is that this encourages and supports low quality. Or perhaps it redirects the resources towards those who need it most, and the average quality improves?
- c0m 6y agoA disk failure is still a cost though in terms of restoring a backup, replacing the failed drive, waiting for an RMA/new drive to come, etc.. If the price delta isn't anything crazy I'd prefer to get something that I know is more reliable.
- ksec 6y agoThose HGSTs are still the best in terms of reliability. My question is, what is happening to HGST the brand? Western Digital were supposed to phase out the brand, and only keeping the Gold or UltraStar branding. That was may be 2 years ago. Except HSGT is still everywhere. And I know people demand HGST. I think partly due to Backblaze's numbers. Even the Western Digital UltraStar still has HGST in labels, stickers or description somewhere. Retailers simply call them HGST WD most likely because that is what customers want or are looking for. It seems Western Digital did at one point try to move their marketing material to focus on UltraStar, but my suspicious is that people still want HGST. Again, may be due to Backblaze quarterly report cause they dont show any UltraStar branding, they use HGST.
- busterarm 6y agoThe market responded to the Ultrastar rebrand poorly and they brought back HGST officially.
- ksec 6y agoOh really? Thank You! That is great news to hear. ( Stupid Westen Digital )
- jmnicolas 6y agoWho cares about the label? The internals of the drive makes the quality or lack thereof.
- derefr 6y ago"UltraStar" wasn't a renaming of "HGST"; it was an existing WD brand with many other drives in it, that the HGST drives were going to become aggregated under. This would have erased the distinction between the drives made at the HGST factories, vs. the drives made at other factories, since both would have just been "UltraStar." It would have been a big detective job with every new WD UltraStar model to figure out whether it was "one of the good ones." Whereas, if some WD drives explicitly say HGST and some don't, you can just buy the HGST ones. (Very closely analogous to smartphones that have the same exact model name, but which vary between Snapdragon and Exynos SoCs depending on the precise model number that only appears in fine print in a few places. When buying one of these phones used, you often have to google the precise model number to figure out whether you're getting a fast phone or a slow one!)
- DavidPeiffer 6y agoBackBlaze has been doing this for years, and they're well received in many online communities. They don't need to do this, and the data is arguably valuable to keep proprietary. Are there any other examples of companies doing similar releases of valuable internal data?
- ckdarby 6y agoThe only reason I know about BackBlaze is because of their reports and the storage pod build outs they do. If they never did these I wouldn't even know about their existence. It is all apart of long term marketing and branding.
- DavidPeiffer 6y ago> It is all apart of long term marketing and branding. For sure - I can't think of any other companies with a similar approach to things though. I appreciate what they do, and the good quality write-up they provide as well. If I remember right, years ago YouTube was releasing their prior year server build. I recall things like a cool air stream being directed right at the processor, rather than cooling the whole room enough for traditional cooling methods to work well. My Google fu isn't finding the blog posts I remember reading, probably from ~2010 era.
- theelfismike 6y ago> Are there any other examples of companies doing similar releases of valuable internal data? the early OkCupid blog was really good at this This one has always stuck with me (archive version: https://www.gwern.net/docs/psychology/okcupid/whyyoushouldneverpayforonlinedating.html https://www.gwern.net/docs/psychology/okcupid/whyyoushouldne...) > It turns out you are 12.4 times more likely to get married this year if you don't subscribe to Match.com. more archives: https://www.reddit.com/r/gwern/comments/aapn1l/okcupid_blog_archives/ https://www.reddit.com/r/gwern/comments/aapn1l/okcupid_blog_...
- atYevP 6y agoYev here -> I LOVED the OkCupid blog posts. So much good stuff in there. <3
- scrooched_moose 6y agoWhile we're on the topic of enterprise hard drives, does anyone know who makes Hewlett Packard Enterprise drives? https://www.cdw.com/product/hpe-hard-drive-600-gb-sas/4503486 https://www.cdw.com/product/hpe-hard-drive-600-gb-sas/450348... We have a few of their machines and I've always been curious but haven't been able to find much.
- mey 6y agoYev, since you seem to be in the comments, if it isn't company secret sauce, what is the primary driver to retire drives at Backblaze? Power consumption, storage density/colo space, drive days, mfg specifications?
- atYevP 6y agoYev here -> Good question! The ops team may have a different theory, but I think it's when they die! We still have 15k 4TB HGST drives in our fleet that are crushing the game, and have been for years. That said, we'll eventually migrate off of them to get more density, but so long as we're not cramped for space, we tend to keep high-performing drives in our fleet for as long as we can. There is a math equation that flips after a certain point where we can get a lot more density by replacing those smaller-capacity drives - so that tends to be the other main driver for migrating the fleet. *Edit -> For giggles here's an OLD post where we dissect drive failure and talk about migration a bit -> https://www.backblaze.com/blog/3tb-hard-drive-failure/ https://www.backblaze.com/blog/3tb-hard-drive-failure/
- csnover 6y agoI wish that more companies would provide information on their disk fleets like this—especially on SSDs, which would be much more useful information for everyday consumers. I also wish that Backblaze would focus on their product security[0] as much as they focus on promoting this quarterly hard drive report. [0] https://twitter.com/zetafleet/status/1304664097989054464 https://twitter.com/zetafleet/status/1304664097989054464
- bdd 6y agoPublishing SSD reliability of datacenter setups would yield no useful information for everyday consumers. See my comment at https://news.ycombinator.com/item?id=24840914 https://news.ycombinator.com/item?id=24840914. Neither load nor the operating environment of such reports will be anything remotely similar to everyday consumers' use and environments. In 2015 Facebook published* its study of flash memory failures at scale. By all means it's a terrific paper (of course I'm biased) but I think it was irrelevant not only for consumers but pretty much 99% of the flash users in the industry. [*] http://users.ece.cmu.edu/~omutlu/pub/flash-memory-failures-in-the-field-at-facebook_sigmetrics15.pdf http://users.ece.cmu.edu/~omutlu/pub/flash-memory-failures-i...
- remote_phone 6y agoIs there a reason why they don’t use WDC drives? Makes me nervous because I just picked up two during amazon prime sale
- hpb42 6y agoDo you have a stash of cool magnets and shiny disks? I use dead HDD's disks as drink coasters. It's pretty cool.
- CokeBlaze 6y agoYev always comes across as doing cocaine but if the servers are safu then whatever. Seems like a Russian polish mafia kind of thing/shop/business
- deleted 6y ago[deleted]
- bdd 6y agoI truly appreciate Backblaze publishing these but it's paramount to remember their disk usage pattern is fundamentally different than yours. This data is almost irrelevant to majority of their audience trying to deduce something out of it to help their buying decisions. In addition to usage patterns, there's a lot more that contributes to mean time between failure of hard drives; like operating temperature (ambient, fluctuations, max), relative humidity, vibration and alike. There's also well researched and published literature around this. From Microsoft 2016: Finds relative humidity to be the dominant factor for disk failure https://www.usenix.org/system/files/conference/fast16/fast16-papers-manousakis.pdf https://www.usenix.org/system/files/conference/fast16/fast16... From UToronto 2012: Finds no correlation with DRAM failures, nor high temp and disk failure but concludes temperature variability affects disk reliability more than maximum operating temperature http://www.cs.toronto.edu/~bianca/papers/temperature_cam.pdf http://www.cs.toronto.edu/~bianca/papers/temperature_cam.pdf From Google 2007: cannot find strong correlation between failure rate and temperature or utilization. https://static.googleusercontent.com/media/research.google.com/en//archive/disk_failures.pdf https://static.googleusercontent.com/media/research.google.c... From Microsoft 2013: focuses on operating temperature being the driving factor of failure, more than load. http://www.cs.virginia.edu/~gurumurthi/papers/acmtos13.pdf http://www.cs.virginia.edu/~gurumurthi/papers/acmtos13.pdf
- WarOnPrivacy 6y agoIt seems a top cause of drive failure is affixing the word "Seagate" to the drive. It's comforting that some things are consistent between the enterprise and consumer worlds.
- rkagerer 6y agoI can anecdotally vouch for those 4TB HGST drives. Bought about thirty H....4040ALE640 units several years back and they've had fantastic reliability. No failures, and IIRC only one's been replaced (as a precaution after its SMART Reallocated Sector Count ticked up). And no RAID hiccups whatsoever.
- rexreed 6y agoWhere can a consumer find those drives in individual units in new non-refurbished condition? Amazon seems to be selling refurbished junk.
- chid 6y agoIs there a data issue here? If you look at ST16000NM001G they reference a failed drive in Q2 2020 (https://www.backblaze.com/blog/backblaze-hard-drive-stats-q2-2020/ https://www.backblaze.com/blog/backblaze-hard-drive-stats-q2...) I quote - >Note: The Seagate 16TB drive (model: ST16000NM001G) does show 59 drives and is listed in the report because the one failed drive had not been replaced at the time the data for this report was collected. How can we be back to zero drive failures?
- atYevP 6y agoYev here -> I should add that we have a Q&A webinar going live tomorrow with Andy the author, you can join the Q&A here: https://www.brighttalk.com/webcast/14807/446738?utm_source=Twitter&utm_medium=Social&utm_campaign=webinar_general https://www.brighttalk.com/webcast/14807/446738?utm_source=T....
- chid 6y agoUnfortunately this webinar is at my 4AM. I have a question about the drive data here - I've downloaded the Q3 data and it looks like there are 59 drives listed as at 2020/9/30 for the ST16000NM001G (and the same at 2020/7/1). I compared it with the April 1 data set and it which appears to have 60. Serial number ZL202NFZ appears to be dropped in the later version and serial number ZL202400 seems to not be reading as at April 1 but is available at a subsequent date. Just curious what's going on here! I recently received one of these and the backblaze drive data was very helpful with my evaluation.