30 ms·
Backblaze Hard Drive Stats for 2020
- Merman_Mike 6y agoI'm planning a few encrypted long-term backups (ie. stick it in a temperature controlled safe for a few years). What's the best medium for this? SSD or HDD?
- mrkurt 6y agoProbably writable blu ray.
- xellisx 6y agoCheck out M Disc: https://mdisc.com/ https://mdisc.com/
- Hamuko 6y agoAFAIK, M Disc really only matters for DVDs due to their organic materials. (Non-LTH) BDs on the other hand have inorganic materials and last pretty well. I think there was a French study that compared DVDs, M Discs and BDs and the HTL BDs fared very well. Can't find the document though.
- smarx007 6y agoI don't think MDisks were compared if that's the study. https://club.myce.com/t/french-study-on-bd-r-for-archival/302333 https://club.myce.com/t/french-study-on-bd-r-for-archival/30... https://francearchives.fr/file/5f281a39048987dcef88202816a5c8ade880552e/static_6187.pdf https://francearchives.fr/file/5f281a39048987dcef88202816a5c...
- Hamuko 6y agoI think it was a separate one.
- deleted 6y ago[deleted]
- kamranjon 6y agothis is the weirdest website
- ineedasername 6y agoSSD's are not as bad as they used to be, but still not rated for long term unpowered storage. HDD would be better for that. But HDD isn't your only other option. How important is the data, How often will you need to access it, and will you need to rewrite to the storage medium? You might want to consider Blu Ray. Or both, stored in different locations. Also look into LTO tape drives. LTO 6 drives should be cheaper than 7/8 (though still not cheap) and have a capacity around 6TB.
- gruez 6y ago>Also look into LTO tape drives. LTO 6 drives should be cheaper than 7/8 (though still not cheap) and have a capacity around 6TB. AFAIK a post on /r/datahoarders says that the breakeven point for tapes vs shucked hard drives from a pure storage perspective is around 50TB. Given the hassle associated with dealing with tapes, it's probably only really worth it if you have 100+TB of data to store.
- klodolph 6y agoI can vouch for the 50TB figure, it’s around there. The amount of hassle depends on your workflow. If you create a backup every day and then bring the media off-site, tape is easier. Easy enough to put a tape in your drive, make the backup, and eject. Tape is not sensitive to shock and you can just chuck the tapes in your care or shove them in your backpack.
- 6y ago
- WrtCdEvrydy 6y agoI'd argue SSD here... those memory chips should be good for a few years.
- Aardwolf 6y agoHow long is a few years? What would be a good recommendation for decades? Time goes fast!
- ineedasername 6y agoA few years isn't archival quality. An HDD will last longer and is cheaper, and speed is much less of an issue for a drive that will be written to and then chucked in a safe.
- WrtCdEvrydy 6y agoThe HDD lubricants are usually bad after 10 years...
- rasz 6y agoI suspect helium will escape first. Then there is rubber perishing (bump stops) after ~30 years turning into liquid and sticking parts together.
- WrtCdEvrydy 6y agoNot all drives are helium-filled, and theoretically helium just reduces power usage. Your lubricant drying out will be the biggest threat in my eye as your startup after 10 years may cause a lot of damage on the mechanical bits.
- ineedasername 6y agoYes, HDD's should probably be considered medium-term storage. Tapes seem a little more robust, but it seems like M Disc, estimated around 1,000 years, take the crown. Unfortunately 100GB is the limit, so very large files will be difficult.
- Proven 6y agoAmazon Glacier
- comboy 6y agoWhy not b2 or glacier since you're encrypting anyway? If you don't have that much data then maybe M-DISC? Personally I think safe is.. unnecessary. What is it protecting you from when your data is encrypted? If you put it in a safe then you probably care enough about the data not to have it in a single location no matter how secure it seemingly is.
- deleted 6y ago[deleted]
- sigstoat 6y ago> What is it protecting you from when your data is encrypted? various forms of physical damage, including fire and accidental crushing where do you think they ought to store their drives? a little safe that will hold easily 100TB costs $50 and can hold your passport and such too.
- emidln 6y agoIgnoring for a moment how insecure most cheap locks are (including locks on safes), little safes are rarely effective vs a prybar + carrying them away to be cut into at the attacker's leisure. Larger safes have some of the same issues w.r.t. cutting, but you can make it less convenient for an adversary to do it (and make them spend more time where they might be caught).
- robotmay 6y agoWhat about tape? I suppose the cost of the drive is prohibitive but I was under the assumption that was used for a lot of long-term storage.
- derekp7 6y agoI would imagine previous generation tape drives (used) can be economical. Just need to find a reliable place that handles testing / refurbishing (cleaning, alignment, belts, etc) used drives. Also the other bit item is needing the appropriate controller and cabling.
- klodolph 6y agoYou (more or less) just need a fiber channel card, they’re pretty mundane otherwise.
- kiririn 6y agoTape drives are open about both their condition and the condition of tapes. It’s all there in the scsi log pages, more detailed than SMART on hard drives. Mechanically and electrically, everything is rated to last several times longer than the head In other words, you just need to buy two used drives (one as spare) and verify they can write a full tape and their head hours and other error counters are sane. There is no reasonable need to refurbish a tape drive other than a head replacement, which is easy to do at home but so expensive (for older generations) that you might as well buy a new drive. All the testing you could hope for is done in POST and by LTT/equivalent (writing a tape and reading logs is good enough)
- DanBC 6y agoGood quality DVD, in tyvek sleeves, with copious amounts of PAR2 data, in multiple places.
- theandrewbailey 6y agoHow long is 'a few years'? Controlled environments shouldn't be necessary for unplugged drives, just keep them at or slightly below room temperature. I've had three external hard drives for 7 years, and none have stopped working. I have one, and keep two somewhere else (office, family). I connect one for a few hours every week/month to update, then leave alone until needed, or rotated with one elsewhere.
- Merman_Mike 6y agoI'd want to verify the existing data and maybe add some data once a year or less.
- toomuchtodo 6y agoLTO tape (specifically that which is rated for 15-30 years of archival storage) with the drive. The tape is usually rated for a couple hundred full passes, which should more than meet your needs if you're writing once and sticking them somewhere safe. SSDs don't have this archival longevity yet, and hard drives are better when powered up and the data is always hot for scrubbing and migrating when indicators of drive failure present.
- einpoklum 6y agoDon't LTO tape drives cost about a zillion dollars each?
- toomuchtodo 6y agoI recommend acquiring them second hand (but validated) for personal use.
- sliken 6y agoI believe it's 15-30 years for archival storage, if you regularly unspool and respool them to avoid print through.
- Hamuko 6y agoI've had the impression that just having a HDD sit around doesn't do it good and it might just fail when you replug it the next time.
- louwrentius 6y agoTape. But that’s probably unrealistic. Otherwise HDD. Archiving is not a one-time thing, but a process. If you care about the data, you should periodically check the media and at some point replace the media as they age.
- Spooky23 6y agoFigure out what you really need and print it on good paper. Put that in a safe place, away from direct light and dampness. Save the rest on two of Google Drive, OneDrive, iCloud, some other cloud storage, a backup service or copy to a computer in your home. Make your selection based on things that you will "touch" in some way at least every 12-24 months. Everything else will fail in a few years. Don’t save crap you don’t need. Don't futz around with optical media, tape or other nonsense. Don't buy safes or safe deposit boxes unless that's going to be part of your routine in some way.
- ghaff 6y ago>Don’t save crap you don’t need. I tend to agree with this although it can be hard to determine what you won't want/need in advance and it probably takes at least some effort to winnow things down. That said, I'm in the middle of going through my photos right now and deleting a bunch of stuff. (Which is a big job.) It's not so much for the storage space as I'll "only" be deleting a few hundred GB. But it's a lot easier to look for stuff and manage it when you don't have reams of near-identical or just lousy pics. One of my takeaways from this exercise is that I should really be better at pruning when I ingest a new batch.
- Spooky23 6y agoIt is! I think that effort is worth it. As it stands, we've all become digital hoarders as the up-front cost to accumulate stuff like photos and documents goes to zero. The problem is you're dumping LOTS of cost into the future. Photos are a big thing for me. Initially, I used applications (Picasa and later iPhoto) to tag photos with metadata to indicate importance, etc. Applications tend to have zero respect for to commitment to preserving metadata. So by the time my kids are going to college, my family is going to have 200,000+ photos alone. What's the point? Am I getting pleasure (or to borrow from the Netflix organizational guru "Does it bring joy?") from this data? Personally, my new strategy, having been burned by the tools is a pyramid: 1. Print and Frame/preserve important or significant pictures (Say 20-30/year) 2. Curate others that we care about. (Say 500/year or 5-8%) 3. Purge stuff of no value. (Say 2500/year or ~25%) Based on my "performance" today, if I keep at it, I'll be able to reduce the rate of growth.
- cm2187 6y agoI'd suggest to save the encryption software along the drive (unencrypted)! Sounds like a good fit for SMR archive HDD.
- theandrewbailey 6y agoSeparately storing the encryption software isn't needed if you use LUKS.
- parliament32 6y agoIt'd probably be cheaper to stick it in Glacier or GC Archive ($0.0012/GB/month).
- stevezsa8 6y agoThis is what I'd recommend. 1. Backup to external SSD or NAS. This is the backup you will rely on if your PC loses all data. It will be fast to replicate to. 2. Mirror the external backup to a second external SSD. And sync it every week or month. Sync more often if your data is changing a lot. 3. The third layer is an external HDD mirror for the long term off-site backups. HDD are cheaper and more suited for being switched off long term. 4. If you can afford the expense of a forth step, every year buy another external HDD and put the previous one aside as an archive to be brought into service if the current one fails to boot. I recommend separating your data into some short of hierarchy and choose what needs to be backed up to what level. So if you have some software ISOs that you could repurchase/redownload, then have a separate drive for junk like that and don't have it go all the way through the backup steps listed above.
- fulafel 6y agoIt's highly dependent on your budget and access patterns. But not SSD certainly.
- numpad0 6y agoThe stats looks stable and consistent with common knowledge -- HGST > WD >> Seagate, Toshiba inconclusive. Do anyone have anecdata on Toshiba MD/MG/MN 7K2 drives(excluding DT because those are HGST OEMs)? They are always less price competitive and thus always light on real-world stories though they seem comparably reliable as HGST.
- dbalan 6y agoHGST is sadly just WD now.
- hinkley 6y agoAnd Audi is just Volkswagen. Except it isn’t (Audi tech shows up on VW when it reaches economies of scale, eg DSG gearboxes ~12 years ago) Is HGST still a separate department or “just” their luxury brand now?
- freeone3000 6y agoIt's still a separate factory making separate drives. This line even uses a different storage controller. But this is also true for luxury ranges, in general, so you may be asking for too fine of a distinction. (Their usual luxury range is the WD Red, however.)
- robhu 6y agoA luxury range where they occasionally sneak shingled drives in without telling you https://blocksandfiles.com/2020/04/14/wd-red-nas-drives-shingled-magnetic-recording/ https://blocksandfiles.com/2020/04/14/wd-red-nas-drives-shin...
- hinkley 6y agoWhich is why I asked. That sounds like the sort of thing that happens when it's just a label instead of a division.
- 6y ago
- louwrentius 6y agoI wonder why Western Digital is almost absent, does anyone know why?
- Hamuko 6y agoAren't all of the HGSTs essentially WDs?
- louwrentius 6y agoYes, but does that explain why there are almost no WD branded drives?
- numpad0 6y agoIIRC, WD > 6TB are all 7K2 and HGST. HGST itself is "a WD company" but for antitrust reasons the corporates are separate.
- syshum 6y agoHGST has a brand was phased out years go, and the company was not separate for Anti-Trust reason WD had to sell and license some HGST assets for Anti-Trust reasons, but there was never a requirement for them to be a seperate company HGST Websites and other things today just redirect to WD, there is no new HGST branded things being made as far as I am aware
- numpad0 6y agoI still see some drivers sold online under HGST branding. Probably market dependent?
- syshum 6y agoUsed or Referbs. Maybe some old stock but at this point I doubt it... You will see "UltraStar" branding but it will be "Western Digital UltraStar" not HGST UltraStar. Some retailers may even still put HGST in the listing (i.e Amazon still has several listing for new Drives that refer to HGST) but the Packaging, offical documents, and everything that comes out of WD will not have any HGST branding on it, it will not say HGST on the drive etc
- i5heu 6y agoIt is always amazing to me how cheap, small and reliable storage has become.
- ramraj07 6y agoAu contraire I feel like drive reliability has gone _down_... Especially for consumers - the big difference between Blackblaze and regular users is that they have their disks spinning continuously, and the reliability numbers seem to only apply in that scenario. If you switch off and store a drive my experience is that after a year it's very high probability it won't switch on again. This is a big problem in academic labs where grad students generate terabytes of data and professors and departments are too stingy to provide managed storage services in that scale so it all sits in degrading drives in some drawer in the lab.
- mmsimanga 6y agoThis. I bought a hard drive docking station and the idea was to go through all my 6 hard drives from the past 10 years which I haven't used. Only the laptop drives worked.
- guenthert 6y agoIs the docking station supplied with enough current? 3.5" drives tend to take considerably more power (and use 12V for their motor) particularly when spinning up. I'd give those drives another try in a different docking station or connect them directly to a PC, RAID or NAS device.
- lostlogin 6y agoMy experience of data storage among academics has been disturbing. Masses and masses of work is stored on USB sticks and laptops. Hundreds of hours of work, maybe even thousands of hours and no backups. I’ve hit it multiple times and it blows my mind each time. Yes, buying a basic backup solution is going to set you back a few hundred dollars minimum (or not, if you go for BB or similar) but it seems like a basic minimum. I don’t know how you change the culture but it’s bad among those I have worked along side. I haven’t bought large drives in years and recently started doing so. I have been really impressed with how good they are and how well they perform in an always-on NAS. I’m so impressed with the Synology I got and can’t speak highly enough of it. I just wish I’d bought one with more bays.
- busterarm 6y agoI know that they're not always the most available, but I find it interesting that they don't purchase _any_ Fujitsu drives. After an experiment with their helium-filled drives, I've gone all-in to great success.
- dsr_ 6y agoFujitsu sold their disk drive business to Toshiba in 2009, didn't they?
- numpad0 6y agoThey still make drives!? I've seen relics like 147GB 15K SAS marked "FUJITSU LIMITED" but don't think they make 14TB 7K2 SATA that Backblaze uses.
- busterarm 6y agoderp, see other reply.
- busterarm 6y agoStupid me, wrong company. I was thinking Toshiba. I bought Toshiba 16TB drives -- MG08ACA16TE And hey! 0.0% failure rate!
- zepearl 6y agoBought 8 Toshiba N300 (HDWN180, 8TB) consumer HDDs, maybe about one year ago, and they're all still working.
- jedberg 6y ago> The AFR for 2020 dropped below 1% down to 0.93%. In 2019, it stood at 1.89%. That’s over a 50% drop year over year... In other words, whether a drive was old or new, or big or small, they performed well in our environment in 2020. If every drive type, new and old, big and small, did better this year, maybe they changed something in their environment this year? Better cooling, different access patterns, etc. If this change doesn't have an obvious root cause, I'd be interested in finding out what it is if I were Backblaze. It could be something they could optimize around even more.
- codezero 6y agoI was wondering if the state of the world in 2020 might have dramatically changed their business / throughput / access patterns in a meaningful enough way to cause this dip. I'm not sure if they have a measure of the disk utilization or read/write load along with the failure rate.
- brianwski 6y agoDisclaimer: I work at Backblaze, but mostly on the client that runs on desktops and laptops. > I'm not sure if Backblaze has a measure of the disk utilization or read/write load along with the failure rate. We publish the complete hard drive SMART stats for anybody to attempt these analysis. Most of us in Backblaze engineering get giddy with excitement when a new article comes out that looks at correlating SMART stats and failures. :-) For example, this article circulated widely at Backblaze a few days ago: https://datto.engineering/post/predicting-hard-drive-failure-with-machine-learning https://datto.engineering/post/predicting-hard-drive-failure...
- codezero 6y agoI anticipate these reports every year and have strong trust in the data - I want to make that clear - Backblaze has done a massive service to the entire industry by collecting and aggregating this kind of data. I'm really super curious about the dip in errors over the past year :)
- thom 6y agoI have always loved these posts, they paint a picture of smart people with good processes in place. It's confidence-building. Unfortunately we were evaluating Backblaze around the time they went down for a weekend and didn't even update their status page, which was a bit of a blow to that confidence.
- andruby 6y agoI hadn't heard about this downtime. When did that happen?
- thom 6y agoThis was the HN thread: https://news.ycombinator.com/item?id=25147951 https://news.ycombinator.com/item?id=25147951 And I must apologise, I was wrong when I said they didn't update their status page during the outage. They don't have a status page.
- csnover 6y agoTheir marketing blog does a great job of painting them as smart people with good processes. Sadly, I learned the hard way (by trialling their software and immediately discovering a handful of dumb bugs that should’ve been caught by QA, plus serious security problems[0], and some OSS licence violations[1]) that it seems to not actually be the case. This situation where they continue to pump out blog posts about hard drive stats, yet don’t even have a status page for reporting outages, is another example of their marketing-driven approach to development. I have mentioned this on HN a couple times now[2][3], including again just yesterday. I really dislike doing this because I feel like I am piling on—but as much as I hate it, I feel even more strongly that people deserve to be informed about these serious ongoing failures at Backblaze so that they can make more informed choices about what storage/backup provider to use. I also genuinely hope that I can incentivise them to actually start following software development best practices, since they do provide a valuable service and I’d like them to succeed. If they keep doing what they’re doing now, I absolutely expect to see a massive data breach and/or data loss event at some point, since they are clearly unwilling or unable to properly handle security or user privacy today—and I’ve gotten the impression over time that some prominent people at the company think these criticisms are invalid and they need not make any substantive changes to their processes. [0] https://twitter.com/zetafleet/status/1304664097989054464 https://twitter.com/zetafleet/status/1304664097989054464 [1] https://twitter.com/bagder/status/1215311286814281728 https://twitter.com/bagder/status/1215311286814281728 [2] https://news.ycombinator.com/item?id=25899802 https://news.ycombinator.com/item?id=25899802 [3] https://news.ycombinator.com/item?id=24839757 https://news.ycombinator.com/item?id=24839757
- lousken 6y agoWith such a high capacity drive I wonder if they changed anything in terms of redundancy - something like zfs resilvering prob takes days on these
- andy4blaze 6y agoAndy from Backblaze here. Larger drives do take longer to rebuild, but to date we haven't changed the encoding algorithms we built. There are other strategies like cloning which can reduce rebuild time. We can also prioritize rebuilds or drop a drive into read-only mode as needed. The system was built expecting drive failures.
- _nickwhite 6y agoIf a Backblaze engineer (or Andy) is reading this, could you comment on environmental temps and vibration you guys keep the disks at? Thanks.
- andy4blaze 6y agoAndy at Backblaze here. All the drives are in data centers with temps around the 75-78 degree mark. Vibrations are kept to a minimum via the chassis design. We publish the data, including the SMART stats for all of the drives and there are attributes for temperature (SMART 194) and vibration (multiple) see https://en.wikipedia.org/wiki/S.M.A.R.T https://en.wikipedia.org/wiki/S.M.A.R.T. for more info in SMART attributes.
- _nickwhite 6y agoThanks Andy! Without me digging through the SMART data, has there been any difference in data center temps over the years for Backblaze? I ask because I've personally seen disk life vary greatly between "warm" datacenters (closer to 79F) and "cool" datacenters (closer to 72F). I don't have a huge dataset, only anecdotal evidence, but it seems to me temperature plays a pretty big role in drive longevity. Have you guys found the same, or is this a variable not controllable by Backblaze?
- tyingq 6y agoI'm always excited for this yearly post. Are there other vendors that provide this kind of insightful info for other types of infrastructure? Also, kudos to Backblaze. I'm sure there's some side brand benefit to all the work that goes into making this public, but it's clear it's mostly just altruism.
- patentatt 6y agoApologies for being slightly off-topic, but presenting a table of text as a image is annoying to me. A table of text ought to be rendered in just plain old html in my old-school opinion.
- qwertox 6y agoI agree. I was reading the Q3 2020 stats yesterday because I'm looking for a new drive. It was somewhat annoying to have to type the HDD-model into the Google search bar instead of just double-clicking and selecting search from the context menu. It irritated me that it was an image.
- andy4blaze 6y agoAndy from Backblaze here. Actually you can download a spreadsheet with all the data from the tables. There's a link at the end of the post. Better than parsing HTML for the data.
- JxLS-cpgbe0 6y agoThe spreadsheets are identical to the screenshots you took of them. The images aren't responsive, aren't available to assistive technologies, provide no text alternative, do not respect a user's text settings, cannot be translated etc. Why is that better than HTML?
- wellthisisgreat 6y agoSince this is a thread about HDDs, can someone recommend a quiet HDD? Or is it a pipe dream and one should stick to SSDs?
- lostlogin 6y agoI can do the opposite - 16tb Seagate exos drives are very loud. Great drives, but horrible noise. If I were you I’d be looking at slow ones, 5400 WD drives.
- sliken 6y agoSadly, at least in some cases the "5400" drives, are just relabled 7200 drives at a lower price point. Apparently it's cheaper than making actually 5400 drives.
- glenneroo 6y agoSince moving to Fractal Design Define (i.e. soundproofed) version 4/5/6/7 cases over the past ~10 years, the only noise I ever hear anymore is from the fans, primarily GPU and case fans when running heavier jobs. In my main system I have 6 spinning rust drives (8-16TB from various manufacturers) running 24/7 and I never hear them, even during heavy read/writes... and I often sleep on the couch nearby ;)
- wellthisisgreat 6y agoI actually have Fractal Define 7XL but am hesitant to get HDDs bc haven’t used them for some 10 years. Does FD7 isolate HDD noise to the level of for example AIO noise?
- glenneroo 6y agoI'd recommend checking out some YouTube reviews in regards to water-cooling, several have talked about AIO but I skipped over those parts. I don't have AIO but if you want to keep absolutely silent, I would suggest mounting AIO fan-kit in the front of the 7XL instead of up top since that requires removing some of the critical soundproofing.
- neogodless 6y ago> Over the last year or so, we moved from using hard drives to SSDs as boot drives. We have a little over 1,200 SSDs acting as boot drives today. We are validating the SMART and failure data we are collecting on these SSD boot drives. We’ll keep you posted if we have anything worth publishing. Would love to see SSD stats like this in the future. Recently was talking to some friends about what SSD to buy. I personally really like my HP EX950 - one friend said he'd never buy HP hardware. He said he was getting an Intel - I said I had an early Intel SSD fail on me, and I don't think QLC is the best option, but it is a nice value play. For performance, I do like Samsung, though they are expensive. Another friend said he'd never buy a Samsung SSD, as he had a reliability issue, and found lots of similar stories when he was researching it. Of course these are all anecdotes and they aren't useful in making an informed choice. I suspect most SSDs are reliable "enough" for most consumer use, and not nearly reliable enough for certain kinds of critical storage needs. But it would still be nice to see the big picture, and be able to factor that into your SSD purchase decisions.
- peter_d_sherman 6y agoThis brings up a good related question, which is: What is the most reliable hard drive of all time, and why? In other words, let's say I didn't care about capacity, and I didn't care about access time or data transfer rate, and I'd like a drive to store data 100, 200+ years into the future and have it have a reasonable chance of getting that data there -- then what kind of hard drive should I choose, and why? It's a purely philosophical question... Perhaps I, or someone else, should ask this question on Ask HN... I think the responses would be interesting...
- PhantomGremlin 6y agoI'd like a drive to store data 100, 200+ years into the future and have it have a reasonable chance of getting that data That's flat out impossible given current drives. So many mechanical parts. So much to go wrong, such as air seals, bearings, etc. Even if the drive survived, will you be able to connect to its interface? No! If you want to store data for 100 years, you will need to copy that data over, approximately every 5 years, onto the then current storage medium.
- sangnoir 6y agoHow does Backblaze source its hard drives? I think their reports are getting big enough to warrant the attention of the manufacturers marketing departments, who are incentivized to game the system by supplying non-representative disks (i.e. the "good ones") to get an edge over the competition, if they can. I suspect any model that looks good in the report will likely shift more numbers than it otherwise would have.
- atYevP 6y agoYev from Backblaze here -> We work with the drive manufacturers and their distributors! One of the reasons we think it's important to state the drive model is so people know exactly what is getting deployed versus just using ___ 14TB or ___16TB. But we source our drives just like any other large cloud storage provider would!