9 ms·
Are hard drives getting better?
- thisislife2 1y agoPersonal anecdote - I would say (a cautious) yes. Bought 3 WD hard drives (1 external, 2 internal, during different time periods; in the last 10+ years) for personal use and 2 failed exactly after the 5 year warranty period ended (within a month or so). One failed just a few weeks before the warranty period, and so WD had to replace it (and I got a replacement HDD that I could use for another 5 years). That's good engineering! (I also have an old 500GB external Seagate drive that has now lasted 10+ years, and still works perfectly - probably an outlier). That said, one thing that I do find very attractive in Seagate HDDs now is that they are also offering free data recovery within the warranty period, with some models. Anybody who has lost data (i.e. idiots like me who didn't care about backups) and had to use such services knows how expensive they can be.
- kvemkon 1y ago> replacement HDD that I could use for another 5 years But the warranty lasts only 5 years since the purchase of the drive, doesn't it?
- thisislife2 1y agoYes, but the warranty is "irrelevant" when the drive actually last the whole 5 years (in other words, I am hoping the replacement drive is as well-engineered as its predecessor and lasts the whole 5 years - and it has so far in the last 3+ years).
- drtgh 1y ago> 5 years [...] well-engineered [...] Programmed obsolescence, evilness, should not be rewarded like this.
- thisislife2 1y agoThat has been my experience (and conclusion) too with WD - it does seem very much like it has been engineered for planned obsolescence. That's why I chose to buy from their competitor this year. Consumers though have less and less option today as WD has already gobbled up the others and now there are only 3 major ones left - Seagate, Toshiba and WD. The other alternative is flash drives, but price and data recovery is a big issue with it that holds me back from fully switching to it.
- terribleperson 1y agoHDD manufacturers offering data recovery...kind of makes sense, and I'm surprised it's never been offered before. They're in a much better position to recover data than anyone else.
- 20after4 1y agoI've bought a lot of WD drives over the years and my experience is they used to last 3 years (back when there was a 3 year warranty) and die right after the warranty expired. I think western digital does a very good job making their drives last to the end of the warranty and not a minute longer.
- gfody 1y agopleasant contradiction to betteridge's law
- eqvinox 1y agoI'm curious what this data would look like collated by drive birth date rather than (or in 3D addition to) age. I wouldn't use that as the "primary" way to look at things, but it could pop some interesting bits. Maybe one of the manufacturers had a shipload of subpar grease? Slightly shittier magnets? Poor quality silicon? There's all kinds of things that could cause a few months of hard drive manufacture to be slightly less reliable… (Also: "Accumulated power on time, hours:minutes 37451*:12, Manufactured in week 27 of year 2014" — I might want to replace these :D — * pretty sure that overflowed at 16 bit, they were powered on almost continuously & adding 65536 makes it 11.7 years.)
- sdenton4 1y agoI think it's helpful to put on our statistics hats when looking at data like this... We have some observed values and a number of available covariates, which, perhaps, help explain the observed variability. Some legitimate sources of variation (eg, proximity to cooling in the NFS box, whether the hard drive was dropped as a child, stray cosmic rays) will remain obscured to us - we cannot fully explain all the variation. But when we average over more instances, those unexplainable sources of variation are captured as a residual to the explanations we can make, given the avialable covariates. The averaging acts a kind of low-pass filter over the data, which helps reveal meaningful trends. Meanwhile, if we slice the data up three ways to hell and back, /all/ we see is unexplainable variation - every point is unique. This is where PCA is helpful - given our set of covariates, what combination of variables best explain the variation, and how much of the residual remains? If there's a lot of residual, we should look for other covariates. If it's a tiny residual, we don't care, and can work on optimizing the known major axes.
- realityfactchex 1y agoPer charts in TFA, it looks like some disks are failing less overall, and failing after a longer period of time. I'm still not sure how to confidently store decent amounts of (personal) data for over 5 years without 1- giving to cloud, 2- burning to M-disk, or 3- replacing multiple HDD every 5 years on average All whilst regularly checking for bitrot and not overwriting good files with bad corrupted files. Who has the easy, self-service, cost-effective solution for basic, durable file storage? Synology? TrueNAS? Debian? UGreen? (1) and (2) both have their annoyances, so (3) seems "best" still, but seems "too complex" for most? I'd consider myself pretty technical, and I'd say (3) presents real challenges if I don't want it to become a somewhat significant hobby.
- LunaSea 1y agoWould tapes not be an option? Not great for easy read access but other than that it might be decent storage.
- dmoy 1y agoI looked at tape a little while ago and decided it wasn't gonna work out for me reliability-wise at home without a more controlled environment (especially humidity).
- gruez 1y ago>Would tapes not be an option? AFAIK someone on reddit did the math and the break-even for tapes is between 50TB to 100TB. Any less and it's cheaper to get a bunch of hard drives.
- aftergibson 1y agoNot from the prices I'm seeing.
- bigwheels 1y agoRecent and related: Disk Prices https://news.ycombinator.com/item?id=45587280 https://news.ycombinator.com/item?id=45587280 - 1 day ago, 67 comments
- hatmatrix 1y agoDo we have enough rare earth metals to provide storage for the AI boom?
- jpitz 1y agoThe question is, do we have enough capacity to mine and refine them at a reasonable price? They're there, in the dirt for the taking.
- asdff 1y agoFuture generations will blame us for damning them out of rare earths to build yet another cellphone. This is like us today with severely diminished whale populations just so Victorians could read the bible for another 2 hours a night. Was it worth it? Most would say no, save for the people who made a fortune off of it I'm sure.
- deepspace 1y agoThat makes no sense whatsoever. We are not consuming rare earths; only moving them from one place to another. Arguably, future generations would find it easier to mine them from former landfill sites, where they would be present in concentrated form, than from some distant mine in the middle of nowhere.
- _factor 1y agoSounds mighty expensive if not impossible for extraction.
- hshdhdhehd 1y agoWhy not do that now?
- jandrese 1y agoIt's called recycling. As long as the "virgin" sources are super cheap its not worth it, but the market can change.
- yalogin 1y agoAh I haven’t seen the yearly backblaze post in some time now, glad it’s back.
- fooker 1y agoHard drives are not getting better. Hard drives you can conveniently buy as a consumer - yes. There's a difference.
- billfor 1y agoI have a 13 years old NAS with 4x1TB consumer drives with over 10y head flying hours and 600,000 head unloads. Only 1 drive failed at around 7 years. The remaining 3 are still spinning and pass the long self test. I do manually set the hdparm -B and -S to balance head flying vs unloads, and I keep the NAS in my basement so everything is thermally cool. I'm kinda of hoping the other drives will fail so I can get a new NAS but no such luck yet :-(
- kiddico 1y agoI admire the "use it until it dies" lifestyle. My NAS is at 7 years and I have no plans to upgrade anytime soon!
- Loughla 1y agoThe problem with setting a nearly maintenance free nas is that you tend to forget about it just running away in the background. Then a drive fails spectacularly. And that's the story of how I thought I lost all our home movies. Luckily the home movies and pictures were backed up.
- fnands 1y agoNo RAID?
- dspillett 1y agoIf you are only going to have one of the two, choose backups, preferably off-site, even better soft-offline, over RAID. Of course both is best if you don't consider the cost of doubling up your storage (assuming R1/R10) and having backup services to be a problem.
- lossyalgo 1y agoRAID isn't a backup, it only handles certain/specific failure scenarios.
- PaulKeeble 1y agoWhen I am projecting prices I tend to assume a 5 year life for a consumer hard drive. I do wonder from this data and the change in purchasing from backblaze if the enterprise class drives might pay for their extra price if they survive out to more like 9 years. 20% extra cost per TB verses about 30%+ more life time. They do tend to consume a bit more power and make more noise as well. I wish they had more data on why the drives were surviving longer, if its purchasing in palettes there isn't a lot we can do, but if its that enterprise drives are a lot better than NAS or basic consumer drives then that we compare cost wise.
- arscan 1y ago> The issue isn’t that the bathtub curve is wrong—it’s that it’s incomplete. Well, yeah. The bathtub curve is a simplified model that is ‘wrong’, but it is also a very useful concept regarding time to failure (with some pretty big and obvious caveats) that you can broadly apply to many manufactured things. Just like Newtonian physics breaks down when you get closer to the speed of light, the bathtub curve breaks down when you introduce firmware into the mix or create dependencies between units so they can fail together. I know the article mentions these things, and I hate to be pedantic, but the bathtub curve is still a useful construct and is alive and well. Just use it properly.
- dan-robertson 1y agoI feel like I’d like to see graphs in the shape you see in some medical trials – time on the x axis and % still alive on the y. You could group drives by the year they were purchased and have multiple lines for different years on there.
- hinkley 1y agoSo I had a random thought about what is the most platters that any hard drive has had. I looked it up it seems that the Western Digital Ultrastar® DC HC690 has eleven platters in a 3.5” form factor. That certainly gives you a lot more bandwidth, though not much help for seek time (unless you do the half-allocated trick).
- kasabali 1y agoI wonder why they don't do double height form factors at this point instead of trying to squeeze 11 platters with helium and micron level tolerances
- jandrese 1y agoBecause most of those drives are getting slotted into sleds in arrays. A double height drive might work fine in a PC case (at least if you're willing to break out a dremel), but the majority of the buyers of big drives are not using ordinary PC cases.
- hinkley 1y agoMost of the bandwidth of a drive is out at the edge of the platter, so you don’t want to make them smaller in order to improve the seek time or make the read arms thinner to stack more platters. Which increases capacity less dramatically each time anyway. There’s an old trick though that if you want a faster drive, only partition part of the space on each drive and thus reduce the average seek time. What one could do is make the axle bigger, shorten the arms, make them a little thinner and add another platter. But the next one is only 9% and that’s if you don’t take 9% off the area to accomplish it. Otherwise you get the same space but faster seek time.
- citizenpaul 1y agoIs backblaze single highhandedly driving QC on hard drive manufacturers with their yearly report? Might be.
- dooglius 1y agoIt seems odd to look at failure rate in isolation, without considering cost and density; at scale, improved cost and density can be converted to lower failure rates via more aggressive RAID redundancy, no?
- p0w3n3d 1y agoConnected, but quite different to this subject, is how to long term store photos (cloud does not count). HDD still seem to be the best solution, but not sure how often should I rewrite them
- Mistletoe 1y agoPrint out the ones you like and put them in an album or on the wall. Think how many photos you have that are like that in a family and still around, when all the rest are gone on dead phones or computers somewhere.
- axiolite 1y agoYou can't get a perfect digital copy of a printed out photo. You're subjecting yourself to generational losses for no good reason. If you're a fan of paper, you could base64 encode the digital photo and print that out onto paper with a small font, or store the digital data in several QR codes. You can include a small preview too. But a couple hard drives or microSD cards will hold many millions of times as many photos in less physical space.
- pastage 1y agoDoing paper backups is hard I could not store much data on paper when I last tried with QR codes. You need to to multiple QR codes since they are only 2KB each. There are some threads about it on HN, but I have never seen anyone do megabytes per paper https://news.ycombinator.com/item?id=29792556 https://news.ycombinator.com/item?id=29792556 https://news.ycombinator.com/item?id=31149427 https://news.ycombinator.com/item?id=31149427 https://news.ycombinator.com/item?id=24669425 https://news.ycombinator.com/item?id=24669425 https://hn.algolia.com/?q=paper+backup https://hn.algolia.com/?q=paper+backup Cuneiform tables do it in clay.
- axiolite 1y agoPaperBack claims 0.5MB per page: https://news.ycombinator.com/item?id=39883531 https://news.ycombinator.com/item?id=39883531 It's likely not possible to do much better. I never said that was the most practical method.
- Fnoord 1y agoDoes this take into account the scandal of old drives being sold as new?
- TimByte 1y agoSo hard drives are getting better but just as importantly, we're also getting smarter about how we use and maintain them
- rcthompson 1y agoIf hard drives increase in capacity while maintaining the same MTBF, does this count as an improvement? If you previously stored your data on 10 drives and now you can store the same data on 5 drives, that reduces the probability of failure of the system as a whole, right? Is there some kind of "failure rate per byte" measure that normalizes for this?
- Chiron1991 1y agoI don't know about this exact metrics, but the Backblaze hard drive report is always a very good read when thinking about failure rates. Maybe check it out and see if you'll get your answers there.
- Retric 1y agoIt depends on what you’re doing and what you’re concerned about. For a simplified example suppose you have X drives storing 20TB vs 2X drives with 10TB in a simple RAID 1 configuration. When a drive fails there’s a risk period before its contents are replicated on another drive. At constant transfer speeds larger disks double that period per drive but half the number of failures. Net result the risk is identical in both setups. However, that assumes a constant transfer speeds, faster transfer rates reduce overall risks.
- rcthompson 1y agoHmm, I hadn't considered that doubling the drive size doubles the resilver time and therefore doubles the exposure time for risk of array loss. I guess the math gets complicated depending on RAID topology.
- hbn 1y ago> If you previously stored your data on 10 drives and now you can store the same data on 5 drives, that reduces the probability of failure of the system as a whole, right? Well it also means in the case of failure you get 2x the spread of damage across the same amount of data
- qwertox 1y agoSlightly related: Anyone noticed how sloppy Samsung has become with the reliability of their NVMe firmware? I learned a lesson when I nearly lost a ton of data in a ZFS mirror of two Samsung Pro 990 4TB NMVes, they might have been from the same batch and had the same firmware version. First one NVMe abandoned the system, then I made a ZFS send of a snapshot of the remaining one, followed by an rsync to have the raw data directly accessible as well, and during the rsync the second NVMe also abandoned ship. A reboot fixed it, and all data was still there, but only upgrading the firmware (to one which now has been superseded again) fixed the problem. I'm really considering no longer buying Samsung Pro NVMes.
- fouc 1y agoBeen hearing of drive failures lately with Samsung, Sandisk, and even Crucial. Possibly because the bulk of recent drive production is getting reserved by the AI datacenters?
- rbranson 1y agoThe 990 Pros have a temperature sensor issue. See https://github.com/openzfs/zfs/discussions/14793#discussioncomment-13130186 https://github.com/openzfs/zfs/discussions/14793#discussionc...
- qwertox 1y agoI think this was either a `number of TBW (terabytes written)` or `% of space used` issue, since both got removed by the OS within 9 hours delta and the same usage (couple of months, ~500GB), because they were in a mirror since the beginning. If it were a sensor issue, SMART data should have shown this. Not saying that a sensor issue does not exist, but I doubt that this was my problem (mine don't have a preinstalled heatsink, I prefer to use my own). I now use a 3-way mirror and am mixing brands. One very nice thing: the Samsung Pro 990 4TB has the exact same space (down to the byte) as the WD_BLACK SN850X 4TB, so they can be replaced without any issues. This rarely was the case with SSDs and HDDs and probably other NVMes. Looks like they learned.
- pkulak 1y ago
- rcthompson 1y agoOf note, assuming that decommissioning of drives is driven primarily by e.g. space concerns rather than signs of impending individual drive failures (which seems to be the case based on the linked article about storage scaling), you could conduct a survival analysis in which decommissioned drives are treated as right-censored to get a better measure of the failure rate over time as well as how that failure rate depends on various factors. Note that the most common choice of a proportional hazards model may not be appropriate here, and an accelerated failure time model may be more appropriate, although I couldn't say for sure without actually working with the data.