5 ms·
I have one of these and have used it high yet, so looking forward to that. In other news, all of the WDRed CMR drives in my ZFS array started throwing read err
by alias_neo 3y ago
I have one of these and have used it high yet, so looking forward to that.
In other news, all of the WDRed CMR drives in my ZFS array started throwing read errors simultaneously so I've ripped them all out.
I'm about done with WD.
- gruez 3y ago>In other news, all of the WDRed CMR drives in my ZFS array started throwing read errors simultaneously so I've ripped them all out. I don't get it, is the implication that them showing errors at time should be a freak occurrence, and such an occurrence means that WD drives are low quality? Drive failures are more correlated than you think. If you bought and installed the drives at the same time, such a failure case is totally expected. The drives probably came from the same batch, and experienced similar amount of wear, so it's only logical that they experienced the same failure case.
- b112 3y agoThese are spinning disks, not SSDs. Spinning disks are not known to fail like this, unless the manufacturer made a horrible mistake.
- gruez 3y agoOn the contrary. The same principle applies to them. Spinning disks might not have NAND cells that wear out, but they have other parts that have manufacturing defects and can experience wear: bearings, actuators, heads, etc. If you have 100 identical cars that rolled off the assembly line in sequence, you would their failure date to be be clustered, rather than being evenly distributed.
- nottorp 3y agoOn a tangent, wouldn't it make it safer if you didn't start a NAS with a bunch of identical drives, but you got them from different manufacturers and models? They'll still fail, but hopefully they'd be less likely to do it at the same time.
- Filligree 3y agoYes, that’s generally good advice.
- stOneskull 3y agothis is news to me and taken away my excitement of having my first nas. it's very good to know though. i wonder about good-guy computer stores, mixing batches for naive customers like me, ordering 2 identical drives thinking it was best. it's crazy, it really defeats the purpose of raid 1 if both drives are likely to fail at the same time. i guess i should buy another different brand drive, and keep one of the current pair in a drawer as the future replacement.
- Filligree 3y agoTo take some further wind out, with current disk sizes there’s a decent chance of a second disk failing while you’re still replacing the first. Reading all the data off a disk is a stressful operation, you see. But that means you need RAIDZ2 or erasure coding to be reasonably safe, which takes you well outside what most of these turnkey systems can handle.
- stOneskull 3y agomaybe i'll start a data-recovery savings account instead.. for whenever it happens
- Filligree 3y agoBackblaze works well enough.
- b112 3y agoSSDs in raids famously all failed at once. Spinning disks basically don't, outside of (typically) firmware faults. Out of 100000 spinning disks I've managed, I've never had them fail at the same time. Mechanical things don't fail as evenly as digital things.
- pixl97 3y agoAs someone that has set up a lot of NAS devices, we're talking thousands of disks over the years, I would not put my money on your statement. There are a lot of simultaneous failure modes. HP had one in SMART firmware that killed disks at some fixed number of hours. There are also temperature excursion events that can lead to arrays failing within 10s of hours of each other, not enough time to do a full rebuild of a single disk. Seagate had really crappy 3/4 tb disks that loved to fail rapidly and nearly all at once.
- dragontamer 3y agoIntel also had a famous bug / 'feature' where the SSD refuses to respond after a fixed number of writes. If all your drives were Intel and part of the same array, they'd all fail at exactly the same time. Reliable storage is hard.
- b112 3y agoSo your examples of failures include a firmware example, which I said "outside of", and an example where the drives were abused? And another poster os talking about SSDs? Sigh
- Volundr 3y agoEh? I've been admining raid arrays since before SSDs were a thing. Making sure you build your array from disks from different batches to reduce the likelihood of simultaneously failing has always been the advice. I make no assertion as to WD quality, but failures like you describe have been the bane of sysadmins quite some time.
- ralferoo 3y agoSpinning disks absolutely do fail this way. Drives from the same batch can easily work 5+ years fine and then fail within a few hours of each other. Even worse, in a RAID the additional load from starting a new replication after a hot swap can push a borderline drive over the edge, and this is precisely the time when your data is at its most vulnerable. I've worked at places where they always maintained a sizable inventory of new disks so that every time a new machine was provisioned, it would receive a RAID comprising of at least 3 disks bought from different batches at least a couple of months apart from each other, and as much as possible from different suppliers.
- Spooky23 3y agoLol. It happens. On my first job a batch of drives started failing daily, in sequential order by serial number. There was a bearing defect - the vendor ended up replacing about 400 drives, which meant many weekends of migration and restores.
- b112 3y agoSounds like the exception that makes the rule. Just because you were in a plane crash, doesn't mean it's common or frequent. (With an apology to that guy who was hit with a nuclear weapon twice)
- Spooky23 3y agoIt happens. That scale is rare, but i can think of 5-6 times that I'm aware of where vendors proactively replace many disks.
- voytec 3y agoAh, the RAID array love story: one of many similar disks fails, you replace it, array starts rebuilding which causes an increase of reads from other drives and before rebuild is complete, more disks from the same batch start failing. I don't deal with storage hardware much for some 6+ years, since I've switched to cloudy solutions. But at the beginning of my shared storage building journey, I considered arrays built from different disks somewhat "lame" or "unprofessional". I changed this viewpoint 180° after seeing multiple similar drives fail in a very short (hours, days) timespan for the second time. I now have more trust for home-built ZFS setup using different disks than a dedicated storage appliance using enterprise-grade disks with almost following serial numbers.
- katbyte 3y agoRaid array should always be scrubbed every few months to prevent this exact situation
- nabeards 3y agoScrubbed as in erased? Where do you temporarily place 1PB+ of data while that’s happening?
- bell-cot 3y ago`zpool scrub $Name_Of_zfs_Storage_Pool` https://docs.oracle.com/cd/E18752_01/html/819-5461/gbbwa.html https://docs.oracle.com/cd/E18752_01/html/819-5461/gbbwa.htm...
- duskwuff 3y agoScrubbed, as in performing a periodic read of the entire array to 1) check the data for consistency and 2) exercise the drives.
- deleted 3y ago[deleted]
- singron 3y agoScrubbing helps fix random errors in the drive data, but it won't prevent correlated disk failure. If a drive fails during the scrub, then you still have to replace the disk, and it's still very likely that another disk can fail in that time. If you just want to eliminate lemons, then you can do a burn-in test before adding it to your array, but IMO even bad disks usually don't fail that quickly.
- gunapologist99 3y agoZFS is very sensitive and takes a long time to rebuild. I only choose RAID-10 for this reason, because I'd rather have most of my data than none of it.
- alias_neo 3y agoI guess so! I was speaking with someone on Mastodon today with some WD Reds and they failed at almost exactly the same number of power on hours as mine (~43k hours). Given mine were in a ZFS mirror, they would have had almost identical wear so I suppose we could say they that they're predictable (though mine are the CMR drives which you don't get in the 4T these days and SMR drives are bad for ZFS). If I was to replace them with the same drives I'd have a rough idea when to replace them before they start failing but I'll be buying larger capacities these days so those numbers are probably out the window.
- praseodym 3y agoCould be defect RAM, unstable power supply or other problem than the disks themselves.
- CTDOCodebases 3y ago> In other news, all of the WDRed CMR drives in my ZFS array started throwing read errors simultaneously so I've ripped them all out. Is this multiple drives purchased around the time failing at the same time? Or is it a bad cable or controller?
- magicalhippo 3y agoI've had multiple cases of IO errors with ZFS, and just two of them were due to the disks. Rest were connection needing reseating or cables needed changing. If all of them went at the same time I'd try swapping to a different controller before I suspected the disks.
- CTDOCodebases 3y agoSame. I would try scrubbing with a different cable and if I was still getting errors I would switch cards and try another scrub.
- alias_neo 3y agoThey were all bought together and had the same load on, so quite reliable/predictable of them to all fail so close together. For the record, they have 43k power-on-hours, though nothing has been written to them much for the last ~3k hours as I had feeling something was off before they started reporting errors and migrated the data away.
- asksomeoneelse 3y agoMore than 2 HDDs starting to fail at the same time would be really bad luck. Maybe the issue is elsewhere, like the HBA or the motherboard ? For the story, I have a NAS with an AMD B550 motherboard and some Asmedia SATA controllers with a bunch of WD HDDs, using ZFS. At some point, they also started to throw weird read errors when under heavy load, but would work just fine the rest of the time. Turned out it was caused by me moving the SATA card from one of the PCIe slots wired directly to the CPU, to a PCIe slot wired to the B550 chipset.