4 ms·
I have definitely seen RAID arrays where the drives were all part of a single manufacturing batch, and multiple drives all failed in rapid succession. I think t
by ekidd 4y ago
I have definitely seen RAID arrays where the drives were all part of a single manufacturing batch, and multiple drives all failed in rapid succession. I think this can be caused by several things:
- Unless you periodically do full-drive reads, you may silently accumulate bad blocks across multiple drives in an array. When you finally detect a failed drive, you discover that other drives have also been failing for months.
- A full RAID rebuild is a high-stress event that tries to read every disk block on every drive, as rapidly as possible.
- And finally, some drive batches are just dodgy, and it may not take much to push them over. And if identically dodgy drives are all exposed to exactly the same thermal stress and the same I/O operations, then I guess they might fail close together?
Honestly, RAID arrays only buy you so much reliability. Hardware RAID controllers are another single point of failure. I once lost two drives and a RAID controller all together during Christmas, which was not a fun time.
I do like the modern idea of S3-like storage, where data is replicated over several independent machines, and the controlling software can recover from losing entire servers (or even data centers). It's not a perfect match for everything, but it works great for lots of things.
- ericbarrett 4y agoYou are spot on with everything, especially RAID controllers. I used to help manage a large fleet of database servers. We found that blocks could "rot" on the underlying storage, yet if they were read often enough they would be held in memory for months and never re-read from the underlying drive. Until you rebooted!
- water8 4y agoYes bitrot is a huge problem with mechanical hard drives and media that hasn’t been read in a long time. What you write to the drive might not be what you read back five years later. That’s why ZFS is critical for systems like this where you have checksums for each block of data and can rebuild from parity if there is a mismatch
- water8 4y agoZFS is the last file system you will ever need
- Mo3 4y ago100% spot on on RAID, I actually had it happen once that a second disk failed while under load of rebuilding the array after the first disk failure, not related to the SSD issue.