4 ms·

Ok, so one thing that is bugging me: This is totally a software based problem. At the moment, if I have a big hard disk I'm almost certainly going to have some

by CHY872 12y ago

Ok, so one thing that is bugging me: This is totally a software based problem. At the moment, if I have a big hard disk I'm almost certainly going to have some kind of big read error in at least a couple of sectors - which practically means that I'll get some garbled bytes on some reads, and I hope that they're in a cache file, or swap file, and not my registry, or programs, etc. So here, they're saying that if you have a RAID 5, and one disk fails, then when it has such an error, it's suddenly game over for the entire array - the rest of the data on the drive must be copied to a new array, because the firmware says so. Apparently that's why RAID 5 is a bad idea. Surely the same argument can be applied to hard drives on their own? If Windows checksummed every file before using it, and forced copying every byte onto a new hard drive whenever an error was found, hard drives themselves would stop working as a meaningful method of storage in 2009. But it doesn't, you (likely) get some garbled text, and it's usually not a problem. File systems get corrupted etc etc (unless you use something like ZFS), and RAID 5 handling it in a particularly non-graceful way is just an implementation detail. So really, this is just a problem with the controller firmwares forcing you to make a whole new array. It's not a problem with RAID 5. Or have I missed something? I know that RAID 5's not a good choice in this modern age (cloud, big SANs etc, SSDs etc), but surely the 'a sector will be bad, and you will not be able to rebuild after a failure' problem would be easily fixed by the raid controller company if their clients complained enough?

simplexion 12y ago
No. Here is a good explanation: http://www.zdnet.com/blog/storage/why-raid-5-stops-working-in-2009/162 http://www.zdnet.com/blog/storage/why-raid-5-stops-working-i...
- CHY872 12y ago
  I don't believe the contradicts what I've said at all. His claim is that when one disk fails completely, we expect a sector or two on each of the two remaining drives to result in a 'bad sector' error. > So the read fails. And when that happens, you are one unhappy camper. The message "we can't read this RAID volume" travels up the chain of command until an error message is presented on the screen. 12 TB of your carefully protected - you thought! - data is gone. Oh, you didn't back it up to tape? Bummer! So, at this point, we've got two hard drives containing millions of sectors for which one or two are bad, and the software claims that the whole thing is broken, and we have to find a new array? As far as I know, RAID 5 is block level - the block level failure should only destroy one block. All of the others (apart from the other dead ones) are fine. This sort of thing happens in all scenarios with a single disk - eventually an operating system will hit a bad sector which it will have to deal with. In other words, why does the RAID controller crap itself when it can't read (with no recourse at all, according to these articles) when it could just do what every other hard drive does and return 'sector unreadable'. Then the operating system can just remap it etc. I know in some situations one would want to be notified of any miniscule error, but it should be possible to ignore the warnings.
  - micro-ram 12y ago
    I think the assumed issues here are the long rebuild times and the expectation of an additional failure happening during the rebuild which could then trigger another disk error. Then the entire array of data would be taken offline and possibly corrupted. I have build many RAID5/6 arrays and have rarely lost any data, but I am leery and now and tend to just stick with smaller RAID 1 or 10's due to the large size of current disks. We really need native ZFS (BTRFS?) on everything now. Data should be automatically distributed to multiple disks and the file system should be able to guarantee via checksum the data read is what I wrote.
    - venus 12y ago
      Are any consumer NAS that implement ZFS even available? Sounds like ZFS's proactive sector sweeps across all managed drives would handily solve the problem the article raises with conventional RAID.
      - lmz 12y ago
        iXsystems sells the 4-bay FreeNAS mini on Amazon: http://www.ixsystems.com/storage/freenas/ http://www.ixsystems.com/storage/freenas/ That probably counts as "consumer".
      - mitchty 12y ago
        So I built a zfs raid nas box with 6x3Tb drives. I have it resilver every 2 weeks. So far, NOTHING has failed checksums for almost 2 years. So while the maths behind a 3-4 tb drive returning incorrect data I'm sure is technically correct, I've not seen issues. If you want "proof" i can dump out zpool status/info/log/etc to show i'm not lying. Note the pool is ~50% in use so its not a great example. Also its raidz2 (raid6) so not a direct comparison. I also bought each drive from different lots to hopefully ensure if a drive failed i'd have 2ish days to get a replacement.
- dannyperson 12y ago
  What happens when a URE is encountered and all the disks are online? It seems that they could be detected early and fixed before a rebuild is necessary by doing a weekly sweep of the entire array, reading every data block.
  - harshreality 12y ago
    With raid 5 if you get a parity block mismatch, you don't know which drive was wrong. You could compare parity bit by bit, to find which bits (it could be anywhere from 1 bit up to the stripe size) was wrong, but you won't be able to figure out how to fix those bit(s) on the stripe without additional information, either a hash or FEC data. Checksumming filesystems let you find the faulty drive by reconstructing data from each n-choose-(n-1) drive set and finding the set with the correct hash. Filesystems using FEC instead of raid (5/z1, 6/z2 ...) can also correct data errors, but I'm not aware of any consumer-level filesystems that implement it. I'm not sure why. Doesn't Amazon use it for S3? Data block and FEC data layout on a disk array has to be a solved problem.
baruch 12y ago
If it's only a sector that died on you in one disk the likelyhood that the equivalent sector in the RAID stripe also failed is fairly small[1]. The most often cited issues is with a full disk failure and then having a sector already dead in another disk that you weren't aware of. Do disk scrubs and you will mostly avoid this risk. Your array (software or hardware) should be capable of disk scrubs, most if not all are capable and will do it. [1] There is a failure mode whereby frequent writes to one sector/track will cause damage to nearby sectors in which case if your RAID stripes are static you may have multiple sector failures in multiple disks in the same stripe. It does require hundreds or thousands of rewrites for all I know so your workload needs to be extreme to reach that case.

by CHY872 12y ago

simplexion 12y ago

No. Here is a good explanation: http://www.zdnet.com/blog/storage/why-raid-5-stops-working-in-2009/162 http://www.zdnet.com/blog/storage/why-raid-5-stops-working-i...

CHY872 12y ago
I don't believe the contradicts what I've said at all. His claim is that when one disk fails completely, we expect a sector or two on each of the two remaining drives to result in a 'bad sector' error. > So the read fails. And when that happens, you are one unhappy camper. The message "we can't read this RAID volume" travels up the chain of command until an error message is presented on the screen. 12 TB of your carefully protected - you thought! - data is gone. Oh, you didn't back it up to tape? Bummer! So, at this point, we've got two hard drives containing millions of sectors for which one or two are bad, and the software claims that the whole thing is broken, and we have to find a new array? As far as I know, RAID 5 is block level - the block level failure should only destroy one block. All of the others (apart from the other dead ones) are fine. This sort of thing happens in all scenarios with a single disk - eventually an operating system will hit a bad sector which it will have to deal with. In other words, why does the RAID controller crap itself when it can't read (with no recourse at all, according to these articles) when it could just do what every other hard drive does and return 'sector unreadable'. Then the operating system can just remap it etc. I know in some situations one would want to be notified of any miniscule error, but it should be possible to ignore the warnings.
- micro-ram 12y ago
  I think the assumed issues here are the long rebuild times and the expectation of an additional failure happening during the rebuild which could then trigger another disk error. Then the entire array of data would be taken offline and possibly corrupted. I have build many RAID5/6 arrays and have rarely lost any data, but I am leery and now and tend to just stick with smaller RAID 1 or 10's due to the large size of current disks. We really need native ZFS (BTRFS?) on everything now. Data should be automatically distributed to multiple disks and the file system should be able to guarantee via checksum the data read is what I wrote.
  - venus 12y ago
    Are any consumer NAS that implement ZFS even available? Sounds like ZFS's proactive sector sweeps across all managed drives would handily solve the problem the article raises with conventional RAID.
    - lmz 12y ago
      iXsystems sells the 4-bay FreeNAS mini on Amazon: http://www.ixsystems.com/storage/freenas/ http://www.ixsystems.com/storage/freenas/ That probably counts as "consumer".
    - mitchty 12y ago
      So I built a zfs raid nas box with 6x3Tb drives. I have it resilver every 2 weeks. So far, NOTHING has failed checksums for almost 2 years. So while the maths behind a 3-4 tb drive returning incorrect data I'm sure is technically correct, I've not seen issues. If you want "proof" i can dump out zpool status/info/log/etc to show i'm not lying. Note the pool is ~50% in use so its not a great example. Also its raidz2 (raid6) so not a direct comparison. I also bought each drive from different lots to hopefully ensure if a drive failed i'd have 2ish days to get a replacement.
dannyperson 12y ago
What happens when a URE is encountered and all the disks are online? It seems that they could be detected early and fixed before a rebuild is necessary by doing a weekly sweep of the entire array, reading every data block.
- harshreality 12y ago
  With raid 5 if you get a parity block mismatch, you don't know which drive was wrong. You could compare parity bit by bit, to find which bits (it could be anywhere from 1 bit up to the stripe size) was wrong, but you won't be able to figure out how to fix those bit(s) on the stripe without additional information, either a hash or FEC data. Checksumming filesystems let you find the faulty drive by reconstructing data from each n-choose-(n-1) drive set and finding the set with the correct hash. Filesystems using FEC instead of raid (5/z1, 6/z2 ...) can also correct data errors, but I'm not aware of any consumer-level filesystems that implement it. I'm not sure why. Doesn't Amazon use it for S3? Data block and FEC data layout on a disk array has to be a solved problem.

baruch 12y ago

If it's only a sector that died on you in one disk the likelyhood that the equivalent sector in the RAID stripe also failed is fairly small[1]. The most often cited issues is with a full disk failure and then having a sector already dead in another disk that you weren't aware of. Do disk scrubs and you will mostly avoid this risk. Your array (software or hardware) should be capable of disk scrubs, most if not all are capable and will do it. [1] There is a failure mode whereby frequent writes to one sector/track will cause damage to nearby sectors in which case if your RAID stripes are static you may have multiple sector failures in multiple disks in the same stripe. It does require hundreds or thousands of rewrites for all I know so your workload needs to be extreme to reach that case.