73 ms·

Never underestimate your RAID controller's ability to fail (silently!) and start writing corrupted garbage to your disks.

by ender7 14y ago

its_so_on 14y ago
EDIT: people didn't like my humor. Well, look, the whole thing that you're buying with a raid controller is...redundancy. So if it's not redundant, failing silently, while telling you it's being redundant, how is this different from, say, paying for a house inspection that doesn't get done? If a raid controller is allowed to silently fail, it becomes a post-experience good. http://en.wikipedia.org/wiki/Experience_good http://en.wikipedia.org/wiki/Experience_good Meaning that even while you're using it, you have no idea if it works. My contention is that it's not a raid array if it can silently stop being redundant without telling you. At best it's an Possibly Redundant Array of Inexpensive Disks. (The below is how my comment first read.) (sarcastic) Yeah, it's only prudent to grab a drive out from time to time and make a surprise inspection of whether it's actually filled up a full 4/5th of the way (or whatever) with the actual data the volume is supposed to contain! And the remaining fifth had better look a damn sight like parity information! Seriously though, a controller that fails like this isn't a RAID controller, since what separates it from a paper plate and a cardboard box. On the paper plate you write "RAID controller" and tape it to an already attached hard drive, and you put the remaining members of the redundant array into the cardboard box. No setup or even connection required! seriously seriously though, what you're suggesting is unacceptable. that's not a raid controller, that's a scam.
- eli 14y ago
  Of course a RAID controller isn't supposed to fail silently, but it can and it does. I can't think of many complex pieces of technology that work 100% all of the time.
  - its_so_on 14y ago
    You don't think that something that only exists to create disk redundancy is in a different category from complex pieces of technology that don't have this in their name? I simply disagree that you should "never underestimate" your raid controller's ability to fail silently (which is the comment I was replying to). If this is even on your radar you don't have a RAID controller. This is literally like saying. "Never underestimate your digest algorithm's ability to hash the same file to different values, making the checksum seem to fail." That's not a digest algorithm, that's a randomized print statement. A RAID controller you should 'never underestimate' the ability of to fail silently is literally sometimes the same as a paper plate with "raid controller" written on it. Call it "sometimes raid". or "maybe raid" or "more raid". You don't have a raid controller.
    - eli 14y ago
      I don't know what you're getting at, but I think you are underestimating the chance that your RAID controller will fail.
      - its_so_on 14y ago
        Actually, I don't have a RAID controller. But at least I don't think I have one! If I did get one, I wouldn't get one that tended to silently fail, since that would pretty well defeat the purpose of thinking my disks were redundant, wouldn't it?
        koeselitz 14y ago
        "Tend to fail silently" is different from "could fail silently at some point, though it is unlikely." You are correct that RAID controllers shouldn't be the former; however, there is absolutely no way to prevent RAID controllers from being the latter.
        its_so_on 14y ago
        Could you tell me your reasoning about the last sentence? If you were talking about anything else, like a normal hard-drive, fine, of course it can fail silently. But the whole thing that a RAID drive is, is another layer on top of hard-drives, to make them redundant and chirp loudly when one dies or starts having wrong data and has to be removed, so that you can replace it and rebuild the RAID. I mean, all the RAID controller does is write data that is always redundant (even when it thinks all drives are working fine). How is it not possible for it to check for this consistency as well? Especially in Raid-6 etc configurations, which are even more consistent? Of course, on a probabilistic level random bit rot means "nothing is certain", but on a practical level, how can you not expect a raid controller not to fail silently, when all it does is corral redundant data around, create checksums, verify what's written, etc. It's the whole reason it exists. To me this is like saying that a checksumming algorithm should be expected to sometimes fail and just return a checksum chosen randomly from the space of all possible checksums, instead of the checksum actually produced by the algorithm for that data. That's ridiculous. I shouldn't have to even think about putting another layer on top of the checksum, so that I can checksum it. The very idea of having to do that means you don't have a checksumming algorithm. This thing should be right up there with bitrot causing bash to execute an rm -rf whenever you drop down to root. Sure that's possible, but that's not even in the scope of anything you have to think about. To me, a RAID is a layer on top of hard-drives that makes them redundant. Any controller that has a realistic chance of failing silently simply does not fit that definition.
wayne_h 14y ago
I once did a RAID data recovery on a system with a high-end Intel raid controller. The controller failed and they sent a new one - only the new one couldn't assemble the raid properly. It turns out that there was a flaw in the logic for where parity was stored. Normally parity is spread evenly across all the drives - not on this version. I had to reverse-engineer the crazy raid pattern and write a program to deraid it. It had gone undetected - as long as it was running.
wayne_h 14y ago
and don't underestimate the manufacturers ability to screw it up. Like when people 'upgrade' the firmware in their Buffalo NAS box and after that they can't see the data anymore. Luckily the data was still there and undamaged but it took data recovery to get it back.

Never underestimate your RAID controller's ability to fail (silently!) and start writing corrupted garbage to your disks.

by ender7 14y ago

Never underestimate your RAID controller's ability to fail (silently!) and start writing corrupted garbage to your disks.

its_so_on 14y ago

EDIT: people didn't like my humor. Well, look, the whole thing that you're buying with a raid controller is...redundancy. So if it's not redundant, failing silently, while telling you it's being redundant, how is this different from, say, paying for a house inspection that doesn't get done? If a raid controller is allowed to silently fail, it becomes a post-experience good. http://en.wikipedia.org/wiki/Experience_good http://en.wikipedia.org/wiki/Experience_good Meaning that even while you're using it, you have no idea if it works. My contention is that it's not a raid array if it can silently stop being redundant without telling you. At best it's an Possibly Redundant Array of Inexpensive Disks. (The below is how my comment first read.) (sarcastic) Yeah, it's only prudent to grab a drive out from time to time and make a surprise inspection of whether it's actually filled up a full 4/5th of the way (or whatever) with the actual data the volume is supposed to contain! And the remaining fifth had better look a damn sight like parity information! Seriously though, a controller that fails like this isn't a RAID controller, since what separates it from a paper plate and a cardboard box. On the paper plate you write "RAID controller" and tape it to an already attached hard drive, and you put the remaining members of the redundant array into the cardboard box. No setup or even connection required! seriously seriously though, what you're suggesting is unacceptable. that's not a raid controller, that's a scam.

eli 14y ago
Of course a RAID controller isn't supposed to fail silently, but it can and it does. I can't think of many complex pieces of technology that work 100% all of the time.
- its_so_on 14y ago
  You don't think that something that only exists to create disk redundancy is in a different category from complex pieces of technology that don't have this in their name? I simply disagree that you should "never underestimate" your raid controller's ability to fail silently (which is the comment I was replying to). If this is even on your radar you don't have a RAID controller. This is literally like saying. "Never underestimate your digest algorithm's ability to hash the same file to different values, making the checksum seem to fail." That's not a digest algorithm, that's a randomized print statement. A RAID controller you should 'never underestimate' the ability of to fail silently is literally sometimes the same as a paper plate with "raid controller" written on it. Call it "sometimes raid". or "maybe raid" or "more raid". You don't have a raid controller.
  - eli 14y ago
    I don't know what you're getting at, but I think you are underestimating the chance that your RAID controller will fail.
    - its_so_on 14y ago
      Actually, I don't have a RAID controller. But at least I don't think I have one! If I did get one, I wouldn't get one that tended to silently fail, since that would pretty well defeat the purpose of thinking my disks were redundant, wouldn't it?
      - koeselitz 14y ago
        "Tend to fail silently" is different from "could fail silently at some point, though it is unlikely." You are correct that RAID controllers shouldn't be the former; however, there is absolutely no way to prevent RAID controllers from being the latter.
        its_so_on 14y ago
        Could you tell me your reasoning about the last sentence? If you were talking about anything else, like a normal hard-drive, fine, of course it can fail silently. But the whole thing that a RAID drive is, is another layer on top of hard-drives, to make them redundant and chirp loudly when one dies or starts having wrong data and has to be removed, so that you can replace it and rebuild the RAID. I mean, all the RAID controller does is write data that is always redundant (even when it thinks all drives are working fine). How is it not possible for it to check for this consistency as well? Especially in Raid-6 etc configurations, which are even more consistent? Of course, on a probabilistic level random bit rot means "nothing is certain", but on a practical level, how can you not expect a raid controller not to fail silently, when all it does is corral redundant data around, create checksums, verify what's written, etc. It's the whole reason it exists. To me this is like saying that a checksumming algorithm should be expected to sometimes fail and just return a checksum chosen randomly from the space of all possible checksums, instead of the checksum actually produced by the algorithm for that data. That's ridiculous. I shouldn't have to even think about putting another layer on top of the checksum, so that I can checksum it. The very idea of having to do that means you don't have a checksumming algorithm. This thing should be right up there with bitrot causing bash to execute an rm -rf whenever you drop down to root. Sure that's possible, but that's not even in the scope of anything you have to think about. To me, a RAID is a layer on top of hard-drives that makes them redundant. Any controller that has a realistic chance of failing silently simply does not fit that definition.

wayne_h 14y ago

I once did a RAID data recovery on a system with a high-end Intel raid controller. The controller failed and they sent a new one - only the new one couldn't assemble the raid properly. It turns out that there was a flaw in the logic for where parity was stored. Normally parity is spread evenly across all the drives - not on this version. I had to reverse-engineer the crazy raid pattern and write a program to deraid it. It had gone undetected - as long as it was running.

wayne_h 14y ago

and don't underestimate the manufacturers ability to screw it up. Like when people 'upgrade' the firmware in their Buffalo NAS box and after that they can't see the data anymore. Luckily the data was still there and undamaged but it took data recovery to get it back.