3 ms·

More than once I've lost a RAID array during a rebuild. That's what backups are for, I suppose.

sliken 6y ago
1st lesson on using a RAID should be that RAID is not a replacement for backups.
klodolph 6y ago
This is to be expected. RAID 5 is not safe. It is common for a RAID 5 array to be running in a degraded state during “normal” operation without the operator’s knowledge. Then, when one drive fails, rebuilding the array is impossible; you have already lost data. This is common because there is often no good way to repair a RAID 5 system which is running in degraded state, and there is often no monitoring to respond to array degradation (if you can’t fix it, why monitor it?) In other words, RAID 5 does not protect very well against drive failures. RAID 6 is more durable but inefficient (there are systems which are both more durable and more efficient than RAID 6). RAID is generally optimized for implementation simplicity over other concerns.
- tutfbhuf 6y ago
  > there are systems which are both more durable and more efficient than RAID 6 RAID 10?
  - speakeron 6y ago
    RAID 10 is more efficient (if that equates to performant) than RAID 6, but not more durable. In fact it's less. As an example, a 4-disk array (which has the same capacity in both RAID 6 and RAID 10) has 4 2-disk failure modes. RAID 6 can handle all of these failure modes without losing data, whereas RAID 10 can only handle 2 of them.
    - klodolph 6y ago
      Efficiency is how much overhead you have. RAID 10 has 50% efficiency (50% of your storage is used for data), RAID 6 has N/N+2 efficiency. Since RAID 10 means using 4 drives, RAID 10 is never more efficient than RAID 6 for the same number of drives. The three main things we can optimize for are storage efficiency, I/O performance, and data durability. For most situations, RAID 6 is not Pareto-optimal. In other words, you can come up with solutions that have better storage efficiency, better I/O performance, or better data durability, without sacrificing anything. The only reason you’d use RAID 6 is because it’s easy to use. It’s measurably worse on any other axis. Combine this with various low-quality RAID implementations and RAID is even worse. Software RAID in the Linux kernel is fairly robust but there are many low-quality hardware RAID implementations.
      - jandrese 6y ago
        Efficiency is not just drive efficiency but controller time. The 4 disk RAID1_0 array may have the same drive space overhead as the RAID6 version, but it may be much slower due to the controller overhead. One of the often overlooked aspects of RAID is that the more complexity there is in the controller the higher the risk of losing your data due to hardware/firmware/software error. The worst part is these errors often don't appear until your array is already compromised in some way, so everything is fine until a single disk failure triggers a previously unknown bug that causes the rebuild procedure to clobber all of the data on all of the drives. I've known some professionals who use RAID1_0 whenever they can get away with it because they don't trust RAID controllers to handle anything more complex.
        Dylan16807 6y ago
        RAID 6 is pretty darn simple, I'm not sure what controller issues you'd have. That said, RAID 10 is great for many professional use cases. It's very fast, scores well on reliability, the cost premium isn't too bad, and you need backups anyway.
        klodolph 6y ago
        > Efficiency is not just drive efficiency but controller time. Sorry, I should be clear. When I say efficiency, I mean “storage efficiency” only. Controller time I count as a part of performance. Perhaps my comment makes more sense with that definition in mind.
- parliament32 6y ago
  I'm having trouble with some of what you're saying. Why is RAID 5 any different than, say, RAID 1? You monitor them both the same way (your raid controller tells you it's degraded), and you fix both the same way (slam in a new drive and rebuild). Why would monitoring/recovery be any different than any other RAID level on a controller?
  - dylan604 6y ago
    RAID 1 is just direct copies of the data onto 2 separate drives. No parity is involved. RAID 5 breaks the data into N-1 segments and then has parity data on the remaing disc. The drive the parity data gets saved to rotates through the drives in the array so that no single drive has all of the parity data. That's what RAID 3 does. So, if you have a RAID 5 array where 1 drive dies, the rebuild process is meant to be able to recreate the missing data on the new drive using the parity data from the rest of the drives in the array. However, if one of those drives has read errors, typically CRC type errors, then the rebuild cannot continue because the data it needs is not available. Bye bye data!! Also, RAID 5 can achieve higher throughput since the data is split between the number of drives in the array. We used to build RAID 51 volumes which is 2 separate RAID 5 arrays, but then add the 2 volumes as RAID 1 volume so they are mirrored. The system only sees 1 volume with the capacity of a single RAID 5 volume. That was before RAID 6.
  - redis_mlc 6y ago
    All things being equal, sure. But I've seen older controllers that were tested well with RAID5, and not tested with other RAID levels. Also, slamming in a "new" disk (but is it really new, or already stale?) might not work if the firmware version is different, or has a different default setting than the others, or what the controller expects. (For example, a different timeout or retry setting.) You may also want to develop internal processes like reducing traffic to the rebuilding volume to avoid many of the possible rebuild problems under load. I'm actually a fairly big fan of the cloud because of the above problems in managing storage. Most smaller companies are not up to the task.

1st lesson on using a RAID should be that RAID is not a replacement for backups.

This is to be expected. RAID 5 is not safe. It is common for a RAID 5 array to be running in a degraded state during “normal” operation without the operator’s knowledge. Then, when one drive fails, rebuilding the array is impossible; you have already lost data. This is common because there is often no good way to repair a RAID 5 system which is running in degraded state, and there is often no monitoring to respond to array degradation (if you can’t fix it, why monitor it?) In other words, RAID 5 does not protect very well against drive failures. RAID 6 is more durable but inefficient (there are systems which are both more durable and more efficient than RAID 6). RAID is generally optimized for implementation simplicity over other concerns.

tutfbhuf 6y ago
> there are systems which are both more durable and more efficient than RAID 6 RAID 10?
- speakeron 6y ago
  RAID 10 is more efficient (if that equates to performant) than RAID 6, but not more durable. In fact it's less. As an example, a 4-disk array (which has the same capacity in both RAID 6 and RAID 10) has 4 2-disk failure modes. RAID 6 can handle all of these failure modes without losing data, whereas RAID 10 can only handle 2 of them.
  - klodolph 6y ago
    Efficiency is how much overhead you have. RAID 10 has 50% efficiency (50% of your storage is used for data), RAID 6 has N/N+2 efficiency. Since RAID 10 means using 4 drives, RAID 10 is never more efficient than RAID 6 for the same number of drives. The three main things we can optimize for are storage efficiency, I/O performance, and data durability. For most situations, RAID 6 is not Pareto-optimal. In other words, you can come up with solutions that have better storage efficiency, better I/O performance, or better data durability, without sacrificing anything. The only reason you’d use RAID 6 is because it’s easy to use. It’s measurably worse on any other axis. Combine this with various low-quality RAID implementations and RAID is even worse. Software RAID in the Linux kernel is fairly robust but there are many low-quality hardware RAID implementations.
    - jandrese 6y ago
      Efficiency is not just drive efficiency but controller time. The 4 disk RAID1_0 array may have the same drive space overhead as the RAID6 version, but it may be much slower due to the controller overhead. One of the often overlooked aspects of RAID is that the more complexity there is in the controller the higher the risk of losing your data due to hardware/firmware/software error. The worst part is these errors often don't appear until your array is already compromised in some way, so everything is fine until a single disk failure triggers a previously unknown bug that causes the rebuild procedure to clobber all of the data on all of the drives. I've known some professionals who use RAID1_0 whenever they can get away with it because they don't trust RAID controllers to handle anything more complex.
      - Dylan16807 6y ago
        RAID 6 is pretty darn simple, I'm not sure what controller issues you'd have. That said, RAID 10 is great for many professional use cases. It's very fast, scores well on reliability, the cost premium isn't too bad, and you need backups anyway.
      - klodolph 6y ago
        > Efficiency is not just drive efficiency but controller time. Sorry, I should be clear. When I say efficiency, I mean “storage efficiency” only. Controller time I count as a part of performance. Perhaps my comment makes more sense with that definition in mind.
parliament32 6y ago
I'm having trouble with some of what you're saying. Why is RAID 5 any different than, say, RAID 1? You monitor them both the same way (your raid controller tells you it's degraded), and you fix both the same way (slam in a new drive and rebuild). Why would monitoring/recovery be any different than any other RAID level on a controller?
- dylan604 6y ago
  RAID 1 is just direct copies of the data onto 2 separate drives. No parity is involved. RAID 5 breaks the data into N-1 segments and then has parity data on the remaing disc. The drive the parity data gets saved to rotates through the drives in the array so that no single drive has all of the parity data. That's what RAID 3 does. So, if you have a RAID 5 array where 1 drive dies, the rebuild process is meant to be able to recreate the missing data on the new drive using the parity data from the rest of the drives in the array. However, if one of those drives has read errors, typically CRC type errors, then the rebuild cannot continue because the data it needs is not available. Bye bye data!! Also, RAID 5 can achieve higher throughput since the data is split between the number of drives in the array. We used to build RAID 51 volumes which is 2 separate RAID 5 arrays, but then add the 2 volumes as RAID 1 volume so they are mirrored. The system only sees 1 volume with the capacity of a single RAID 5 volume. That was before RAID 6.
- redis_mlc 6y ago
  All things being equal, sure. But I've seen older controllers that were tested well with RAID5, and not tested with other RAID levels. Also, slamming in a "new" disk (but is it really new, or already stale?) might not work if the firmware version is different, or has a different default setting than the others, or what the controller expects. (For example, a different timeout or retry setting.) You may also want to develop internal processes like reducing traffic to the rebuilding volume to avoid many of the possible rebuild problems under load. I'm actually a fairly big fan of the cloud because of the above problems in managing storage. Most smaller companies are not up to the task.