5 ms·

I wanted to do something like this, but after reading about the failure rates of drives and raid, once one disk fails, it seems like you're fucked. Anybody got

I wanted to do something like this, but after reading about the failure rates of drives and raid, once one disk fails, it seems like you're fucked. Anybody got some stories or numbers that prove that while repairing my raid for one disk failure I won't just destroy the rest of my drives?

zwerdlds 10y ago
I've got a FreeNAS box which has endured two disk failures no problem. I have scheduled weekly rebuilds which I'm hoping finds bad blocks ahead of a failure-base rebuild.
ultramancool 10y ago
> I wanted to do something like this, but after reading about the failure rates of drives and raid, once one disk fails, it seems like you're fucked. Anybody got some stories or numbers that prove that while repairing my raid for one disk failure I won't just destroy the rest of my drives? Yeah, I've recovered several failed drives without issue, even on LVM systems where some partitions failed because they were striped but others worked because they were mirrored. No real issues overall. The only case where you might run into problems is when you get conflicting data between 2 drives, but I've never seen that happen in the real world. Some people will scare you into thinking that happens often but it hasn't been my experience. Often one drive just goes kaput completely or has an entire section which is completely unreadable or extremely slow, for which recovery is just a matter of marking that drive offline, swapping a new drive in and telling it to rebuild. The only case in which you lose everything is if you use RAID0. If you can do RAID10 that's the way to go IMO. RAID5 can be a little riskier because if one drive is damaged and one fails then you can wind up losing the bunch, but I use it on my personal NAS because it's mostly just TV and the important stuff is syncthing'd to my other machines. Regular scrubbing prevents these issues usually.
- pixl97 10y ago
  RAID6 is pretty safe. If you're using ZFS that's a RAIDZ2, you can lose 2 drives in a volume. RAID5 can be pretty risky with large drives that are the same type. >conflicting data between 2 drives, but I've never seen that happen in the real world Heh, run enough servers and you'll see everything eventually. Had a really fun one where a hardware raid card was telling us that writes were successful, but when data was coming back corrupt. It was writing bad data to two of the drives in the array.
pixl97 10y ago
Most raid failures are the fault of the user not understanding how raid works and leaving their raid improperly managed. First, buy good drives. Buying desktop drives that suck are a great way to end up with a corrupt SAN. In general you want drives that support TLER so if something goes wrong, you hear about it right away, rather than the drive trying to fix the problem silently while causing the disk to have long delays. You don't have to go super expensive enterprise, but buying from the bottom of the barrel, or is a 'green' or efficient drive is a great way to lose data. Second, no matter if you are using hardware or software raid, set up the monitoring utilities properly. You need an immediate alert if a drive is going bad or has failed. SMART should be enabled and when it alerts, replace the drive. The software should also do a verify, patrol read, or scrub (different terms for the same thing) that occasionally check the surface media of the disk and the validity of the data. If anything comes back with an error, you replace the disk right then. Lastly, one of the big issues why people lose raids all at once is they go out and buy a huge stack of the same kind of disk, made on the same day, with the same firmware, and possibly the same inattentive QA person. Sometimes when problems crop up, it can be a systemic problem with that model, and when all you have is one model, bye bye everything.
digler999 10y ago
I don't know if this has been mentioned yet in the conversation, but it's very important to know that raid is _not_ a backup. You still need some kind of backup solution even with a RAID system. if you want to learn more about this topic, come on over to www.reddit.com/r/datahoarder and read and ask about it. We're friendly.

zwerdlds 10y ago

I've got a FreeNAS box which has endured two disk failures no problem. I have scheduled weekly rebuilds which I'm hoping finds bad blocks ahead of a failure-base rebuild.

ultramancool 10y ago

> I wanted to do something like this, but after reading about the failure rates of drives and raid, once one disk fails, it seems like you're fucked. Anybody got some stories or numbers that prove that while repairing my raid for one disk failure I won't just destroy the rest of my drives? Yeah, I've recovered several failed drives without issue, even on LVM systems where some partitions failed because they were striped but others worked because they were mirrored. No real issues overall. The only case where you might run into problems is when you get conflicting data between 2 drives, but I've never seen that happen in the real world. Some people will scare you into thinking that happens often but it hasn't been my experience. Often one drive just goes kaput completely or has an entire section which is completely unreadable or extremely slow, for which recovery is just a matter of marking that drive offline, swapping a new drive in and telling it to rebuild. The only case in which you lose everything is if you use RAID0. If you can do RAID10 that's the way to go IMO. RAID5 can be a little riskier because if one drive is damaged and one fails then you can wind up losing the bunch, but I use it on my personal NAS because it's mostly just TV and the important stuff is syncthing'd to my other machines. Regular scrubbing prevents these issues usually.

pixl97 10y ago
RAID6 is pretty safe. If you're using ZFS that's a RAIDZ2, you can lose 2 drives in a volume. RAID5 can be pretty risky with large drives that are the same type. >conflicting data between 2 drives, but I've never seen that happen in the real world Heh, run enough servers and you'll see everything eventually. Had a really fun one where a hardware raid card was telling us that writes were successful, but when data was coming back corrupt. It was writing bad data to two of the drives in the array.

pixl97 10y ago

Most raid failures are the fault of the user not understanding how raid works and leaving their raid improperly managed. First, buy good drives. Buying desktop drives that suck are a great way to end up with a corrupt SAN. In general you want drives that support TLER so if something goes wrong, you hear about it right away, rather than the drive trying to fix the problem silently while causing the disk to have long delays. You don't have to go super expensive enterprise, but buying from the bottom of the barrel, or is a 'green' or efficient drive is a great way to lose data. Second, no matter if you are using hardware or software raid, set up the monitoring utilities properly. You need an immediate alert if a drive is going bad or has failed. SMART should be enabled and when it alerts, replace the drive. The software should also do a verify, patrol read, or scrub (different terms for the same thing) that occasionally check the surface media of the disk and the validity of the data. If anything comes back with an error, you replace the disk right then. Lastly, one of the big issues why people lose raids all at once is they go out and buy a huge stack of the same kind of disk, made on the same day, with the same firmware, and possibly the same inattentive QA person. Sometimes when problems crop up, it can be a systemic problem with that model, and when all you have is one model, bye bye everything.

digler999 10y ago

I don't know if this has been mentioned yet in the conversation, but it's very important to know that raid is _not_ a backup. You still need some kind of backup solution even with a RAID system. if you want to learn more about this topic, come on over to www.reddit.com/r/datahoarder and read and ask about it. We're friendly.