3 ms·

Your SATA RAID has a 56% chance of data loss

55 points by oxyona 18y ago

keyes 18y ago
Any decent hardware RAID controller has a background "patrol read" which takes care of this problem.
patrickg-zill 18y ago
ZFS however would protect against this, in that it checksums each chunk of data. And "zfs scrub poolname" will go through and do the checksums for you in the event you are storing a lot of data and not necessarily reading/writing it often.
- spydez 18y ago
  Well, maybe I should look into transition my home server from Ubuntu and XFS to Nexenta or OpenSolaris and ZFS... I'd hate to lose some of the data on that thing, though I'm currently not doing any RAID. Just a disk of important stuff rsnapshotted every 4 hours to another disk.
  - lallysingh 18y ago
    I've been running opensol/zfs for almost a year now. Zero problems, good perf & reliability. My research is on two mirrored sata/160gb drives. The most important stuff is on a FS on there with copies=2, making 2 copies on each drive. Also, backing up zfs is really, really easy. 48 lines of shell scripts do daily incrementals and weekly snapshot backups. But, do yourself a favor and put /usr/gnu/bin first in your path. oh, and alias tar=gtar. Old solarisisms will otherwise be irritating.
- attack 18y ago
  No, but it will notify you of a problem. However, I suspect that smartmontools would do an even better job of notifying you of a problem before it even occurs. If you want something that will actually protect against bit level errors well, generate some extra data using an appropriate Error Correcting Code for your important archives/files. Something like what par2repair generates. This is incredibly slow compared to simple erasure codes like RAID uses, though. There are some filesystems that do this for you automatically, I wrote one:) Actually, other than that I don't know of any filesystems that do this. If the above is impractical for performance reasons, Google recommends mirroring everything on three disks...
- iigs 18y ago
  As theoretically awesome as ZFS is, the math behind the human factors are more important than the math behind the computer failure rates, and this is something I don't believe we can fix with our current development methodologies. It looks like Joyent was able to rebuild most (all?) of their data in their ZFS data loss situation, but it could have played out drastically differently. http://discuss.joyent.com/viewtopic.php?id=19430 http://discuss.joyent.com/viewtopic.php?id=19430
lallysingh 18y ago
Note that it mentions that SATA has a higher bit-error rate than SAS/SCSI. Apparently through the use of fewer ECC bits. But yeah, ZFS rocks.
ars 18y ago
You could also set smartmon tools to do a short self test every day, and a long one (which reads the entire disk) every week. At least you'll know if there was an error.
attack 18y ago
This is what keeps encryption impractical, especially for laptops.
- lonestar 18y ago
  Why? Assuming you're using CBC mode within each sector, you wouldn't lose any additional data due to a sector error compared to an unencrypted disk.
  - cstejerean 18y ago
    http://en.wikipedia.org/wiki/Disk_encryption_theory http://en.wikipedia.org/wiki/Disk_encryption_theory
  - attack 18y ago
    These are bit level errors. So instead of losing a bit or two, you lose a massive chunk. That's the difference between a strange character in your Doc file and losing the whole thing. I think it's 128 bits for truecrypt, although internet searches point to many people using higher (1024 and 4096). And if the bit level error rates are as high as he says, looks bad.
    - wmf 18y ago
      If the hard disk detects the bit error, it will return an I/O error instead of the corrupt data. Likewise, ZFS will not return corrupt data. So there are plenty of cases where bit errors are promoted to block errors. I can understand the desire to minimize propagation of corruption, but I'd rather just use RAID.
ars 18y ago
Wow. If you have an 8TB raid, that's 7TB of data so 61,572,651,155,456 bits A hard disk has a bit error rate of 10^15: 1,000,000,000,000,000 That's about 16 times higher. That's horrible. Taken to a single 750GB disk that means you can read the entire disk an average of 155 times before you'll encounter an error! That's shockingly bad.
- Andys 18y ago
  I am, among other things, a sysadmin who babysits several RAIDs, including a 10TB one. I can confirm that the large capacity SATA disks have a noticeably high error rate, which I have never seen before. It used to be that RAID protected you from disk failures, but we aren't seeing that nearly as much now as we are unrecoverable bit errors on otherwise healthy disks. I have set all our RAID controllers to scrub the data and compare the mirrors every weekend. In the past couple of years we've only had a couple of unrecoverable read errors, but many recoverable read errors. Recoverable read errors are still bad, the disk gets taken offline and out of the RAID set for up to a minute while the controller retries many times until the data can be read. Then the sector is remapped to another part of the disk. So yes, it is a real problem, but it isn't too hard to manage in this era of filesystem snapshotting plus high speed RAID6 controllers. In return we've been given huge amounts of storage space.
JoelSutherland 18y ago
I keep waiting for a product that solves the failing hard drive problem at the consumer level. For most people I know, the most likely cause of data loss is hardware failure. Are any manufacturers offering a package that contains multiple hard drives but behaves as one? If performance isn't a factor, is there something I don't understand that makes this impossible?
- thorax 18y ago
  Windows Home Server actually uses a duplication scheme that makes sure all your machine backups (and shared folders) are stored, in full, on more than one drive on the server. It behaves pretty much like one big drive and duplicates across all the drives as needed. It's pretty likely a consumer won't need more than this since that equates to three copies of your data at any given time (laptop, server disk B, and server disk C).
- wmf 18y ago
  Drobo? Time Capsule?
  - JoelSutherland 18y ago
    Drobo is along the lines of what I am looking for. (I use a D-Link DNS 323) Instead though, I want it to have a SATA interface so the problem would be solved in hardware. Time Capsule fails if the disk fails (and backups aren't stored elsewhere).
    - wmf 18y ago
      I want it to have a SATA interface so the problem would be solved in hardware. I don't see much difference between SATA and Firewire; in both cases Drobo is hardware RAID. Time Capsule fails if the disk fails (and backups aren't stored elsewhere). But then you just make a new backup. To protect against disk failure, you can use backup or RAID. (RAIDed backup is a little expensive for personal use.) Backup also protects against file deletion or corruption.
- briansmith 18y ago
sabat 18y ago
I don't have any hard data, but my BS meter is going off.
deleted 18y ago
[deleted]
aparrie 18y ago
Anyone use ZFS on Linux? http://www.linuxworld.com/news/2007/061807-zfs-on-linux.html http://www.linuxworld.com/news/2007/061807-zfs-on-linux.html

55 points by oxyona 18y ago

keyes 18y ago

Any decent hardware RAID controller has a background "patrol read" which takes care of this problem.

ZFS however would protect against this, in that it checksums each chunk of data. And "zfs scrub poolname" will go through and do the checksums for you in the event you are storing a lot of data and not necessarily reading/writing it often.

spydez 18y ago
Well, maybe I should look into transition my home server from Ubuntu and XFS to Nexenta or OpenSolaris and ZFS... I'd hate to lose some of the data on that thing, though I'm currently not doing any RAID. Just a disk of important stuff rsnapshotted every 4 hours to another disk.
- lallysingh 18y ago
  I've been running opensol/zfs for almost a year now. Zero problems, good perf & reliability. My research is on two mirrored sata/160gb drives. The most important stuff is on a FS on there with copies=2, making 2 copies on each drive. Also, backing up zfs is really, really easy. 48 lines of shell scripts do daily incrementals and weekly snapshot backups. But, do yourself a favor and put /usr/gnu/bin first in your path. oh, and alias tar=gtar. Old solarisisms will otherwise be irritating.
attack 18y ago
No, but it will notify you of a problem. However, I suspect that smartmontools would do an even better job of notifying you of a problem before it even occurs. If you want something that will actually protect against bit level errors well, generate some extra data using an appropriate Error Correcting Code for your important archives/files. Something like what par2repair generates. This is incredibly slow compared to simple erasure codes like RAID uses, though. There are some filesystems that do this for you automatically, I wrote one:) Actually, other than that I don't know of any filesystems that do this. If the above is impractical for performance reasons, Google recommends mirroring everything on three disks...
iigs 18y ago
As theoretically awesome as ZFS is, the math behind the human factors are more important than the math behind the computer failure rates, and this is something I don't believe we can fix with our current development methodologies. It looks like Joyent was able to rebuild most (all?) of their data in their ZFS data loss situation, but it could have played out drastically differently. http://discuss.joyent.com/viewtopic.php?id=19430 http://discuss.joyent.com/viewtopic.php?id=19430

lallysingh 18y ago

Note that it mentions that SATA has a higher bit-error rate than SAS/SCSI. Apparently through the use of fewer ECC bits. But yeah, ZFS rocks.

ars 18y ago

You could also set smartmon tools to do a short self test every day, and a long one (which reads the entire disk) every week. At least you'll know if there was an error.

attack 18y ago

This is what keeps encryption impractical, especially for laptops.

lonestar 18y ago
Why? Assuming you're using CBC mode within each sector, you wouldn't lose any additional data due to a sector error compared to an unencrypted disk.
- cstejerean 18y ago
  http://en.wikipedia.org/wiki/Disk_encryption_theory http://en.wikipedia.org/wiki/Disk_encryption_theory
- attack 18y ago
  These are bit level errors. So instead of losing a bit or two, you lose a massive chunk. That's the difference between a strange character in your Doc file and losing the whole thing. I think it's 128 bits for truecrypt, although internet searches point to many people using higher (1024 and 4096). And if the bit level error rates are as high as he says, looks bad.
  - wmf 18y ago
    If the hard disk detects the bit error, it will return an I/O error instead of the corrupt data. Likewise, ZFS will not return corrupt data. So there are plenty of cases where bit errors are promoted to block errors. I can understand the desire to minimize propagation of corruption, but I'd rather just use RAID.

ars 18y ago

Wow. If you have an 8TB raid, that's 7TB of data so 61,572,651,155,456 bits A hard disk has a bit error rate of 10^15: 1,000,000,000,000,000 That's about 16 times higher. That's horrible. Taken to a single 750GB disk that means you can read the entire disk an average of 155 times before you'll encounter an error! That's shockingly bad.

Andys 18y ago
I am, among other things, a sysadmin who babysits several RAIDs, including a 10TB one. I can confirm that the large capacity SATA disks have a noticeably high error rate, which I have never seen before. It used to be that RAID protected you from disk failures, but we aren't seeing that nearly as much now as we are unrecoverable bit errors on otherwise healthy disks. I have set all our RAID controllers to scrub the data and compare the mirrors every weekend. In the past couple of years we've only had a couple of unrecoverable read errors, but many recoverable read errors. Recoverable read errors are still bad, the disk gets taken offline and out of the RAID set for up to a minute while the controller retries many times until the data can be read. Then the sector is remapped to another part of the disk. So yes, it is a real problem, but it isn't too hard to manage in this era of filesystem snapshotting plus high speed RAID6 controllers. In return we've been given huge amounts of storage space.

JoelSutherland 18y ago

I keep waiting for a product that solves the failing hard drive problem at the consumer level. For most people I know, the most likely cause of data loss is hardware failure. Are any manufacturers offering a package that contains multiple hard drives but behaves as one? If performance isn't a factor, is there something I don't understand that makes this impossible?

thorax 18y ago
Windows Home Server actually uses a duplication scheme that makes sure all your machine backups (and shared folders) are stored, in full, on more than one drive on the server. It behaves pretty much like one big drive and duplicates across all the drives as needed. It's pretty likely a consumer won't need more than this since that equates to three copies of your data at any given time (laptop, server disk B, and server disk C).
wmf 18y ago
Drobo? Time Capsule?
- JoelSutherland 18y ago
  Drobo is along the lines of what I am looking for. (I use a D-Link DNS 323) Instead though, I want it to have a SATA interface so the problem would be solved in hardware. Time Capsule fails if the disk fails (and backups aren't stored elsewhere).
  - wmf 18y ago
    I want it to have a SATA interface so the problem would be solved in hardware. I don't see much difference between SATA and Firewire; in both cases Drobo is hardware RAID. Time Capsule fails if the disk fails (and backups aren't stored elsewhere). But then you just make a new backup. To protect against disk failure, you can use backup or RAID. (RAIDed backup is a little expensive for personal use.) Backup also protects against file deletion or corruption.
briansmith 18y ago

sabat 18y ago

I don't have any hard data, but my BS meter is going off.

deleted 18y ago

[deleted]

aparrie 18y ago

Anyone use ZFS on Linux? http://www.linuxworld.com/news/2007/061807-zfs-on-linux.html http://www.linuxworld.com/news/2007/061807-zfs-on-linux.html