5 ms·

Other FS won't try to correct sane data on HDD but corrupted when loaded in memory.

by Thaxll 12y ago

ryao 12y ago
You are describing what undefined behavior might manifest in response to a bit flip. It is undefined for the reason that anything can go wrong, although I have trouble seeing how the particular scenario you describe here would cause a problem. If the checksum verification fails on a read because of a bit flip in the data, ZFS will just load another copy. That being said, it is not a good idea to use hardware that lacks ECC RAM. No filesystem developer will claim that their kernel driver can operate properly without ECC RAM because it is not possible. Email the linux-fsdevel mailing list if you want confirmation.
XorNot 12y ago
ZFS does not do this. Your assumptions about what other FSs do is inaccurate. Consider: you load data from ext4 into RAM, it gets corrupted. You change some bytes, then save it. Corrupted data is then written to the disk. Could be anything inside the allocation unit size, which is 4K generally - not insubstantial. ZFS's worst case behavior is exactly the same: it can't protect you from corruption in RAM after checksum validation if you then ask it to save that data to disk. There is no difference: if you do not have ECC RAM, you are vulnerable to bitflips corrupting your data - and they will have been. ZFS won't corrupt data which gets altered in memory due to a bitflip but is only ever read. It can't - because in-memory bitflips can't be detected. Even if at some point during checksum validation there was a mismatch, the restore is done from checksum protected parity/mirror image data. And if the restore block were corrupted, the checksum won't match and ZFS will rebuild the block correctly next time. If you do not have ECC RAM, every filesystem will potentially corrupt your data. ZFS is more resilient then pretty much all of them even in this case.
- Thaxll 12y ago
  Yes it does or I don't understand that: http://louwrentius.com/please-use-zfs-with-ecc-memory.html http://louwrentius.com/please-use-zfs-with-ecc-memory.html https://pthree.org/2013/12/10/zfs-administration-appendix-c-why-you-should-use-ecc-ram/ https://pthree.org/2013/12/10/zfs-administration-appendix-c-... Always fun to post on HN and getting downvoted ;o
  - XorNot 12y ago
    The first article doesn't contribute useful metrics - it simply infers that it would be bad. Here's the thing: have you run filesystem recovery tools lately? Because if you're at that point, your data is toast. It's toast if you were running mdadm or RAID. It's toast if a file gets deleted and recovered by ext3/4 (good luck figuring out which chunk you're looking at in lost+found). The second article is simply stressing my original point: ZFS cannot protect you from bad RAM. No file system can. If data is corrupted in RAM, ZFS will checksum that and write it to disk. This is exactly what will happen with any other filesystem. ZFS storage pools can recover from fairly dramatic failures. You can lose a disk out of a non-redundant stripe and still have complete metadata (just incomplete data, but checksums tell you exactly what you lost). Unimportable pools are not something that easily happens without severe damage - a corrupted uberblock can recovered by using the zpool import -T to rewind a txg and get back a verifiably complete copy of your data. This idea that somehow the failure mode of other filesystems to faulty RAM is better is destructive fiction. You have no idea if your data is valid on disk. You have no way to verify if the data is valid in the first place. And the degree and number of failures required to wipe out a pool would also wipe out any other storage system. The idea that recovery tools will save you in a catastrophic failure (of what kind?) is laughable. This can be trivially explored by trying to recover a deleted file in ext4. Sure, it can technically be done. Good luck with doing it more then once.
    - kurlberg 12y ago
      As far as I've understood, the gist of the "ZFS without ECC is scary"-mantra is that one bad memory location (say a flaw that results in a word always getting zeroed) might corrupt all your on disk data due to scrubbing ("read data, compute checksum, write data, repeat") Is this in fact a non-issue with ZFS?
      - ryao 12y ago
        The mechanism that you describe should be impossible to achieve with one bit-flip. I have no idea how you came to think this.
- j00lz 12y ago
  The worst case with ZFS and ram corruption is that you can lose your ENTIRE Zpool. As there are no zfs recovery tools available this means your data is as good as gone or a minimum 15k spend to get it re-assembled. This makes it far more risky to run ZFS with non server grade parts.
  - ryao 12y ago
    It is possible for this to happen with other filesystems too. The only thing is that when it happens with another filesystem, it is not news.
    - j00lz 12y ago
      It becomes the news as there are no recovery tools available for ZFS as there are with most other file systems. Meaning that the possibility of losing ALL data due to ram corruption becomes a real threat.

Other FS won't try to correct sane data on HDD but corrupted when loaded in memory.

by Thaxll 12y ago

Other FS won't try to correct sane data on HDD but corrupted when loaded in memory.

ryao 12y ago

You are describing what undefined behavior might manifest in response to a bit flip. It is undefined for the reason that anything can go wrong, although I have trouble seeing how the particular scenario you describe here would cause a problem. If the checksum verification fails on a read because of a bit flip in the data, ZFS will just load another copy. That being said, it is not a good idea to use hardware that lacks ECC RAM. No filesystem developer will claim that their kernel driver can operate properly without ECC RAM because it is not possible. Email the linux-fsdevel mailing list if you want confirmation.

XorNot 12y ago

ZFS does not do this. Your assumptions about what other FSs do is inaccurate. Consider: you load data from ext4 into RAM, it gets corrupted. You change some bytes, then save it. Corrupted data is then written to the disk. Could be anything inside the allocation unit size, which is 4K generally - not insubstantial. ZFS's worst case behavior is exactly the same: it can't protect you from corruption in RAM after checksum validation if you then ask it to save that data to disk. There is no difference: if you do not have ECC RAM, you are vulnerable to bitflips corrupting your data - and they will have been. ZFS won't corrupt data which gets altered in memory due to a bitflip but is only ever read. It can't - because in-memory bitflips can't be detected. Even if at some point during checksum validation there was a mismatch, the restore is done from checksum protected parity/mirror image data. And if the restore block were corrupted, the checksum won't match and ZFS will rebuild the block correctly next time. If you do not have ECC RAM, every filesystem will potentially corrupt your data. ZFS is more resilient then pretty much all of them even in this case.

Thaxll 12y ago
Yes it does or I don't understand that: http://louwrentius.com/please-use-zfs-with-ecc-memory.html http://louwrentius.com/please-use-zfs-with-ecc-memory.html https://pthree.org/2013/12/10/zfs-administration-appendix-c-why-you-should-use-ecc-ram/ https://pthree.org/2013/12/10/zfs-administration-appendix-c-... Always fun to post on HN and getting downvoted ;o
- XorNot 12y ago
  The first article doesn't contribute useful metrics - it simply infers that it would be bad. Here's the thing: have you run filesystem recovery tools lately? Because if you're at that point, your data is toast. It's toast if you were running mdadm or RAID. It's toast if a file gets deleted and recovered by ext3/4 (good luck figuring out which chunk you're looking at in lost+found). The second article is simply stressing my original point: ZFS cannot protect you from bad RAM. No file system can. If data is corrupted in RAM, ZFS will checksum that and write it to disk. This is exactly what will happen with any other filesystem. ZFS storage pools can recover from fairly dramatic failures. You can lose a disk out of a non-redundant stripe and still have complete metadata (just incomplete data, but checksums tell you exactly what you lost). Unimportable pools are not something that easily happens without severe damage - a corrupted uberblock can recovered by using the zpool import -T to rewind a txg and get back a verifiably complete copy of your data. This idea that somehow the failure mode of other filesystems to faulty RAM is better is destructive fiction. You have no idea if your data is valid on disk. You have no way to verify if the data is valid in the first place. And the degree and number of failures required to wipe out a pool would also wipe out any other storage system. The idea that recovery tools will save you in a catastrophic failure (of what kind?) is laughable. This can be trivially explored by trying to recover a deleted file in ext4. Sure, it can technically be done. Good luck with doing it more then once.
  - kurlberg 12y ago
    As far as I've understood, the gist of the "ZFS without ECC is scary"-mantra is that one bad memory location (say a flaw that results in a word always getting zeroed) might corrupt all your on disk data due to scrubbing ("read data, compute checksum, write data, repeat") Is this in fact a non-issue with ZFS?
    - ryao 12y ago
      The mechanism that you describe should be impossible to achieve with one bit-flip. I have no idea how you came to think this.
j00lz 12y ago
The worst case with ZFS and ram corruption is that you can lose your ENTIRE Zpool. As there are no zfs recovery tools available this means your data is as good as gone or a minimum 15k spend to get it re-assembled. This makes it far more risky to run ZFS with non server grade parts.
- ryao 12y ago
  It is possible for this to happen with other filesystems too. The only thing is that when it happens with another filesystem, it is not news.
  - j00lz 12y ago
    It becomes the news as there are no recovery tools available for ZFS as there are with most other file systems. Meaning that the possibility of losing ALL data due to ram corruption becomes a real threat.