3 ms·

> a bit flipping in an HDD from electromagnetic radiation or an SSD from electron leakage will not be caught by a journal Isn't that what the scrubbing process

> a bit flipping in an HDD from electromagnetic radiation or an SSD from electron leakage will not be caught by a journal Isn't that what the scrubbing process is for? My understanding was that RAID is keeping parity bits to make sure bitrot hasn't happened and it periodically scrubs and repairs when bits do rot. Is the difference that ZFS scrubs in real-time? Sorry if I'm being dense here, but I did a fair amount of reading about this, and I'm trying to understand what I'm missing.

cthalupa 7y ago
>Isn't that what the scrubbing process is for? My understanding was that RAID is keeping parity bits to make sure bitrot hasn't happened and it periodically scrubs and repairs when bits do rot. Nope. Parity bits aren't used to detect bitrot. If you get a URE when you go to read, the parity bits will be used to get you data. If you read successfully, nothing happens at all. If you scrub, and the data is found to be inconsistent, md assumes that the parity is wrong and overwrites it. https://linux.die.net/man/4/md https://linux.die.net/man/4/md >If a read error is detected during this process, the normal read-error handling causes correct data to be found from other devices and to be written back to the faulty device. In many case this will effectively fix the bad block. >If all blocks read successfully but are found to not be consistent, then this is regarded as a mismatch. >If check was used, then no action is taken to handle the mismatch, it is simply recorded. If repair was used, then a mismatch will be repaired in the same way that resync repairs arrays. For RAID5/RAID6 new parity blocks are written. For RAID1/RAID10, all but one block are overwritten with the content of that one block.
- anon9001 7y ago
  Thanks for the explanation. I didn't realize md didn't use the parity checks unless there's a URE. Is it correct to say that ZFS parity is used to reconcile on each read even without UREs, but with md parity is only used when there are UREs? If that's true, it makes much more sense why ZFS is considered the right choice. Also, do you know if adding luks in the mix changes that? My data goes md->lvm->luks->ext4 and I hadn't considered which part of that is supposed to catch the error if it wasn't a URE. It makes sense that md would update parity if there was no URE. It also makes sense that LVM wouldn't know or care about data integrity. I would think luks would require data integrity to work properly, but I'm not sure what happens if there is bitrot in a luks block. I'm also surprised ext4 doen't have checksumming, but apparently it's only for able to checksum for metadata and it's disabled by default: https://serverfault.com/questions/456319/how-to-get-checksum-data-integrity-on-ext4 https://serverfault.com/questions/456319/how-to-get-checksum... Dm-integrity sounds worth a look, but https://raid.wiki.kernel.org/index.php/Dm-integrity https://raid.wiki.kernel.org/index.php/Dm-integrity doesn't inspire much confidence.
  - Dylan16807 7y ago
    I don't think ZFS checks the parity constantly, but it verifies the checksums of every block it does read.
  - nickik 7y ago
    ZFS has native encryption now that is solid. So you can even avoid luks if you can live with some higher meta data leakage. There was a great talk about the details you can find on yt.
  - simcop2387 7y ago
    luks won't change that at all, it'll still be at the md level overall. The main encryption modes used by luks will lose a block to bitrot and such but won't lose the rest of the drive. They essentially generate a new key (really a new IV/nonce I think?) for every block through various means. This is the only way to allow for random access to any part of the disk without having to decrypt the previous blocks, and rewriting the rest of the disk whenever you change a part in the middle. I think you can setup an authenticated luks mode, but I've not looked at it and that should catch errors but that's all it would do, catch them.
  - cthalupa 7y ago
    The other replies answered most of your questions so I won't retread those. >Dm-integrity sounds worth a look, but https://raid.wiki.kernel.org/index.php/Dm-integrity https://raid.wiki.kernel.org/index.php/Dm-integrity doesn't inspire much confidence. That wiki page is pretty yikes - I was under the impression dm-integrity was farther along. Some misc other stuff I like about ZFS: l2arc/zil on ssd/optane devices for fast read/write caching Easy NFS/SMB sharing zdb ( https://manpages.debian.org/testing/zfsutils-linux/zdb.8.en.html https://manpages.debian.org/testing/zfsutils-linux/zdb.8.en.... ) I don't think you would be disappointed if you checked it out.

cthalupa 7y ago

>Isn't that what the scrubbing process is for? My understanding was that RAID is keeping parity bits to make sure bitrot hasn't happened and it periodically scrubs and repairs when bits do rot. Nope. Parity bits aren't used to detect bitrot. If you get a URE when you go to read, the parity bits will be used to get you data. If you read successfully, nothing happens at all. If you scrub, and the data is found to be inconsistent, md assumes that the parity is wrong and overwrites it. https://linux.die.net/man/4/md https://linux.die.net/man/4/md >If a read error is detected during this process, the normal read-error handling causes correct data to be found from other devices and to be written back to the faulty device. In many case this will effectively fix the bad block. >If all blocks read successfully but are found to not be consistent, then this is regarded as a mismatch. >If check was used, then no action is taken to handle the mismatch, it is simply recorded. If repair was used, then a mismatch will be repaired in the same way that resync repairs arrays. For RAID5/RAID6 new parity blocks are written. For RAID1/RAID10, all but one block are overwritten with the content of that one block.

anon9001 7y ago
Thanks for the explanation. I didn't realize md didn't use the parity checks unless there's a URE. Is it correct to say that ZFS parity is used to reconcile on each read even without UREs, but with md parity is only used when there are UREs? If that's true, it makes much more sense why ZFS is considered the right choice. Also, do you know if adding luks in the mix changes that? My data goes md->lvm->luks->ext4 and I hadn't considered which part of that is supposed to catch the error if it wasn't a URE. It makes sense that md would update parity if there was no URE. It also makes sense that LVM wouldn't know or care about data integrity. I would think luks would require data integrity to work properly, but I'm not sure what happens if there is bitrot in a luks block. I'm also surprised ext4 doen't have checksumming, but apparently it's only for able to checksum for metadata and it's disabled by default: https://serverfault.com/questions/456319/how-to-get-checksum-data-integrity-on-ext4 https://serverfault.com/questions/456319/how-to-get-checksum... Dm-integrity sounds worth a look, but https://raid.wiki.kernel.org/index.php/Dm-integrity https://raid.wiki.kernel.org/index.php/Dm-integrity doesn't inspire much confidence.
- Dylan16807 7y ago
  I don't think ZFS checks the parity constantly, but it verifies the checksums of every block it does read.
- nickik 7y ago
  ZFS has native encryption now that is solid. So you can even avoid luks if you can live with some higher meta data leakage. There was a great talk about the details you can find on yt.
- simcop2387 7y ago
  luks won't change that at all, it'll still be at the md level overall. The main encryption modes used by luks will lose a block to bitrot and such but won't lose the rest of the drive. They essentially generate a new key (really a new IV/nonce I think?) for every block through various means. This is the only way to allow for random access to any part of the disk without having to decrypt the previous blocks, and rewriting the rest of the disk whenever you change a part in the middle. I think you can setup an authenticated luks mode, but I've not looked at it and that should catch errors but that's all it would do, catch them.
- cthalupa 7y ago
  The other replies answered most of your questions so I won't retread those. >Dm-integrity sounds worth a look, but https://raid.wiki.kernel.org/index.php/Dm-integrity https://raid.wiki.kernel.org/index.php/Dm-integrity doesn't inspire much confidence. That wiki page is pretty yikes - I was under the impression dm-integrity was farther along. Some misc other stuff I like about ZFS: l2arc/zil on ssd/optane devices for fast read/write caching Easy NFS/SMB sharing zdb ( https://manpages.debian.org/testing/zfsutils-linux/zdb.8.en.html https://manpages.debian.org/testing/zfsutils-linux/zdb.8.en.... ) I don't think you would be disappointed if you checked it out.