3 ms·

> find potentially corrupted files (with some risk of false positives) by searching for zero-byte blocks I believe it's not the best method as it will require

by csdvrx 3y ago

> find potentially corrupted files (with some risk of false positives) by searching for zero-byte blocks I believe it's not the best method as it will require carefully defining how many null-bytes blocks (and of which size) are a sign of corruption. Another reason is the frequency of corruption was shown to vary during tests, and to be based on the IO load : the observed zero-byte blocks may have been caused by the artificial scenario to intentionally reproduce the bug. Natural occurrences of this bug may have a different pattern. You should use ctime+checksums from old backups instead.

n8henrie 3y ago
Thanks for your input. Do you have any more detailed instructions for the ctime + checksum method? Would a zfs snapshot be sufficient?
- csdvrx 3y ago
  I will soon prepare some scripts to do that, but I first want to confirm the core ideas and design something that I will only have to do once, and the discussion on https://github.com/openzfs/zfs/issues/15526#issuecomment-1826075625 https://github.com/openzfs/zfs/issues/15526#issuecomment-182... is still ongoing. If you want more detail about my proposed approach, please check the discussion on https://old.reddit.com/r/zfs/comments/182x5wy/with_old_backups_how_to_check_for_silent/ https://old.reddit.com/r/zfs/comments/182x5wy/with_old_backu... I have 18 months of backups, and could go back earlier if needed but accessing and processing each backup will take time. I don't want to do that multiple times. Anything you can mount, using any filesystem keeping this metadata, should be usable as in input: this means from NTFS to zfs snapshots. It may even be possible to use ZIP files (which keep date and time) to feed a medata sqlite database. Comparing the metadata DB to the actual ZFS filesystem would give you a list of suspicious files, and the most recent backup you could use to restore them Alternatively, it could be possible to deduplicate all the backups when measuring the metadata, then to keep local copies of all the files, but it may add complexity and storage requirements. If you are in a rush, write a script like that, but given how the null-bytes detection approach now seems flawed, you may have to rewrite that as we learn more details. Also, there's no real fix for this bug yet that isn't introducing other problems. While we're still learning the ins and out of this >18 years old bug, I recommend to keep using a 2.1 version of zfs with zfs_dmu_offset_next_sync=0, and to bet on the low probability of corruption that allowed this bug to persist for such a long time. Of course, keep your cold backups (don't delete!) but they can't accumulate corruption if you don't access them.

by csdvrx 3y ago

n8henrie 3y ago

Thanks for your input. Do you have any more detailed instructions for the ctime + checksum method? Would a zfs snapshot be sufficient?

csdvrx 3y ago
I will soon prepare some scripts to do that, but I first want to confirm the core ideas and design something that I will only have to do once, and the discussion on https://github.com/openzfs/zfs/issues/15526#issuecomment-1826075625 https://github.com/openzfs/zfs/issues/15526#issuecomment-182... is still ongoing. If you want more detail about my proposed approach, please check the discussion on https://old.reddit.com/r/zfs/comments/182x5wy/with_old_backups_how_to_check_for_silent/ https://old.reddit.com/r/zfs/comments/182x5wy/with_old_backu... I have 18 months of backups, and could go back earlier if needed but accessing and processing each backup will take time. I don't want to do that multiple times. Anything you can mount, using any filesystem keeping this metadata, should be usable as in input: this means from NTFS to zfs snapshots. It may even be possible to use ZIP files (which keep date and time) to feed a medata sqlite database. Comparing the metadata DB to the actual ZFS filesystem would give you a list of suspicious files, and the most recent backup you could use to restore them Alternatively, it could be possible to deduplicate all the backups when measuring the metadata, then to keep local copies of all the files, but it may add complexity and storage requirements. If you are in a rush, write a script like that, but given how the null-bytes detection approach now seems flawed, you may have to rewrite that as we learn more details. Also, there's no real fix for this bug yet that isn't introducing other problems. While we're still learning the ins and out of this >18 years old bug, I recommend to keep using a 2.1 version of zfs with zfs_dmu_offset_next_sync=0, and to bet on the low probability of corruption that allowed this bug to persist for such a long time. Of course, keep your cold backups (don't delete!) but they can't accumulate corruption if you don't access them.