3 ms·

It's a very bad bug, so it is important to note there's an apparently-effective mitigation[1], which doesn't make it impossible to hit, but reduces the risk of

by veidr 3y ago

It's a very bad bug, so it is important to note there's an apparently-effective mitigation[1], which doesn't make it impossible to hit, but reduces the risk of it in the real world by a lot. # as root, or equivalent: echo 0 > /sys/module/zfs/parameters/zfs_dmu_offset_next_sync Before making this change, I was able to easily reproduce this bug on all of my ZFS filesystems (using the reproducer.sh[2] script from the bug thread). After making this change, I could no longer reproduce the bug at all. It seems that the bug is relatively rare (although still very bad if it happens) in most real-world scenarios; one user doing a heuristic scan to look for corrupted files found 0.00027% of their files (7 out of ~2,500,000) were likely affected[3]. The mitigation above (disabling zfs_dmu_offset_next_sync) seems to reduce those odds of the bug happening significantly. Almost everybody reports they can no longer reproduce the bug after changing it. Note that changing the setting doesn't persist across reboots, so you have to automate that somehow (e.g. editing /etc/modprobe.d/zfs.conf or whatever the normal way to control the setting is on your system; the GitHub thread has info for how to do it on Mac and Windows). Why this is a spectacularly bad bug is that it not only corrupts data, but it may be impossible to know if your existing files have been hit by this bug, during its long existence. There is active discussion in the bug thread about what kind of heuristics can be used to find "this file was probably corrupted by the bug" but there's no way (so far, and probably ever) to tell for sure (short of having some known-good copy of the file elsewhere to compare it to). Which makes the above mitigation all the more important while we wait for the fix! [1]: the Reddit thread advocating for this is here: https://www.reddit.com/r/zfs/comments/1826lgs/psa_its_not_block_cloning_its_a_data_co https://www.reddit.com/r/zfs/comments/1826lgs/psa_its_not_bl... rruption/ [2]: https://gist.github.com/masonmark/03c7cb08e22968b1f2feb1a6ac3c9701 https://gist.github.com/masonmark/03c7cb08e22968b1f2feb1a6ac... [3]: https://github.com/openzfs/zfs/issues/15526#issuecomment-1826182928 https://github.com/openzfs/zfs/issues/15526#issuecomment-182...

mustache_kimono 3y ago
> It seems that the bug is relatively rare (although still very bad if it happens) in most real-world scenarios; one user doing a heuristic scan to look for corrupted files found 0.00027% of their files (7 out of ~2,500,000) were likely affected. I'm running the following script to detect corruption.[0] The two files I've found so far seem like false positives. [0]: https://gist.github.com/kimono-koans/2696a8c8eac0a6babf7b2d9b26a82f65 https://gist.github.com/kimono-koans/2696a8c8eac0a6babf7b2d9...
- veidr 3y ago
  Yeah, I am running a similar script I got from the GitHub bug thread; so far I have not found any suspected-corrupt files at all, except for in the files generated by the reproducer.sh script, e.g.: Possible corruption in /mnt/slow/reproducer_146495_720
qwertox 3y ago
It seems that this doesn't really fixe the bug [0] and also enables another one [1][2] [0] https://github.com/openzfs/zfs/issues/15526#issuecomment-1826125014 https://github.com/openzfs/zfs/issues/15526#issuecomment-182... [1] https://github.com/openzfs/zfs/issues/15526#issuecomment-1826378265 https://github.com/openzfs/zfs/issues/15526#issuecomment-182... [2] https://github.com/openzfs/zfs/issues/6958 https://github.com/openzfs/zfs/issues/6958
- veidr 3y ago
  (>_<) Oh man, I knew about [0] when I posted (which is why I said it just reduces the chance of hitting the bug (by a lot)). But after spending all Saturday JST on it, I went to bed before [1] was posted. Skimming through #6958 though, it seems like it's the lesser of evils, compared to #15526... I think? It's less obvious (to me) what the impact of #6958 is. Is it silent undetectable corruption of your precious data potentially over years, or more likely to cause a crash or runtime error? Reports like https://github.com/intel/bmap-tools/issues/65 https://github.com/intel/bmap-tools/issues/65 make it seem more like the latter. But I have to read more. But since the zfs_dmu_offset_next_sync setting was disabled by default until recently, I still suspect (but yeah, don't know for sure) that disabling is the safest thing we can currently do on unmodified ZFS systems.
veidr 3y ago
UPDATE: There is now a very good simple explantaion of the bug, and how it became much more likely to happen recently, even though it has existed for a very long time: https://github.com/openzfs/zfs/issues/15526#issuecomment-1826412289 https://github.com/openzfs/zfs/issues/15526#issuecomment-182...

by veidr 3y ago

mustache_kimono 3y ago

> It seems that the bug is relatively rare (although still very bad if it happens) in most real-world scenarios; one user doing a heuristic scan to look for corrupted files found 0.00027% of their files (7 out of ~2,500,000) were likely affected. I'm running the following script to detect corruption.[0] The two files I've found so far seem like false positives. [0]: https://gist.github.com/kimono-koans/2696a8c8eac0a6babf7b2d9b26a82f65 https://gist.github.com/kimono-koans/2696a8c8eac0a6babf7b2d9...

veidr 3y ago
Yeah, I am running a similar script I got from the GitHub bug thread; so far I have not found any suspected-corrupt files at all, except for in the files generated by the reproducer.sh script, e.g.: Possible corruption in /mnt/slow/reproducer_146495_720

qwertox 3y ago

It seems that this doesn't really fixe the bug [0] and also enables another one [1][2] [0] https://github.com/openzfs/zfs/issues/15526#issuecomment-1826125014 https://github.com/openzfs/zfs/issues/15526#issuecomment-182... [1] https://github.com/openzfs/zfs/issues/15526#issuecomment-1826378265 https://github.com/openzfs/zfs/issues/15526#issuecomment-182... [2] https://github.com/openzfs/zfs/issues/6958 https://github.com/openzfs/zfs/issues/6958

veidr 3y ago
(>_<) Oh man, I knew about [0] when I posted (which is why I said it just reduces the chance of hitting the bug (by a lot)). But after spending all Saturday JST on it, I went to bed before [1] was posted. Skimming through #6958 though, it seems like it's the lesser of evils, compared to #15526... I think? It's less obvious (to me) what the impact of #6958 is. Is it silent undetectable corruption of your precious data potentially over years, or more likely to cause a crash or runtime error? Reports like https://github.com/intel/bmap-tools/issues/65 https://github.com/intel/bmap-tools/issues/65 make it seem more like the latter. But I have to read more. But since the zfs_dmu_offset_next_sync setting was disabled by default until recently, I still suspect (but yeah, don't know for sure) that disabling is the safest thing we can currently do on unmodified ZFS systems.

veidr 3y ago

UPDATE: There is now a very good simple explantaion of the bug, and how it became much more likely to happen recently, even though it has existed for a very long time: https://github.com/openzfs/zfs/issues/15526#issuecomment-1826412289 https://github.com/openzfs/zfs/issues/15526#issuecomment-182...