6 ms·

ZFS silent corruption bug found: replaces chunks inside copied files by zeroes

101 points by csdvrx 3y ago

csdvrx 3y ago
The root cause of the bug may have been present for a long time. It increased in probability a bit after 2.1.4 (when zfs_dmu_offset_next_sync=1 became the default), and even more after 2.2.0 (with block cloning) Quick workaround: echo 0 > /sys/module/zfs/parameters/zfs_dmu_offset_next_sync
remram 3y ago
dupe: https://news.ycombinator.com/item?id=38380240 https://news.ycombinator.com/item?id=38380240 (2 days ago)
- csdvrx 3y ago
  Thanks, I had seen the title with the cloning issue, and I even commented, but I thought it didn't apply to me as new features are never deployed until at least a few months have passed and other people have confirmed they work well. I was only worried about the zfs send | zfs receive bug corrupting both pools. I had been too lazy to check the details, so I had missed that the bug could be triggered even when you DON'T use block cloning at all but just copy files, and even on old versions like 2.1.4: it's only the probability of the bug that increases. I only caught this issue through reddit. Now thanks to your link after going through all the github comments I'm rereading all the comments from the HN thread from 2 days ago to decide how to analyze several months worth of backups, some of them not on ZFS, but all of them sourced from ZFS and therefore now suspicious of silent corruption. Hopefully, this warning will stay long enough in the toplist for affected users to deploy the simple countermeasures or at least preserve their backups until we know how to identify the affected files reliably enough (though number of contiguous zeroes, repeating patterns inside the file etc)
  - k8svet 3y ago
    Wait, is scrubbing it and checking for errors not sufficient?
    - PaulCarrack 3y ago
      It's "silent corruption" so a scrub would not detect it. This is the worst possible scenario which is why this is getting a lot of publicity in contrast to bugs in the past. What's worse is that the bug has been in production for 18 months (since 2.1.4) so if you've ever put a file on a ZFS filesystem in the last year or so, you may be affected. The only true way of knowing whether you are are victim of this is to look at every single file and compare it against the known good that has never been on a ZFS filesystem.
      - gavinhoward 3y ago
        Oh...this is bad. I have a filesystem that is not ZFS, but it is a backup of a ZFS filesystem. Am I screwed?
        csdvrx 3y ago
        > Oh...this is bad. Yes, it's extremely bad, and the title of the original submission from 2 days ago may not have caught your attention. It should have been "If your version of ZFS is less than 18 months old, you may have silent data corruption". I editorialized the git title as little as I could while still describing the essential problem. > I have a filesystem that is not ZFS, but it is a backup of a ZFS filesystem. Am I screwed? If it's a backcup of a ZFS filesystem created with a version of OpenZFS more recent than 18 months, maybe. That's because even if it's a 100% perfect backup, you can't know if the backup contains files that where silently corrupted when they were on the original ZFS filesystem (unless you have copies of the files before they were on the ZFS) The bug is deemed "unlikely" from 2.1.4 to the version 2.2 which introduced block cloning and increated the probability of the bug showing up, but you won't know if 0% or 0.1% (or any other proportion) of your files are affected until after you compare checksums. I'd suggest to wait until more is known. If this bug flew under the radar for so long, it should be rare.
steponlego 3y ago
Note this is an issue with OpenZFS, not ZFS.
- kadoban 3y ago
  What other implementation of ZFS is commonly used?
  - steponlego 3y ago
    Sun's ZFS, now owned and maintained by Oracle. ZFS is actually a Sun project.
    - 15457345234 3y ago
      Remain In Light
- justaj 3y ago
  Is there any indication this bug was not present in (Oracle) ZFS?
  - mustache_kimono 3y ago
    Yeah, right now, it's actually not known whether roots of this bug dates back to even the Sun days. And because Oracle ZFS is not open source we can't know if this or other bugs are lurking. It seems to require certain extremely specific situations to trigger it. Like using copy_file_range, which is the new feature which exposed it, or writing a file and then writing a larger than recordsize hole in that file in the same transaction group.
    - chlorion 3y ago
      It's worth noting that copy_file_range is used by a lot of things. Most programming languages "copy_file" functions use copy_file_range, everything from Rust to Emacs Lisp! The only language I can think of that doesn't use copy_file_range when copying files is Python. On Gentoo, the portage package manager is written in Python but has some "native extensions", one of these extensions is copying files with copy_file_range, which is used when merging images to the root filesystem from my understanding. Also GNU coreutils "cp" command uses it by default in recent releases, I'm not sure which release specifically introduced this change. There are other things required to trigger the bug that are a lot less common though.
ptx 3y ago
The recent FreeBSD 14 release apparently failed to build for one of the platforms because a file "somehow ended up being full of NUL bytes" [0]. I wonder if that's due to this bug? (Could be just a coincidence, of course.) OpenZFS 2.1.4 was included in FreeBSD 13.1, according to the release notes. [0] https://www.daemonology.net/blog/2023-11-21-late-breaking-FreeBSD-14-breakage.html https://www.daemonology.net/blog/2023-11-21-late-breaking-Fr...
- inferiorhuman 3y ago
  looks at the list looks at the FreeBSD 14.0 errata on freebsd.org This is precisely why I wouldn't run FreeBSD in production. Look at this shit. - You need to freebsd-update fetch install before you upgrade - EC2 AMIs can't handle binary user-data - FreeBSD Update reports 14.0-RELEASE approaching its EoL Two ways to break booting and one really stupid mistake and NONE of it is listed on freebsd.org. Sigh.
  - wkat4242 3y ago
    freebsd-update fetch install is part of the normal upgrade process. So yeah of course that's necessary, it's how you upgrade. Not sure what #2 is about, I don't use cloud. And the EoL thing did not happen to me, I did hear of it happening though but it's not omnipresent.
    - inferiorhuman 3y ago
      freebsd-update fetch install is part of the normal upgrade process If the man page and literally every other piece of documentation about upgrading is correct: no, no it's not. If the intent of freebsd-update were only to allow migration from the latest patch level, it would fail instead of corrupting its state, no? The only caveat provided by the freebsd-update man page is the admonishment to read the release notes for special instructions, yet the release notes completely fail to mention this footgun. Or is your assertion that intuition should take the place of documentation?
      - 3y ago
AndrewDavis 3y ago
RobN has opened a PR consisting of the patch he asked people to test in the issue. https://github.com/openzfs/zfs/pull/15571 https://github.com/openzfs/zfs/pull/15571 Pretty crazy the bug might date back 10 years.
veidr 3y ago
It's a very bad bug, so it is important to note there's an apparently-effective mitigation[1], which doesn't make it impossible to hit, but reduces the risk of it in the real world by a lot. # as root, or equivalent: echo 0 > /sys/module/zfs/parameters/zfs_dmu_offset_next_sync Before making this change, I was able to easily reproduce this bug on all of my ZFS filesystems (using the reproducer.sh[2] script from the bug thread). After making this change, I could no longer reproduce the bug at all. It seems that the bug is relatively rare (although still very bad if it happens) in most real-world scenarios; one user doing a heuristic scan to look for corrupted files found 0.00027% of their files (7 out of ~2,500,000) were likely affected[3]. The mitigation above (disabling zfs_dmu_offset_next_sync) seems to reduce those odds of the bug happening significantly. Almost everybody reports they can no longer reproduce the bug after changing it. Note that changing the setting doesn't persist across reboots, so you have to automate that somehow (e.g. editing /etc/modprobe.d/zfs.conf or whatever the normal way to control the setting is on your system; the GitHub thread has info for how to do it on Mac and Windows). Why this is a spectacularly bad bug is that it not only corrupts data, but it may be impossible to know if your existing files have been hit by this bug, during its long existence. There is active discussion in the bug thread about what kind of heuristics can be used to find "this file was probably corrupted by the bug" but there's no way (so far, and probably ever) to tell for sure (short of having some known-good copy of the file elsewhere to compare it to). Which makes the above mitigation all the more important while we wait for the fix! [1]: the Reddit thread advocating for this is here: https://www.reddit.com/r/zfs/comments/1826lgs/psa_its_not_block_cloning_its_a_data_co https://www.reddit.com/r/zfs/comments/1826lgs/psa_its_not_bl... rruption/ [2]: https://gist.github.com/masonmark/03c7cb08e22968b1f2feb1a6ac3c9701 https://gist.github.com/masonmark/03c7cb08e22968b1f2feb1a6ac... [3]: https://github.com/openzfs/zfs/issues/15526#issuecomment-1826182928 https://github.com/openzfs/zfs/issues/15526#issuecomment-182...
- mustache_kimono 3y ago
  > It seems that the bug is relatively rare (although still very bad if it happens) in most real-world scenarios; one user doing a heuristic scan to look for corrupted files found 0.00027% of their files (7 out of ~2,500,000) were likely affected. I'm running the following script to detect corruption.[0] The two files I've found so far seem like false positives. [0]: https://gist.github.com/kimono-koans/2696a8c8eac0a6babf7b2d9b26a82f65 https://gist.github.com/kimono-koans/2696a8c8eac0a6babf7b2d9...
kevvok 3y ago
Interestingly, a CVE has been assigned for this issue: https://www.cve.org/CVERecord?id=CVE-2023-49298 https://www.cve.org/CVERecord?id=CVE-2023-49298
- mekster 3y ago
  It says because it could overwrite config files but not always a security problem.
n8henrie 3y ago
For those just wondering "am I affected?", my understanding is: zpool get all | grep bclone > If the result is 0 for both bcloneused and bclonesaved then it's safe to say that you don't have silent corruption. [0] People are using `reproducer.sh` (https://gist.github.com/tonyhutter/d69f305508ae3b7ff6e9263b22031a84#file-reproducer-sh https://gist.github.com/tonyhutter/d69f305508ae3b7ff6e9263b2...) to see if they can reproduce the bug intentionally: [1] https://github.com/0x0177b11f/zfs-issue-15526-check-file https://github.com/0x0177b11f/zfs-issue-15526-check-file tries to find potentially corrupted files (with some risk of false positives) by searching for zero-byte blocks: [2] [0]: https://github.com/openzfs/zfs/issues/15526#issuecomment-1810819382 https://github.com/openzfs/zfs/issues/15526#issuecomment-181... [1]: https://github.com/openzfs/zfs/issues/15526#issuecomment-1823575126 https://github.com/openzfs/zfs/issues/15526#issuecomment-182... [2]: https://github.com/openzfs/zfs/issues/15526#issuecomment-1826453702 https://github.com/openzfs/zfs/issues/15526#issuecomment-182...
- csdvrx 3y ago
  > find potentially corrupted files (with some risk of false positives) by searching for zero-byte blocks I believe it's not the best method as it will require carefully defining how many null-bytes blocks (and of which size) are a sign of corruption. Another reason is the frequency of corruption was shown to vary during tests, and to be based on the IO load : the observed zero-byte blocks may have been caused by the artificial scenario to intentionally reproduce the bug. Natural occurrences of this bug may have a different pattern. You should use ctime+checksums from old backups instead.
  - n8henrie 3y ago
    Thanks for your input. Do you have any more detailed instructions for the ctime + checksum method? Would a zfs snapshot be sufficient?
    - csdvrx 3y ago
      I will soon prepare some scripts to do that, but I first want to confirm the core ideas and design something that I will only have to do once, and the discussion on https://github.com/openzfs/zfs/issues/15526#issuecomment-1826075625 https://github.com/openzfs/zfs/issues/15526#issuecomment-182... is still ongoing. If you want more detail about my proposed approach, please check the discussion on https://old.reddit.com/r/zfs/comments/182x5wy/with_old_backups_how_to_check_for_silent/ https://old.reddit.com/r/zfs/comments/182x5wy/with_old_backu... I have 18 months of backups, and could go back earlier if needed but accessing and processing each backup will take time. I don't want to do that multiple times. Anything you can mount, using any filesystem keeping this metadata, should be usable as in input: this means from NTFS to zfs snapshots. It may even be possible to use ZIP files (which keep date and time) to feed a medata sqlite database. Comparing the metadata DB to the actual ZFS filesystem would give you a list of suspicious files, and the most recent backup you could use to restore them Alternatively, it could be possible to deduplicate all the backups when measuring the metadata, then to keep local copies of all the files, but it may add complexity and storage requirements. If you are in a rush, write a script like that, but given how the null-bytes detection approach now seems flawed, you may have to rewrite that as we learn more details. Also, there's no real fix for this bug yet that isn't introducing other problems. While we're still learning the ins and out of this >18 years old bug, I recommend to keep using a 2.1 version of zfs with zfs_dmu_offset_next_sync=0, and to bet on the low probability of corruption that allowed this bug to persist for such a long time. Of course, keep your cold backups (don't delete!) but they can't accumulate corruption if you don't access them.

101 points by csdvrx 3y ago

The root cause of the bug may have been present for a long time. It increased in probability a bit after 2.1.4 (when zfs_dmu_offset_next_sync=1 became the default), and even more after 2.2.0 (with block cloning) Quick workaround: echo 0 > /sys/module/zfs/parameters/zfs_dmu_offset_next_sync

remram 3y ago

dupe: https://news.ycombinator.com/item?id=38380240 https://news.ycombinator.com/item?id=38380240 (2 days ago)

csdvrx 3y ago
Thanks, I had seen the title with the cloning issue, and I even commented, but I thought it didn't apply to me as new features are never deployed until at least a few months have passed and other people have confirmed they work well. I was only worried about the zfs send | zfs receive bug corrupting both pools. I had been too lazy to check the details, so I had missed that the bug could be triggered even when you DON'T use block cloning at all but just copy files, and even on old versions like 2.1.4: it's only the probability of the bug that increases. I only caught this issue through reddit. Now thanks to your link after going through all the github comments I'm rereading all the comments from the HN thread from 2 days ago to decide how to analyze several months worth of backups, some of them not on ZFS, but all of them sourced from ZFS and therefore now suspicious of silent corruption. Hopefully, this warning will stay long enough in the toplist for affected users to deploy the simple countermeasures or at least preserve their backups until we know how to identify the affected files reliably enough (though number of contiguous zeroes, repeating patterns inside the file etc)
- k8svet 3y ago
  Wait, is scrubbing it and checking for errors not sufficient?
  - PaulCarrack 3y ago
    It's "silent corruption" so a scrub would not detect it. This is the worst possible scenario which is why this is getting a lot of publicity in contrast to bugs in the past. What's worse is that the bug has been in production for 18 months (since 2.1.4) so if you've ever put a file on a ZFS filesystem in the last year or so, you may be affected. The only true way of knowing whether you are are victim of this is to look at every single file and compare it against the known good that has never been on a ZFS filesystem.
    - gavinhoward 3y ago
      Oh...this is bad. I have a filesystem that is not ZFS, but it is a backup of a ZFS filesystem. Am I screwed?
      - csdvrx 3y ago
        > Oh...this is bad. Yes, it's extremely bad, and the title of the original submission from 2 days ago may not have caught your attention. It should have been "If your version of ZFS is less than 18 months old, you may have silent data corruption". I editorialized the git title as little as I could while still describing the essential problem. > I have a filesystem that is not ZFS, but it is a backup of a ZFS filesystem. Am I screwed? If it's a backcup of a ZFS filesystem created with a version of OpenZFS more recent than 18 months, maybe. That's because even if it's a 100% perfect backup, you can't know if the backup contains files that where silently corrupted when they were on the original ZFS filesystem (unless you have copies of the files before they were on the ZFS) The bug is deemed "unlikely" from 2.1.4 to the version 2.2 which introduced block cloning and increated the probability of the bug showing up, but you won't know if 0% or 0.1% (or any other proportion) of your files are affected until after you compare checksums. I'd suggest to wait until more is known. If this bug flew under the radar for so long, it should be rare.

steponlego 3y ago

Note this is an issue with OpenZFS, not ZFS.

kadoban 3y ago
What other implementation of ZFS is commonly used?
- steponlego 3y ago
  Sun's ZFS, now owned and maintained by Oracle. ZFS is actually a Sun project.
  - 15457345234 3y ago
    Remain In Light
justaj 3y ago
Is there any indication this bug was not present in (Oracle) ZFS?
- mustache_kimono 3y ago
  Yeah, right now, it's actually not known whether roots of this bug dates back to even the Sun days. And because Oracle ZFS is not open source we can't know if this or other bugs are lurking. It seems to require certain extremely specific situations to trigger it. Like using copy_file_range, which is the new feature which exposed it, or writing a file and then writing a larger than recordsize hole in that file in the same transaction group.
  - chlorion 3y ago
    It's worth noting that copy_file_range is used by a lot of things. Most programming languages "copy_file" functions use copy_file_range, everything from Rust to Emacs Lisp! The only language I can think of that doesn't use copy_file_range when copying files is Python. On Gentoo, the portage package manager is written in Python but has some "native extensions", one of these extensions is copying files with copy_file_range, which is used when merging images to the root filesystem from my understanding. Also GNU coreutils "cp" command uses it by default in recent releases, I'm not sure which release specifically introduced this change. There are other things required to trigger the bug that are a lot less common though.

ptx 3y ago

The recent FreeBSD 14 release apparently failed to build for one of the platforms because a file "somehow ended up being full of NUL bytes" [0]. I wonder if that's due to this bug? (Could be just a coincidence, of course.) OpenZFS 2.1.4 was included in FreeBSD 13.1, according to the release notes. [0] https://www.daemonology.net/blog/2023-11-21-late-breaking-FreeBSD-14-breakage.html https://www.daemonology.net/blog/2023-11-21-late-breaking-Fr...

inferiorhuman 3y ago
looks at the list looks at the FreeBSD 14.0 errata on freebsd.org This is precisely why I wouldn't run FreeBSD in production. Look at this shit. - You need to freebsd-update fetch install before you upgrade - EC2 AMIs can't handle binary user-data - FreeBSD Update reports 14.0-RELEASE approaching its EoL Two ways to break booting and one really stupid mistake and NONE of it is listed on freebsd.org. Sigh.
- wkat4242 3y ago
  freebsd-update fetch install is part of the normal upgrade process. So yeah of course that's necessary, it's how you upgrade. Not sure what #2 is about, I don't use cloud. And the EoL thing did not happen to me, I did hear of it happening though but it's not omnipresent.
  - inferiorhuman 3y ago
    freebsd-update fetch install is part of the normal upgrade process If the man page and literally every other piece of documentation about upgrading is correct: no, no it's not. If the intent of freebsd-update were only to allow migration from the latest patch level, it would fail instead of corrupting its state, no? The only caveat provided by the freebsd-update man page is the admonishment to read the release notes for special instructions, yet the release notes completely fail to mention this footgun. Or is your assertion that intuition should take the place of documentation?
    - 3y ago

AndrewDavis 3y ago

RobN has opened a PR consisting of the patch he asked people to test in the issue. https://github.com/openzfs/zfs/pull/15571 https://github.com/openzfs/zfs/pull/15571 Pretty crazy the bug might date back 10 years.

veidr 3y ago

It's a very bad bug, so it is important to note there's an apparently-effective mitigation[1], which doesn't make it impossible to hit, but reduces the risk of it in the real world by a lot. # as root, or equivalent: echo 0 > /sys/module/zfs/parameters/zfs_dmu_offset_next_sync Before making this change, I was able to easily reproduce this bug on all of my ZFS filesystems (using the reproducer.sh[2] script from the bug thread). After making this change, I could no longer reproduce the bug at all. It seems that the bug is relatively rare (although still very bad if it happens) in most real-world scenarios; one user doing a heuristic scan to look for corrupted files found 0.00027% of their files (7 out of ~2,500,000) were likely affected[3]. The mitigation above (disabling zfs_dmu_offset_next_sync) seems to reduce those odds of the bug happening significantly. Almost everybody reports they can no longer reproduce the bug after changing it. Note that changing the setting doesn't persist across reboots, so you have to automate that somehow (e.g. editing /etc/modprobe.d/zfs.conf or whatever the normal way to control the setting is on your system; the GitHub thread has info for how to do it on Mac and Windows). Why this is a spectacularly bad bug is that it not only corrupts data, but it may be impossible to know if your existing files have been hit by this bug, during its long existence. There is active discussion in the bug thread about what kind of heuristics can be used to find "this file was probably corrupted by the bug" but there's no way (so far, and probably ever) to tell for sure (short of having some known-good copy of the file elsewhere to compare it to). Which makes the above mitigation all the more important while we wait for the fix! [1]: the Reddit thread advocating for this is here: https://www.reddit.com/r/zfs/comments/1826lgs/psa_its_not_block_cloning_its_a_data_co https://www.reddit.com/r/zfs/comments/1826lgs/psa_its_not_bl... rruption/ [2]: https://gist.github.com/masonmark/03c7cb08e22968b1f2feb1a6ac3c9701 https://gist.github.com/masonmark/03c7cb08e22968b1f2feb1a6ac... [3]: https://github.com/openzfs/zfs/issues/15526#issuecomment-1826182928 https://github.com/openzfs/zfs/issues/15526#issuecomment-182...

mustache_kimono 3y ago
> It seems that the bug is relatively rare (although still very bad if it happens) in most real-world scenarios; one user doing a heuristic scan to look for corrupted files found 0.00027% of their files (7 out of ~2,500,000) were likely affected. I'm running the following script to detect corruption.[0] The two files I've found so far seem like false positives. [0]: https://gist.github.com/kimono-koans/2696a8c8eac0a6babf7b2d9b26a82f65 https://gist.github.com/kimono-koans/2696a8c8eac0a6babf7b2d9...

kevvok 3y ago

Interestingly, a CVE has been assigned for this issue: https://www.cve.org/CVERecord?id=CVE-2023-49298 https://www.cve.org/CVERecord?id=CVE-2023-49298

mekster 3y ago
It says because it could overwrite config files but not always a security problem.

n8henrie 3y ago

For those just wondering "am I affected?", my understanding is: zpool get all | grep bclone > If the result is 0 for both bcloneused and bclonesaved then it's safe to say that you don't have silent corruption. [0] People are using `reproducer.sh` (https://gist.github.com/tonyhutter/d69f305508ae3b7ff6e9263b22031a84#file-reproducer-sh https://gist.github.com/tonyhutter/d69f305508ae3b7ff6e9263b2...) to see if they can reproduce the bug intentionally: [1] https://github.com/0x0177b11f/zfs-issue-15526-check-file https://github.com/0x0177b11f/zfs-issue-15526-check-file tries to find potentially corrupted files (with some risk of false positives) by searching for zero-byte blocks: [2] [0]: https://github.com/openzfs/zfs/issues/15526#issuecomment-1810819382 https://github.com/openzfs/zfs/issues/15526#issuecomment-181... [1]: https://github.com/openzfs/zfs/issues/15526#issuecomment-1823575126 https://github.com/openzfs/zfs/issues/15526#issuecomment-182... [2]: https://github.com/openzfs/zfs/issues/15526#issuecomment-1826453702 https://github.com/openzfs/zfs/issues/15526#issuecomment-182...

csdvrx 3y ago
> find potentially corrupted files (with some risk of false positives) by searching for zero-byte blocks I believe it's not the best method as it will require carefully defining how many null-bytes blocks (and of which size) are a sign of corruption. Another reason is the frequency of corruption was shown to vary during tests, and to be based on the IO load : the observed zero-byte blocks may have been caused by the artificial scenario to intentionally reproduce the bug. Natural occurrences of this bug may have a different pattern. You should use ctime+checksums from old backups instead.
- n8henrie 3y ago
  Thanks for your input. Do you have any more detailed instructions for the ctime + checksum method? Would a zfs snapshot be sufficient?
  - csdvrx 3y ago
    I will soon prepare some scripts to do that, but I first want to confirm the core ideas and design something that I will only have to do once, and the discussion on https://github.com/openzfs/zfs/issues/15526#issuecomment-1826075625 https://github.com/openzfs/zfs/issues/15526#issuecomment-182... is still ongoing. If you want more detail about my proposed approach, please check the discussion on https://old.reddit.com/r/zfs/comments/182x5wy/with_old_backups_how_to_check_for_silent/ https://old.reddit.com/r/zfs/comments/182x5wy/with_old_backu... I have 18 months of backups, and could go back earlier if needed but accessing and processing each backup will take time. I don't want to do that multiple times. Anything you can mount, using any filesystem keeping this metadata, should be usable as in input: this means from NTFS to zfs snapshots. It may even be possible to use ZIP files (which keep date and time) to feed a medata sqlite database. Comparing the metadata DB to the actual ZFS filesystem would give you a list of suspicious files, and the most recent backup you could use to restore them Alternatively, it could be possible to deduplicate all the backups when measuring the metadata, then to keep local copies of all the files, but it may add complexity and storage requirements. If you are in a rush, write a script like that, but given how the null-bytes detection approach now seems flawed, you may have to rewrite that as we learn more details. Also, there's no real fix for this bug yet that isn't introducing other problems. While we're still learning the ins and out of this >18 years old bug, I recommend to keep using a 2.1 version of zfs with zfs_dmu_offset_next_sync=0, and to bet on the low probability of corruption that allowed this bug to persist for such a long time. Of course, keep your cold backups (don't delete!) but they can't accumulate corruption if you don't access them.