7 ms·

Tell HN: ZFS silent data corruption bugfix – my research results

109 points by sandreas 3y ago

Hey HN, since I'm personally affected by this bug[1] and this scared me somehow, I thought I'd post my research results to have a discussion about the details. I must say I lost a bit of trust in ZFS, since it was 'sold' as rock solid unbreakable file system and now what... the sheer complexity of ZFS let's me think that this was not the only bug of this kind existing undetected in the source code. However the communication was clear and precise, other filesystems had similar problems and I just love the feature set of zfs. So I'll give it another shot hoping that future revisions will be safe(r). Here are my conclusions so far: 1. ZFS had a silent data corruption bug recently discussed on HN [6][7] 2. `Silent` means, you can't do anything about it without upgrading zfs (scrubs, checks, etc. don't help) 3. There is no tool to check if your data has been corrupted 4. Only once written and untouched data should be relatively safe (e.g. backups), the bug mainly affects reading data [4] 5. Setting `zfs_dmu_offset_next_sync=0` reduces probability of being affected, but no guarantee [4] 6. The bug affected any version before 2.2.2 and 2.1.14[2], but were more likely between 2.1.4 and 2.2.1 because a behaviour change in coreutils 7. The causing problem already existed much longer (since 2006?) - this is a question mark [4] 8. do a `zfs version` to see the version number 9. If you're scared (I was), please read this comment: https://news.ycombinator.com/item?id=38553342 So the only way to ENSURE your data is ok is checksumming all files on another filesystem at file level and compare or re-transfering all data from another filesystem after upgrading zfs to the latest version. Please let me know, if I missed something or got something wrong. Sources: [1]: https://www.phoronix.com/news/OpenZFS-Data-Corruption-Battle https://www.phoronix.com/news/OpenZFS-Data-Corruption-Battle [2]: https://www.phoronix.com/news/OpenZFS-2.2.2-Released https://www.phoronix.com/news/OpenZFS-2.2.2-Released [3]: https://www.theregister.com/2023/12/04/two_new_versions_of_openzfs/ https://www.theregister.com/2023/12/04/two_new_versions_of_o... [4]: https://www.reddit.com/r/zfs/comments/1826lgs/psa_its_not_block_cloning_its_a_data_corruption/ https://www.reddit.com/r/zfs/comments/1826lgs/psa_its_not_bl... [5]: https://news.ycombinator.com/item?id=38519382 https://news.ycombinator.com/item?id=38519382 [6]: https://news.ycombinator.com/item?id=38405731 https://news.ycombinator.com/item?id=38405731 [7]: https://news.ycombinator.com/item?id=38380240 https://news.ycombinator.com/item?id=38380240

KyleSanderson 3y ago
https://github.com/openzfs/zfs/pull/15529#pullrequestreview-1732508242 https://github.com/openzfs/zfs/pull/15529#pullrequestreview-... Honestly, ZFS is the best thing on the (Free)BSDs only... On Linux it doesn't even use the page cache, and you conflict severely with L2ARC. I know there's a variety of people who don't care, but still for real users it's not an actual option.
- antongribok 3y ago
  For me it was the previous data corruption bug [1] that killed any enthusiasm for ZoL. After that annoyances like the caching issues you mention and the constant kernel upgrades breaking DKMS on Fedora just stopped being worth it for me. I finally moved to btrfs earlier this year, and so far I'm glad I did. I run raid1 on my primary array, and raid5 on my off-site backup array at my mom's apartment connecting with Tailscale. Replication is with rsync and borg, not snapshots. Yes, it's more painful to replace disks after a failure, but once you get a bit used to it, it's really no big deal. On my main workstation/laptop the dedupe and compression work much better than my experience with ZFS. [1] https://news.ycombinator.com/item?id=16797644 https://news.ycombinator.com/item?id=16797644
  - sandreas 3y ago
    Does btrfs support native encryption?
    - antongribok 3y ago
      Btrfs does not have native encryption. I use LUKS / dm-crypt on entire drive partitions. Did that with ZFS too, and by the time ZFS got this feature I was already planning my migration to btrfs. I like encrypting with LUKS, because I can have a drive configured with mine and my wife's password. Either one of us can take that drive, plug it in, and Gnome will put a nice graphical prompt asking for a password, then decrypt the drive and mount the filesystem. If I get hit by a bus, LUKS makes it much easier for my family to get access to important data without having to have that data sit around somewhere in plaintext.

109 points by sandreas 3y ago

KyleSanderson 3y ago

https://github.com/openzfs/zfs/pull/15529#pullrequestreview-1732508242 https://github.com/openzfs/zfs/pull/15529#pullrequestreview-... Honestly, ZFS is the best thing on the (Free)BSDs only... On Linux it doesn't even use the page cache, and you conflict severely with L2ARC. I know there's a variety of people who don't care, but still for real users it's not an actual option.

antongribok 3y ago
For me it was the previous data corruption bug [1] that killed any enthusiasm for ZoL. After that annoyances like the caching issues you mention and the constant kernel upgrades breaking DKMS on Fedora just stopped being worth it for me. I finally moved to btrfs earlier this year, and so far I'm glad I did. I run raid1 on my primary array, and raid5 on my off-site backup array at my mom's apartment connecting with Tailscale. Replication is with rsync and borg, not snapshots. Yes, it's more painful to replace disks after a failure, but once you get a bit used to it, it's really no big deal. On my main workstation/laptop the dedupe and compression work much better than my experience with ZFS. [1] https://news.ycombinator.com/item?id=16797644 https://news.ycombinator.com/item?id=16797644
- sandreas 3y ago
  Does btrfs support native encryption?
  - antongribok 3y ago
    Btrfs does not have native encryption. I use LUKS / dm-crypt on entire drive partitions. Did that with ZFS too, and by the time ZFS got this feature I was already planning my migration to btrfs. I like encrypting with LUKS, because I can have a drive configured with mine and my wife's password. Either one of us can take that drive, plug it in, and Gnome will put a nice graphical prompt asking for a password, then decrypt the drive and mount the filesystem. If I get hit by a bus, LUKS makes it much easier for my family to get access to important data without having to have that data sit around somewhere in plaintext.