8 ms·

ZFS 2.2.1: Block Cloning disabled due to data corruption

135 points by turrini 3y ago

thrtythreeforty 3y ago
I was glad for this feature to land, but being conservative with my main storage array, I decided not to immediately jump on this. Prudence pays off I guess. That said, for my main workstation, I plan to migrate to bcachefs very quickly once it's mainlined. I haven't done enough introspection to be able to tell you why I can hold both opinions in my head so well.
- exitheone 3y ago
  I had the exact same reaction. Saw the massive list of new features and decided to wait for .1 or .2 releases to shake out the bugs. Seems like patience is paying off as always.
- tmikaeld 3y ago
  Same boat, while they keep shipping impressive features, I've been bitten far too many times on btrfs and zfs.
  - yjftsjthsd-h 3y ago
    I've heard and experienced plenty of problems with BTRFS, but it's pretty rare that I hear of problems with ZFS (aside the the current article, obviously); what problems have you hit? (As a very heavy user of ZFS, I would very much like to know about possible problems before they bite me)
    - mastax 3y ago
      I have two identical Debian 11 on ZFS servers with ZFS encryption enabled. On one pool on one machine it will start reporting errors after about 1 week of uptime. The errors are caused by failure to decrypt within the Kernel Cryptography Framework. No read/write/cksum errors are reported for any of the zpool devices. Running a scrub finds no errors and clears the errors. The presence of the errors causes my snapshot replication to fail so I need to reboot the server weekly and run a scrub. One person has reported that updating to 2.1.13 fixed the issue, IIRC. That version hasn't been released to Debian 11 yet and that version also included a commit which removed the KCF kstat code so I wouldn't be able to monitor the KCF errors anymore. Another issue I ran into is when SQLite is run in synchronous mode (the default) with WAL (not default but recommended) and the default locking mode (so it creates an -shm file) and the shm file is stored on a ZFS filesystem. SQLite frequently executes ftruncate on the shm file when different processes access the same SQLite database, and for some reason ZFS can cause the ftruncate call to block until the txg timeout which is usually 5 seconds [0]. If you were running a program which records every shell command you run to a SQLite database, for example, that would cause a 5 second hang before any shell command would execute [1]. The workaround is to disable synchronous in SQLite or the ZFS dataset, which is probably safe because of the ZFS atomicity guarantees. Those are two examples I have run into recently. I'm sure people run into issues with e.g. ext4 as well but I think they are a bit more frequent with ZFS on Linux, especially if you use more fringe features like encryption. [0]: https://github.com/openzfs/zfs/issues/14290 https://github.com/openzfs/zfs/issues/14290 [1]: https://github.com/atuinsh/atuin/issues/952 https://github.com/atuinsh/atuin/issues/952
      - ryao 3y ago
        Unfortunately, encryption never was quite as robust as the rest of ZFS. That said, things have become much better in the past year, as there have been targetted efforts focused on improving it, which found and fixed multiple bugs. That said, most of the developers are likely not using it on their own systems, which probably allowed bugs to stick around abnormally long compared to the rest of the code base (although some really ancient bugs were found last year in other areas).
    - antongribok 3y ago
      Here's an example of a nasty data corruption bug in ZFS from not that long ago: https://github.com/openzfs/zfs/issues/7401 https://github.com/openzfs/zfs/issues/7401 Corresponding HN discussion at the time: https://news.ycombinator.com/item?id=16797644 https://news.ycombinator.com/item?id=16797644
      - ryao 3y ago
        That was 5 years ago. What would you consider to be a long time ago?
        deleted 3y ago
        [deleted]
    - viraptor 3y ago
      You can find many corruption issues on the openzfs bug tracker. The btrfs corruptions are a bit of a meme now and everyone has their own story from (old system, unsupported feature) failure long time ago, so it's quite hard to really compare how stable they are in realistic scenarios.
    - tmikaeld 3y ago
      Since you asked me, it was many years ago now (5+) - but it hurt. I lost a 16TB cluster to btrfs, the raid just failed for no apparent reason (this wasn't uncommon back then according to forums). And on ZFS it was a kernel bug where on boot, it would mount ZFS after waiting for root - so root never appeared. (This was due to ZFS running as a user-space driver on linux back when it wasn't yet in the kernel)
- dralley 3y ago
  I'd probably give it a release or three (or 5). Still plenty of development actively happening despite it being mostly just Kent. But yes I plan to use it soon-ish as well.
- ryao 3y ago
  Kent does plenty of things right during the development process, but having a decade of experience contributing to OpenZFS/ZFSOnLinux, I can say that I will be very surprised if everything goes as well as people seem to think. There are many bugs in a complex code base and a simple 10x increase in userbase will undoubtably result in more reports of people hitting them. That said, I think he is laying a good foundation (from what I have heard/read about his development process), but there are many times when the best of us have been confident in code that turned out to have problems and I doubt he is an exception to this.
  - j1elo 3y ago
    > he is laying a good foundation (from what I have heard/read about his development process) Could you please share some blog or resource to read a bit about his dev process myself? That stuff usually interests me, to improve my own process.
    - ryao 3y ago
      It is mostly from external observation. He seems to be the reason that bcachefs isn't being shipped in production today, as he is trying to work through all of his backlog rather than shipping it in a WIP state. This is very different than how developers of a certain filesystem in Linux's kernel source tree that I shall not name do things. If I recall correclty, a developer of that filesystem told me several years ago that if they did not ship the code for users to use it, they would not find bugs. That file system is not much less buggy today than it was back then. :/
      - WD40ForRust2 3y ago
        btrfs
    - viraptor 3y ago
      So the bcachefs is built on the basics of bcache which existed on its own as a block layer and worked in production for many years. This is layering that doesn't really exist in other filesystems. Not sure if that's what the parent meant.
davisr 3y ago
How the heck did this ever make it through? I don't think this ever would have happened pre-OpenZFS, and undermines the stable reputation of ZFS. OpenZFS needs to do better and look to FreeBSD developers, not Linux developers, as role models.
- deleted 3y ago
  [deleted]
- ryao 3y ago
  A FreeBSD developer wrote this code.
  - keep_reading 3y ago
    So far nobody has proven that this bug exists on FreeBSD, though. :)
    - Always_Anon 3y ago
      Because FreeBSD is behind Linux in copying with reflinks. They couldn't even reproduce the issue if they wanted to.
      - ryao 3y ago
        This code was implemented for FreeBSD before it was implemented for Linux. I do not think it has made it to a FreeBSD release yet though.
        Always_Anon 3y ago
        No it wasn't: https://github.com/openzfs/zfs/issues/405 https://github.com/openzfs/zfs/issues/405 This hit Linux first, which is why Linux experienced the issue. FreeBSD does not currently have reflink cp.
        mnd999 3y ago
        It’s in FreeBSD 14. But they disabled block cloning by default.
m4r1k 3y ago
ZFS is undeniably impressive – I first started using it back in 2009 with OpenSolaris. This year, motivated mainly by the power consumption of my previous OmniOS setup, I made the move to the "dark side" with Synology. After more than a decade of relying on ZFS, I must admit, the transition to something less rigid but still robust has been quite refreshing. It could be a subjective feeling, but the recent years of OpenZFS development have reminded me a bit of the OpenStack experience. It seemed like almost anyone could contribute, sometimes resulting in features that were questionable in terms of stability, development, and overall thoughtfulness. Perhaps this is why iXsystems has taken a more cautious (albeit slower) approach in enabling new features in TrueNAS.
- deleted 3y ago
  [deleted]
- deleted 3y ago
  [deleted]
- mmh0000 3y ago
  I'm in nearly the same boat as you. I've been running my current 32TB ZFS storage array on an HP DL380 in my basement. The power consumption and noise are both incredible. I'm closley watching black friday/monday deals hoping for something on the Synology DS1522+.
  - doublepg23 3y ago
    I quite like my DS1819+. Not exactly a powerhouse but it's rock solid and has zero drama.
- nortonham 3y ago
  What I liked about solaris--and illumos based OS's--is the time slider built in to the file manager. Would live to see something like that in freebsd and/or linux
mnd999 3y ago
This wasn’t the only block cloning bug. They also completely broke cp from unencrypted to encrypted by cloning unencrypted blocks into the encrypted file system.
- 65a 3y ago
  I reported maybe the first instance of that (#15275), and lost my home pool (but was able to get my files back). It was not a good weekend.
terinjokes 3y ago
Reporter of #15526 here. I've still only ever seen this corruption in files being built, which seems to be a situation that the race can be won repeatably by some packages. I reviewed all my user data and I've found no corruption to it, but my workloads don't involve block clones.
- thulle 3y ago
  2nd in the bug here. I actually have some blockcloned data, f.e. steam uses it for all proton installs, but no issues other than the go build.
  - thulle 3y ago
    Coming back home and checking 2.2.1 out zfs instantly started spewing write & checksum errors due to #15533. Both this and #15526 seem to have underlying issues from <2.2 that are just more easily triggered now. First one also confirmed on FreeBSD now. Holding off on 2.2 seems recommended, and if you're keeping critical data on OpenZFS it might be a good idea to give the issues a glance. The 2nd one might have the same underlying solution as an issue that has given me system freezes when closing in on ~90% pool usage using 2.1 on top of LUKS.
- csdvrx 3y ago
  Don't forget how `zfs send | zfs receive` could cause both pools to be corrupted, reported in - https://github.com/openzfs/zfs/issues/15140 https://github.com/openzfs/zfs/issues/15140 : supposed to be from flushing largepages - https://github.com/openzfs/zfs/issues/15275 https://github.com/openzfs/zfs/issues/15275 fixed by https://github.com/openzfs/zfs/issues/15464 https://github.com/openzfs/zfs/issues/15464 : linked to block cloning Block cloning is a large suspect.
deleted 3y ago
[deleted]
giantg2 3y ago
I was excited to try zfs on a separate disk with Ubuntu, until I screwed up my windows boot... guess sit will be a while before I get to try it while I fix my mess.
antongribok 3y ago
This is why you should always have proper backups, and a great reminder that snapshots are not backups.
- hdhdjdjd 3y ago
  Except that in this case your backups might have already been corrupted... because the first version of the file was already messed up.
papichulo2023 3y ago
Am I the only waiting for real write cache on ssds (not just ram)?
veidr 3y ago
NOTE: This situation is worse than it seemed when this thread landed on HN; see also https://news.ycombinator.com/item?id=38405731 https://news.ycombinator.com/item?id=38405731

135 points by turrini 3y ago

I was glad for this feature to land, but being conservative with my main storage array, I decided not to immediately jump on this. Prudence pays off I guess. That said, for my main workstation, I plan to migrate to bcachefs very quickly once it's mainlined. I haven't done enough introspection to be able to tell you why I can hold both opinions in my head so well.

exitheone 3y ago
I had the exact same reaction. Saw the massive list of new features and decided to wait for .1 or .2 releases to shake out the bugs. Seems like patience is paying off as always.
tmikaeld 3y ago
Same boat, while they keep shipping impressive features, I've been bitten far too many times on btrfs and zfs.
- yjftsjthsd-h 3y ago
  I've heard and experienced plenty of problems with BTRFS, but it's pretty rare that I hear of problems with ZFS (aside the the current article, obviously); what problems have you hit? (As a very heavy user of ZFS, I would very much like to know about possible problems before they bite me)
  - mastax 3y ago
    I have two identical Debian 11 on ZFS servers with ZFS encryption enabled. On one pool on one machine it will start reporting errors after about 1 week of uptime. The errors are caused by failure to decrypt within the Kernel Cryptography Framework. No read/write/cksum errors are reported for any of the zpool devices. Running a scrub finds no errors and clears the errors. The presence of the errors causes my snapshot replication to fail so I need to reboot the server weekly and run a scrub. One person has reported that updating to 2.1.13 fixed the issue, IIRC. That version hasn't been released to Debian 11 yet and that version also included a commit which removed the KCF kstat code so I wouldn't be able to monitor the KCF errors anymore. Another issue I ran into is when SQLite is run in synchronous mode (the default) with WAL (not default but recommended) and the default locking mode (so it creates an -shm file) and the shm file is stored on a ZFS filesystem. SQLite frequently executes ftruncate on the shm file when different processes access the same SQLite database, and for some reason ZFS can cause the ftruncate call to block until the txg timeout which is usually 5 seconds [0]. If you were running a program which records every shell command you run to a SQLite database, for example, that would cause a 5 second hang before any shell command would execute [1]. The workaround is to disable synchronous in SQLite or the ZFS dataset, which is probably safe because of the ZFS atomicity guarantees. Those are two examples I have run into recently. I'm sure people run into issues with e.g. ext4 as well but I think they are a bit more frequent with ZFS on Linux, especially if you use more fringe features like encryption. [0]: https://github.com/openzfs/zfs/issues/14290 https://github.com/openzfs/zfs/issues/14290 [1]: https://github.com/atuinsh/atuin/issues/952 https://github.com/atuinsh/atuin/issues/952
    - ryao 3y ago
      Unfortunately, encryption never was quite as robust as the rest of ZFS. That said, things have become much better in the past year, as there have been targetted efforts focused on improving it, which found and fixed multiple bugs. That said, most of the developers are likely not using it on their own systems, which probably allowed bugs to stick around abnormally long compared to the rest of the code base (although some really ancient bugs were found last year in other areas).
  - antongribok 3y ago
    Here's an example of a nasty data corruption bug in ZFS from not that long ago: https://github.com/openzfs/zfs/issues/7401 https://github.com/openzfs/zfs/issues/7401 Corresponding HN discussion at the time: https://news.ycombinator.com/item?id=16797644 https://news.ycombinator.com/item?id=16797644
    - ryao 3y ago
      That was 5 years ago. What would you consider to be a long time ago?
      - deleted 3y ago
        [deleted]
  - viraptor 3y ago
    You can find many corruption issues on the openzfs bug tracker. The btrfs corruptions are a bit of a meme now and everyone has their own story from (old system, unsupported feature) failure long time ago, so it's quite hard to really compare how stable they are in realistic scenarios.
  - tmikaeld 3y ago
    Since you asked me, it was many years ago now (5+) - but it hurt. I lost a 16TB cluster to btrfs, the raid just failed for no apparent reason (this wasn't uncommon back then according to forums). And on ZFS it was a kernel bug where on boot, it would mount ZFS after waiting for root - so root never appeared. (This was due to ZFS running as a user-space driver on linux back when it wasn't yet in the kernel)
dralley 3y ago
I'd probably give it a release or three (or 5). Still plenty of development actively happening despite it being mostly just Kent. But yes I plan to use it soon-ish as well.
ryao 3y ago
Kent does plenty of things right during the development process, but having a decade of experience contributing to OpenZFS/ZFSOnLinux, I can say that I will be very surprised if everything goes as well as people seem to think. There are many bugs in a complex code base and a simple 10x increase in userbase will undoubtably result in more reports of people hitting them. That said, I think he is laying a good foundation (from what I have heard/read about his development process), but there are many times when the best of us have been confident in code that turned out to have problems and I doubt he is an exception to this.
- j1elo 3y ago
  > he is laying a good foundation (from what I have heard/read about his development process) Could you please share some blog or resource to read a bit about his dev process myself? That stuff usually interests me, to improve my own process.
  - ryao 3y ago
    It is mostly from external observation. He seems to be the reason that bcachefs isn't being shipped in production today, as he is trying to work through all of his backlog rather than shipping it in a WIP state. This is very different than how developers of a certain filesystem in Linux's kernel source tree that I shall not name do things. If I recall correclty, a developer of that filesystem told me several years ago that if they did not ship the code for users to use it, they would not find bugs. That file system is not much less buggy today than it was back then. :/
    - WD40ForRust2 3y ago
      btrfs
  - viraptor 3y ago
    So the bcachefs is built on the basics of bcache which existed on its own as a block layer and worked in production for many years. This is layering that doesn't really exist in other filesystems. Not sure if that's what the parent meant.

davisr 3y ago

How the heck did this ever make it through? I don't think this ever would have happened pre-OpenZFS, and undermines the stable reputation of ZFS. OpenZFS needs to do better and look to FreeBSD developers, not Linux developers, as role models.

deleted 3y ago
[deleted]
ryao 3y ago
A FreeBSD developer wrote this code.
- keep_reading 3y ago
  So far nobody has proven that this bug exists on FreeBSD, though. :)
  - Always_Anon 3y ago
    Because FreeBSD is behind Linux in copying with reflinks. They couldn't even reproduce the issue if they wanted to.
    - ryao 3y ago
      This code was implemented for FreeBSD before it was implemented for Linux. I do not think it has made it to a FreeBSD release yet though.
      - Always_Anon 3y ago
        No it wasn't: https://github.com/openzfs/zfs/issues/405 https://github.com/openzfs/zfs/issues/405 This hit Linux first, which is why Linux experienced the issue. FreeBSD does not currently have reflink cp.
      - mnd999 3y ago
        It’s in FreeBSD 14. But they disabled block cloning by default.

m4r1k 3y ago

ZFS is undeniably impressive – I first started using it back in 2009 with OpenSolaris. This year, motivated mainly by the power consumption of my previous OmniOS setup, I made the move to the "dark side" with Synology. After more than a decade of relying on ZFS, I must admit, the transition to something less rigid but still robust has been quite refreshing. It could be a subjective feeling, but the recent years of OpenZFS development have reminded me a bit of the OpenStack experience. It seemed like almost anyone could contribute, sometimes resulting in features that were questionable in terms of stability, development, and overall thoughtfulness. Perhaps this is why iXsystems has taken a more cautious (albeit slower) approach in enabling new features in TrueNAS.

deleted 3y ago
[deleted]
deleted 3y ago
[deleted]
mmh0000 3y ago
I'm in nearly the same boat as you. I've been running my current 32TB ZFS storage array on an HP DL380 in my basement. The power consumption and noise are both incredible. I'm closley watching black friday/monday deals hoping for something on the Synology DS1522+.
- doublepg23 3y ago
  I quite like my DS1819+. Not exactly a powerhouse but it's rock solid and has zero drama.
nortonham 3y ago
What I liked about solaris--and illumos based OS's--is the time slider built in to the file manager. Would live to see something like that in freebsd and/or linux

mnd999 3y ago

This wasn’t the only block cloning bug. They also completely broke cp from unencrypted to encrypted by cloning unencrypted blocks into the encrypted file system.

65a 3y ago
I reported maybe the first instance of that (#15275), and lost my home pool (but was able to get my files back). It was not a good weekend.

terinjokes 3y ago

Reporter of #15526 here. I've still only ever seen this corruption in files being built, which seems to be a situation that the race can be won repeatably by some packages. I reviewed all my user data and I've found no corruption to it, but my workloads don't involve block clones.

thulle 3y ago
2nd in the bug here. I actually have some blockcloned data, f.e. steam uses it for all proton installs, but no issues other than the go build.
- thulle 3y ago
  Coming back home and checking 2.2.1 out zfs instantly started spewing write & checksum errors due to #15533. Both this and #15526 seem to have underlying issues from <2.2 that are just more easily triggered now. First one also confirmed on FreeBSD now. Holding off on 2.2 seems recommended, and if you're keeping critical data on OpenZFS it might be a good idea to give the issues a glance. The 2nd one might have the same underlying solution as an issue that has given me system freezes when closing in on ~90% pool usage using 2.1 on top of LUKS.
csdvrx 3y ago
Don't forget how `zfs send | zfs receive` could cause both pools to be corrupted, reported in - https://github.com/openzfs/zfs/issues/15140 https://github.com/openzfs/zfs/issues/15140 : supposed to be from flushing largepages - https://github.com/openzfs/zfs/issues/15275 https://github.com/openzfs/zfs/issues/15275 fixed by https://github.com/openzfs/zfs/issues/15464 https://github.com/openzfs/zfs/issues/15464 : linked to block cloning Block cloning is a large suspect.

deleted 3y ago

[deleted]

giantg2 3y ago

I was excited to try zfs on a separate disk with Ubuntu, until I screwed up my windows boot... guess sit will be a while before I get to try it while I fix my mess.

antongribok 3y ago

This is why you should always have proper backups, and a great reminder that snapshots are not backups.

hdhdjdjd 3y ago
Except that in this case your backups might have already been corrupted... because the first version of the file was already messed up.

papichulo2023 3y ago

Am I the only waiting for real write cache on ssds (not just ram)?

veidr 3y ago

NOTE: This situation is worse than it seemed when this thread landed on HN; see also https://news.ycombinator.com/item?id=38405731 https://news.ycombinator.com/item?id=38405731