3 ms·

Same boat, while they keep shipping impressive features, I've been bitten far too many times on btrfs and zfs.

yjftsjthsd-h 3y ago
I've heard and experienced plenty of problems with BTRFS, but it's pretty rare that I hear of problems with ZFS (aside the the current article, obviously); what problems have you hit? (As a very heavy user of ZFS, I would very much like to know about possible problems before they bite me)
- mastax 3y ago
  I have two identical Debian 11 on ZFS servers with ZFS encryption enabled. On one pool on one machine it will start reporting errors after about 1 week of uptime. The errors are caused by failure to decrypt within the Kernel Cryptography Framework. No read/write/cksum errors are reported for any of the zpool devices. Running a scrub finds no errors and clears the errors. The presence of the errors causes my snapshot replication to fail so I need to reboot the server weekly and run a scrub. One person has reported that updating to 2.1.13 fixed the issue, IIRC. That version hasn't been released to Debian 11 yet and that version also included a commit which removed the KCF kstat code so I wouldn't be able to monitor the KCF errors anymore. Another issue I ran into is when SQLite is run in synchronous mode (the default) with WAL (not default but recommended) and the default locking mode (so it creates an -shm file) and the shm file is stored on a ZFS filesystem. SQLite frequently executes ftruncate on the shm file when different processes access the same SQLite database, and for some reason ZFS can cause the ftruncate call to block until the txg timeout which is usually 5 seconds [0]. If you were running a program which records every shell command you run to a SQLite database, for example, that would cause a 5 second hang before any shell command would execute [1]. The workaround is to disable synchronous in SQLite or the ZFS dataset, which is probably safe because of the ZFS atomicity guarantees. Those are two examples I have run into recently. I'm sure people run into issues with e.g. ext4 as well but I think they are a bit more frequent with ZFS on Linux, especially if you use more fringe features like encryption. [0]: https://github.com/openzfs/zfs/issues/14290 https://github.com/openzfs/zfs/issues/14290 [1]: https://github.com/atuinsh/atuin/issues/952 https://github.com/atuinsh/atuin/issues/952
  - ryao 3y ago
    Unfortunately, encryption never was quite as robust as the rest of ZFS. That said, things have become much better in the past year, as there have been targetted efforts focused on improving it, which found and fixed multiple bugs. That said, most of the developers are likely not using it on their own systems, which probably allowed bugs to stick around abnormally long compared to the rest of the code base (although some really ancient bugs were found last year in other areas).
- antongribok 3y ago
  Here's an example of a nasty data corruption bug in ZFS from not that long ago: https://github.com/openzfs/zfs/issues/7401 https://github.com/openzfs/zfs/issues/7401 Corresponding HN discussion at the time: https://news.ycombinator.com/item?id=16797644 https://news.ycombinator.com/item?id=16797644
  - ryao 3y ago
    That was 5 years ago. What would you consider to be a long time ago?
    - deleted 3y ago
      [deleted]
- viraptor 3y ago
  You can find many corruption issues on the openzfs bug tracker. The btrfs corruptions are a bit of a meme now and everyone has their own story from (old system, unsupported feature) failure long time ago, so it's quite hard to really compare how stable they are in realistic scenarios.
- tmikaeld 3y ago
  Since you asked me, it was many years ago now (5+) - but it hurt. I lost a 16TB cluster to btrfs, the raid just failed for no apparent reason (this wasn't uncommon back then according to forums). And on ZFS it was a kernel bug where on boot, it would mount ZFS after waiting for root - so root never appeared. (This was due to ZFS running as a user-space driver on linux back when it wasn't yet in the kernel)

I've heard and experienced plenty of problems with BTRFS, but it's pretty rare that I hear of problems with ZFS (aside the the current article, obviously); what problems have you hit? (As a very heavy user of ZFS, I would very much like to know about possible problems before they bite me)

mastax 3y ago
I have two identical Debian 11 on ZFS servers with ZFS encryption enabled. On one pool on one machine it will start reporting errors after about 1 week of uptime. The errors are caused by failure to decrypt within the Kernel Cryptography Framework. No read/write/cksum errors are reported for any of the zpool devices. Running a scrub finds no errors and clears the errors. The presence of the errors causes my snapshot replication to fail so I need to reboot the server weekly and run a scrub. One person has reported that updating to 2.1.13 fixed the issue, IIRC. That version hasn't been released to Debian 11 yet and that version also included a commit which removed the KCF kstat code so I wouldn't be able to monitor the KCF errors anymore. Another issue I ran into is when SQLite is run in synchronous mode (the default) with WAL (not default but recommended) and the default locking mode (so it creates an -shm file) and the shm file is stored on a ZFS filesystem. SQLite frequently executes ftruncate on the shm file when different processes access the same SQLite database, and for some reason ZFS can cause the ftruncate call to block until the txg timeout which is usually 5 seconds [0]. If you were running a program which records every shell command you run to a SQLite database, for example, that would cause a 5 second hang before any shell command would execute [1]. The workaround is to disable synchronous in SQLite or the ZFS dataset, which is probably safe because of the ZFS atomicity guarantees. Those are two examples I have run into recently. I'm sure people run into issues with e.g. ext4 as well but I think they are a bit more frequent with ZFS on Linux, especially if you use more fringe features like encryption. [0]: https://github.com/openzfs/zfs/issues/14290 https://github.com/openzfs/zfs/issues/14290 [1]: https://github.com/atuinsh/atuin/issues/952 https://github.com/atuinsh/atuin/issues/952
- ryao 3y ago
  Unfortunately, encryption never was quite as robust as the rest of ZFS. That said, things have become much better in the past year, as there have been targetted efforts focused on improving it, which found and fixed multiple bugs. That said, most of the developers are likely not using it on their own systems, which probably allowed bugs to stick around abnormally long compared to the rest of the code base (although some really ancient bugs were found last year in other areas).
antongribok 3y ago
Here's an example of a nasty data corruption bug in ZFS from not that long ago: https://github.com/openzfs/zfs/issues/7401 https://github.com/openzfs/zfs/issues/7401 Corresponding HN discussion at the time: https://news.ycombinator.com/item?id=16797644 https://news.ycombinator.com/item?id=16797644
- ryao 3y ago
  That was 5 years ago. What would you consider to be a long time ago?
  - deleted 3y ago
    [deleted]
viraptor 3y ago
You can find many corruption issues on the openzfs bug tracker. The btrfs corruptions are a bit of a meme now and everyone has their own story from (old system, unsupported feature) failure long time ago, so it's quite hard to really compare how stable they are in realistic scenarios.
tmikaeld 3y ago
Since you asked me, it was many years ago now (5+) - but it hurt. I lost a 16TB cluster to btrfs, the raid just failed for no apparent reason (this wasn't uncommon back then according to forums). And on ZFS it was a kernel bug where on boot, it would mount ZFS after waiting for root - so root never appeared. (This was due to ZFS running as a user-space driver on linux back when it wasn't yet in the kernel)