3 ms·

I have two identical Debian 11 on ZFS servers with ZFS encryption enabled. On one pool on one machine it will start reporting errors after about 1 week of uptim

by mastax 3y ago

I have two identical Debian 11 on ZFS servers with ZFS encryption enabled. On one pool on one machine it will start reporting errors after about 1 week of uptime. The errors are caused by failure to decrypt within the Kernel Cryptography Framework. No read/write/cksum errors are reported for any of the zpool devices. Running a scrub finds no errors and clears the errors. The presence of the errors causes my snapshot replication to fail so I need to reboot the server weekly and run a scrub. One person has reported that updating to 2.1.13 fixed the issue, IIRC. That version hasn't been released to Debian 11 yet and that version also included a commit which removed the KCF kstat code so I wouldn't be able to monitor the KCF errors anymore. Another issue I ran into is when SQLite is run in synchronous mode (the default) with WAL (not default but recommended) and the default locking mode (so it creates an -shm file) and the shm file is stored on a ZFS filesystem. SQLite frequently executes ftruncate on the shm file when different processes access the same SQLite database, and for some reason ZFS can cause the ftruncate call to block until the txg timeout which is usually 5 seconds [0]. If you were running a program which records every shell command you run to a SQLite database, for example, that would cause a 5 second hang before any shell command would execute [1]. The workaround is to disable synchronous in SQLite or the ZFS dataset, which is probably safe because of the ZFS atomicity guarantees. Those are two examples I have run into recently. I'm sure people run into issues with e.g. ext4 as well but I think they are a bit more frequent with ZFS on Linux, especially if you use more fringe features like encryption. [0]: https://github.com/openzfs/zfs/issues/14290 https://github.com/openzfs/zfs/issues/14290 [1]: https://github.com/atuinsh/atuin/issues/952 https://github.com/atuinsh/atuin/issues/952

ryao 3y ago
Unfortunately, encryption never was quite as robust as the rest of ZFS. That said, things have become much better in the past year, as there have been targetted efforts focused on improving it, which found and fixed multiple bugs. That said, most of the developers are likely not using it on their own systems, which probably allowed bugs to stick around abnormally long compared to the rest of the code base (although some really ancient bugs were found last year in other areas).

by mastax 3y ago

ryao 3y ago

Unfortunately, encryption never was quite as robust as the rest of ZFS. That said, things have become much better in the past year, as there have been targetted efforts focused on improving it, which found and fixed multiple bugs. That said, most of the developers are likely not using it on their own systems, which probably allowed bugs to stick around abnormally long compared to the rest of the code base (although some really ancient bugs were found last year in other areas).