3 ms·

Right. I'd like to hear your experience with running OpenZFS on nixOS. E.g. how long have you tried it and your update cadence and robustness etc. I'm in my qu

Right. I'd like to hear your experience with running OpenZFS on nixOS. E.g. how long have you tried it and your update cadence and robustness etc. I'm in my quest to find the "best supported" way to run ZFS on Linux and I've tried Ubuntu, Arch, RHEL, OpenSUSE. (I also tried FreeNAS (i.e. FreeBSD) in the past.) Ubuntu has knowingly made a version of OpenZFS with (with their own patch causing) data corruption bug in the stable release. So since then I lost trust in them. Arch Linux never officially supports it, and Arch's rolling release model with OpenZFS as kernel modules are not a good match for UX. RHEL with kABI tracking kmod maintained by upstream seems to be a great, robust support combination, but later I found out that the kABI is not fully covered so it working relies on a bit of luck. AFAIK, it is still not completely resolved. OpenSUSE with the "filesystem" project seems to be a semi-official support of seamless integration of OpenZFS with OpenSUSE. So far so good but I'm still evaluating. In principle, nix and nixOS should be quite robust from its design (practically reproducible build). But I am not trusting nix that much yet and prefer something more battle tested. Coincidentally, I like the documentation of all the previous OSes I mentioned above (where Ubuntu falls a bit shorter among the crowd.) And from my limited experience, nix and friends are quite poorly documented.

yjftsjthsd-h 2y ago
> I'd like to hear your experience with running OpenZFS on nixOS. E.g. how long have you tried it and your update cadence and robustness etc. My main laptop, and the main laptop before it, have run NixOS with ZFS root since November 2022. Commit history seems to show that I started on version 22.11 and I think I've stepped forward through every version since then (might have skipped one in the middle, not 100% sure; I didn't switch to flakes until partway through and channels aren't in the commit history). In that time, the only problem I can recall ever hitting is accidentally filling up the /boot partition because I wasn't cleaning up old generations. If I was doing it over I would either make /boot like 8GB, or just use zfsbootmenu so the kernel+initramfs live in the root filesystem and it's irrelevant. Other than that, it's been smooth sailing. > I'm in my quest to find the "best supported" way to run ZFS on Linux and I've tried Ubuntu, Arch, RHEL, OpenSUSE. (I also tried FreeNAS (i.e. FreeBSD) in the past.) Ah yeah, if the rest of your software+hardware+preferences support it, I would easily believe that FreeBSD is better at ZFS than any Linux distro. IIRC they even ship boot environments in the base system which is great. > Arch Linux never officially supports it, and Arch's rolling release model with OpenZFS as kernel modules are not a good match for UX. IIRC there's some unofficial repo that provides both the ZFS packages and kernels together? In any event, in NixOS just use boot.kernelPackages = config.boot.zfs.package.latestCompatibleLinuxPackages; to pin your kernel to the latest compatible version and you're set. > In principle, nix and nixOS should be quite robust from its design (practically reproducible build). But I am not trusting nix that much yet and prefer something more battle tested. One advantage of ZFS is that you can compliment nix's versioning (seriously I do recommend using flakes and keeping your system flake in version control) with regular old ZFS snapshots. Just run `zfs snapshot -r rpool@todaysdate` before any change and you can roll back just about anything. Also you can zfs send those snapshots to some external storage and have full system backups with easy incrementals as desired. > Coincidentally, I like the documentation of all the previous OSes I mentioned above (where Ubuntu falls a bit shorter among the crowd.) And from my limited experience, nix and friends are quite poorly documented. Yeah, that's fair. It's improving with time, but anything off the beaten path tends to devolve into random blog posts. OTOH, if you keep the NixOS system to a simple base system and run most of your stuff in distrobox or some other container, you can have the best of both worlds.
- KolenCh 2y ago
  Thanks. Interesting to know! The thing I worried most is not the kind of problems fixable by rolling back, but (potentially silent) data corruption. Do you know how exactly is OpenZFS on nixOS supported? For example, currently both the Arch Linux and nixOS guide in the OpenZFS documentation said people should mention @ne9z in filing issues on those platforms, but @ne9z no longer exists. I have seen how maintainers can screw up and causes data corruption (Ubuntu specifically) so I worry about that aspect as well.
  - yjftsjthsd-h 2y ago
    Hm, so I've not really touched that side of things; mostly I just interact as a happy user. I think the relevant sources are https://github.com/NixOS/nixpkgs/blob/master/pkgs/os-specific/linux/zfs/2_2.nix https://github.com/NixOS/nixpkgs/blob/master/pkgs/os-specifi... and https://github.com/NixOS/nixpkgs/blob/master/pkgs/os-specific/linux/zfs/generic.nix https://github.com/NixOS/nixpkgs/blob/master/pkgs/os-specifi... , with the former listing a couple of maintainers and the latter I think showing that NixOS is taking upstream OpenZFS with zero source code patches (because mkDerivation is passed `patches = extraPatches;` and I'm reasonably sure `extraPatches ? []` defines an empty list). I'm not sure why those two files list one maintainer in one and two in the other.

yjftsjthsd-h 2y ago

> I'd like to hear your experience with running OpenZFS on nixOS. E.g. how long have you tried it and your update cadence and robustness etc. My main laptop, and the main laptop before it, have run NixOS with ZFS root since November 2022. Commit history seems to show that I started on version 22.11 and I think I've stepped forward through every version since then (might have skipped one in the middle, not 100% sure; I didn't switch to flakes until partway through and channels aren't in the commit history). In that time, the only problem I can recall ever hitting is accidentally filling up the /boot partition because I wasn't cleaning up old generations. If I was doing it over I would either make /boot like 8GB, or just use zfsbootmenu so the kernel+initramfs live in the root filesystem and it's irrelevant. Other than that, it's been smooth sailing. > I'm in my quest to find the "best supported" way to run ZFS on Linux and I've tried Ubuntu, Arch, RHEL, OpenSUSE. (I also tried FreeNAS (i.e. FreeBSD) in the past.) Ah yeah, if the rest of your software+hardware+preferences support it, I would easily believe that FreeBSD is better at ZFS than any Linux distro. IIRC they even ship boot environments in the base system which is great. > Arch Linux never officially supports it, and Arch's rolling release model with OpenZFS as kernel modules are not a good match for UX. IIRC there's some unofficial repo that provides both the ZFS packages and kernels together? In any event, in NixOS just use boot.kernelPackages = config.boot.zfs.package.latestCompatibleLinuxPackages; to pin your kernel to the latest compatible version and you're set. > In principle, nix and nixOS should be quite robust from its design (practically reproducible build). But I am not trusting nix that much yet and prefer something more battle tested. One advantage of ZFS is that you can compliment nix's versioning (seriously I do recommend using flakes and keeping your system flake in version control) with regular old ZFS snapshots. Just run `zfs snapshot -r rpool@todaysdate` before any change and you can roll back just about anything. Also you can zfs send those snapshots to some external storage and have full system backups with easy incrementals as desired. > Coincidentally, I like the documentation of all the previous OSes I mentioned above (where Ubuntu falls a bit shorter among the crowd.) And from my limited experience, nix and friends are quite poorly documented. Yeah, that's fair. It's improving with time, but anything off the beaten path tends to devolve into random blog posts. OTOH, if you keep the NixOS system to a simple base system and run most of your stuff in distrobox or some other container, you can have the best of both worlds.

KolenCh 2y ago
Thanks. Interesting to know! The thing I worried most is not the kind of problems fixable by rolling back, but (potentially silent) data corruption. Do you know how exactly is OpenZFS on nixOS supported? For example, currently both the Arch Linux and nixOS guide in the OpenZFS documentation said people should mention @ne9z in filing issues on those platforms, but @ne9z no longer exists. I have seen how maintainers can screw up and causes data corruption (Ubuntu specifically) so I worry about that aspect as well.
- yjftsjthsd-h 2y ago
  Hm, so I've not really touched that side of things; mostly I just interact as a happy user. I think the relevant sources are https://github.com/NixOS/nixpkgs/blob/master/pkgs/os-specific/linux/zfs/2_2.nix https://github.com/NixOS/nixpkgs/blob/master/pkgs/os-specifi... and https://github.com/NixOS/nixpkgs/blob/master/pkgs/os-specific/linux/zfs/generic.nix https://github.com/NixOS/nixpkgs/blob/master/pkgs/os-specifi... , with the former listing a couple of maintainers and the latter I think showing that NixOS is taking upstream OpenZFS with zero source code patches (because mkDerivation is passed `patches = extraPatches;` and I'm reasonably sure `extraPatches ? []` defines an empty list). I'm not sure why those two files list one maintainer in one and two in the other.