8 ms·

I'd love to get rid of my FreeNAS VM and run ZFS directly on my Linux desktop, but having to mess with the kernel has kept me from attempting it so far. Maybe I

I'd love to get rid of my FreeNAS VM and run ZFS directly on my Linux desktop, but having to mess with the kernel has kept me from attempting it so far. Maybe I'm worrying about nothing. btrfs seems like the main alternative if you want native kernel support, but when I checked a couple years ago there seemed to be a lot of concerns about the stability. Is that still the case?

paulsmal 6y ago
You know Ubuntu support ZFS since 20.04. Experimental, but quite stable for me. Just select file system during installation process.
- sliken 6y ago
  Or say apt install zfs-utils, that's it, not even a reboot.
  - paulsmal 6y ago
    Oh yeah, right. Installation method if one needs ZFS on root.
arwineap 6y ago
debian has zfs in the contrib repo since stretch; no manual hacking required just have to enable contrib I switched my freebsd box over to debian about two years ago. No complaints so far :)
nix23 6y ago
>ZFS directly on my Linux desktop Use BTRFS trust me it's stable now...well the commands are terrible compared to ZFS. All my Server are FreeBSD but on the Laptop and on one Workstation i have openSUSE Tumbleweed since like 2 years and it works great.
- neolog 6y ago
  > the commands are terrible what does that mean?
  - vetinari 6y ago
    For zfs specific features, there are 'zfs' and 'zpool' commands (well, binaries, and the first parameter is a command). For btrfs, there is 'btrfs'. So I guess that the GP considers /usr/sbin/{zfs,zpool} more intuitive than /usr/sbin/btrfs.
    - nix23 6y ago
      Well zpool is for the pool which consist of filesystems/datasets or zfs's. So i have a clear distinction of the whole pool (pools or logical groups) or filesystems/datasets (zfs or logical volumes). It has nothing todo with /usr/sbin/x
  - nix23 6y ago
    https://btrfs.wiki.kernel.org/index.php/Manpage/btrfs https://btrfs.wiki.kernel.org/index.php/Manpage/btrfs >what does that mean? Not functional but logical (for me)
  - chungy 6y ago
    The "btrfs" tool has a lot of leaky abstractions, confusing intended usage, and gotchas all over the place. If you aren't a btrfs developer, it is difficult to know what exactly you want to do and how to accomplish it. ZFS on the other hand has just two commands for common administration tasks: zpool and zfs. zpool controls pool-level operations, mainly ones that deal with the storage layer; zfs controls the logical file systems and volumes that are contained within a pool. The zpool and zfs commands have been meticulously crafted to not expose much of the underlying software architecture and focus only on what administrators want, and all of it is clearly documented. There are actually a few other commands that come with ZFS if you really want or need to deal with low-level and difficult details, commands like zdb, zinject, zstreamdump. You almost never need any of them.
- 6y ago
ariabuckles 6y ago
Both openSUSE and [as of very recently] Fedora use btrfs by default, so btrfs support seems pretty stable these days. (But as others have pointed out, there are options for using zfs on linux, too)
- ed25519FUUU 6y ago
  btrfs was a really underutilized filesystem. It still has some superior features to zfs (such as offline deduplication), but the momentum now is clearly with zfs.
- bmn__ 6y ago
  Attempting to use zfs for the root partition is a huge headache because the software lives in the supplementary `filesystems` repo. https://build.opensuse.org/package/show/filesystems/zfs https://build.opensuse.org/package/show/filesystems/zfs 1. It often happens that the main repo offers a new kernel, but the corresponding module is not ready on obs yet. This means upgrading to the latest rolling release cannot just happen at any time, but requires careful planning. This is a big inconvenience. 2. In the past dracut sometimes just failed to pick up the module for the initrd, causing a boot failure at the next system start. I could not figure out why, however this never happened with the first class supported ext/xfs. 3. The distro's boot/rescue media do not contain the driver. This means a third-party boot medium is required to go into a broken system, and repairing it when chroot is involved is now much more complicated because of the different distro.
weitzj 6y ago
The easiest way is using t Proxmox installer which has ZFS as a filesystem. Underneath it is a Debian 10 installation. Last time I tried you could not enable ZFS encryption. I don’t know what is the case with Openzfs 2.0 Do we have encryption,yet?
- paulsmal 6y ago
  Ubuntu 20.10 has an option in installer to use ZFS encryption for root partition.
  - remram 6y ago
    Oh sweet, I don't think encryption was an option in 20.04. Is it still under an "experimental" option in the installer?
    - paulsmal 6y ago
      Yeah, still experimental but improved over 20.04. Added encryption and "autotrim=on" by default. I think Canonical planing is to have ZFS as an experimental option only for Desktop version of Ubuntu until next LTS release.
- vetinari 6y ago
  Proxmox uses their own kernels, so they build the zfs modules as well. They are bundled in the pve-kernel package.
Ericson2314 6y ago
ZFS is no extra work with NixOS! You just declare the filesystem type like any other in the config and it takes care of kernel modules and what-not.
- brnt 6y ago
  Is there a guide for this? Sounds interesting!
  - jfb 6y ago
    For instance, I have: fileSystems."/zfs/media" = { device = "tank/media"; fsType = "zfs"; }; in my hardware-configuration.nix. tank/media is defined as using a legacy mount-point or whatever the ZFS terminology is. Done. ETA: I mean, I had to do all the gruntwork to get the pool built, yeah. But once it was defined, getting it mounted and all the kernel bits and bobs set was trivial like that.
    - evil-olive 6y ago
      In addition, if you just want to play around with it: boot.supportedFilesystems = [ "zfs" ]; Both installs the necessary kernel modules and adds zpool(1) / zfs(1) to $PATH.
- pbronez 6y ago
  Sure - after you figure out NixOS lol
  - takeda 6y ago
    Actually no, NixOS is probably easier to use than other Linuxes. It gets more difficult when you need to package something new that it doesn't have, then you have to know the Nix language and how nixpkgs work.
    - mook 6y ago
      Interesting; when I tried it a while back it seemed like you needed to know the language to manage your configurations. Was that impression incorrect? I may have tried it far enough back that I pretty much immediately encountered packages I wished it had and tried (and failed) to package it myself, though, and got the experience mixed up…
kbumsik 6y ago
I personally use ZFS on Arch Linux. The DKMS package works almost out of the box and I haven't had any troubles. It takes a long time (but not too much) to compile though. Or you can use the latest Ubuntu that is shipped with ZFS.
- nvllsvm 6y ago
  Be aware of possible incompatibilities with the regular Arch kernel. I switched my NAS to linux-lts when a kernel point-release proved incompatible with openzfs.
  - kbumsik 6y ago
    Right, I forgot to mention I always use the LTS kernel, since I had getrandom() booting problem last year. https://lwn.net/Articles/800509/ https://lwn.net/Articles/800509/
- ed25519FUUU 6y ago
  Reporting in from Ubuntu 20.04 with ZFS on root. It works great. No issues so far, other than docker requires "zfs" as the storage-driver and doesn't support overlay2. See: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1718761 https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/171...
  - kbumsik 6y ago
    Interesting. I thought overlay2 can be agnostic to the base filesystems.
Jnr 6y ago
I'm using btrfs and my system still works. :)
- muxator 6y ago
  Me too. I have been running full btrfs since at least 5 years (single and RAID1 disks), which no data loss whatsoever. Now that there is support for swapfiles, I do not even need to create a dedicated partition for them.
briskstorm 6y ago
To mirror other comments, btrfs is pretty stable now and I run it on my server. The main problems now are with RAID5/6 profiles, the implementation has lots of issues still that can cause data loss[1]. Seems most of the core developers don't use those profiles so it hasn't been getting better. If you want to use RAID it would be safer to stick to RAID1/10[2] [1] https://lore.kernel.org/linux-btrfs/20200627032414.GX10769@hungrycats.org/ https://lore.kernel.org/linux-btrfs/20200627032414.GX10769@h... [2] https://www.man7.org/linux/man-pages/man8/mkfs.btrfs.8.html#PROFILES https://www.man7.org/linux/man-pages/man8/mkfs.btrfs.8.html#...
curt15 6y ago
On the btrfs mailing list [1], there are still sporadic reports of unrecoverable FS corruption for whatever reason. See [2], [3] for some recent examples. [1] https://lore.kernel.org/linux-btrfs/ https://lore.kernel.org/linux-btrfs/ [2] https://lore.kernel.org/linux-btrfs/CAD7Y51i=mTDnEWEJtSnUsq=xqbEtGC2NP0Yo4vAcPkSu+Wq+Rg@mail.gmail.com/T/#t https://lore.kernel.org/linux-btrfs/CAD7Y51i=mTDnEWEJtSnUsq=... [3] https://lore.kernel.org/linux-btrfs/CAMXR++KUj2L7qpR7QZeiM2Txav_EqerKKNM3ti1mK0q2-0n_-g@mail.gmail.com/T/#t https://lore.kernel.org/linux-btrfs/CAMXR++KUj2L7qpR7QZeiM2T...
- viraptor 6y ago
  [3] is interesting, thanks for linking that. [2] has so many moving parts, I wouldn't expect it to be related to btrfs without more information. I mean, there's both the fs and the cache layer being resized down, with unknown method.
rodgerd 6y ago
> but having to mess with the kernel has kept me from attempting it so far. Maybe I'm worrying about nothing. For the most part, yes. Occasionally a kernel developer who seems to be bitter about a company that doesn't exist any more tries to break compat with ZFS, but it's generally smooth sailing on Fedora, Debian, and CentOS, with dkms handling the building of modules seamlessly.
linsomniac 6y ago
I've been running encrypted ZFS on 20.04 on my main workstation since it came out and it's worked great. Wrote up details here, it's a slight hack for encryption, no hack if you don't want crypto. https://linsomniac.gitlab.io/post/2020-04-09-ubuntu-2004-encrypted-zfs/ https://linsomniac.gitlab.io/post/2020-04-09-ubuntu-2004-enc... A friend did a video based on my blog: https://www.youtube.com/watch?v=PILrUcXYwmc https://www.youtube.com/watch?v=PILrUcXYwmc
- koolba 6y ago
  Why ZFS encryption vs unencrypted ZFS atop LUKS?
  - yjftsjthsd-h 6y ago
    Why add another separate layer?
    - koolba 6y ago
      Because traditionally, programs do one thing well. And in my experience LUKS works great.
      - yjftsjthsd-h 6y ago
        If you're using ZFS, you're already okay with some level of ignoring that; ZFS is a inherently huge layering violation. It's a filesystem and a volume manager with encryption and compression, its own user access system (zfs allow), and its own NFS implementation.
        p_l 6y ago
        Umm, that's untrue. It's a very cleanly layered system, it just doesn't bother end user with details (as implementor, you can play with them, thus LustreZFS): there's separate SPA (block), DMU (OSD) and ZPL (FS) & ZVOL (emulated block device) layers. Compression and encryption are integrated at DMU level because that's a logical place for them. NFS actually calls OS nfs server.
heavyset_go 6y ago
I use btrfs on both servers and laptops without a problem these days. This wasn't the case almost a decade ago, though, when I got bit by its then-instability.
plmu 6y ago
Just use something else for your root file system, and zfs for the rest. I've been running Zol for 10 years (on arch) and had to recover a few times, but it was never difficult because of the totally standard setup except for the data disks.

paulsmal 6y ago

You know Ubuntu support ZFS since 20.04. Experimental, but quite stable for me. Just select file system during installation process.

sliken 6y ago
Or say apt install zfs-utils, that's it, not even a reboot.
- paulsmal 6y ago
  Oh yeah, right. Installation method if one needs ZFS on root.

arwineap 6y ago

debian has zfs in the contrib repo since stretch; no manual hacking required just have to enable contrib I switched my freebsd box over to debian about two years ago. No complaints so far :)

nix23 6y ago

>ZFS directly on my Linux desktop Use BTRFS trust me it's stable now...well the commands are terrible compared to ZFS. All my Server are FreeBSD but on the Laptop and on one Workstation i have openSUSE Tumbleweed since like 2 years and it works great.

neolog 6y ago
> the commands are terrible what does that mean?
- vetinari 6y ago
  For zfs specific features, there are 'zfs' and 'zpool' commands (well, binaries, and the first parameter is a command). For btrfs, there is 'btrfs'. So I guess that the GP considers /usr/sbin/{zfs,zpool} more intuitive than /usr/sbin/btrfs.
  - nix23 6y ago
    Well zpool is for the pool which consist of filesystems/datasets or zfs's. So i have a clear distinction of the whole pool (pools or logical groups) or filesystems/datasets (zfs or logical volumes). It has nothing todo with /usr/sbin/x
- nix23 6y ago
  https://btrfs.wiki.kernel.org/index.php/Manpage/btrfs https://btrfs.wiki.kernel.org/index.php/Manpage/btrfs >what does that mean? Not functional but logical (for me)
- chungy 6y ago
  The "btrfs" tool has a lot of leaky abstractions, confusing intended usage, and gotchas all over the place. If you aren't a btrfs developer, it is difficult to know what exactly you want to do and how to accomplish it. ZFS on the other hand has just two commands for common administration tasks: zpool and zfs. zpool controls pool-level operations, mainly ones that deal with the storage layer; zfs controls the logical file systems and volumes that are contained within a pool. The zpool and zfs commands have been meticulously crafted to not expose much of the underlying software architecture and focus only on what administrators want, and all of it is clearly documented. There are actually a few other commands that come with ZFS if you really want or need to deal with low-level and difficult details, commands like zdb, zinject, zstreamdump. You almost never need any of them.
6y ago

ariabuckles 6y ago

Both openSUSE and [as of very recently] Fedora use btrfs by default, so btrfs support seems pretty stable these days. (But as others have pointed out, there are options for using zfs on linux, too)

ed25519FUUU 6y ago
btrfs was a really underutilized filesystem. It still has some superior features to zfs (such as offline deduplication), but the momentum now is clearly with zfs.
bmn__ 6y ago
Attempting to use zfs for the root partition is a huge headache because the software lives in the supplementary `filesystems` repo. https://build.opensuse.org/package/show/filesystems/zfs https://build.opensuse.org/package/show/filesystems/zfs 1. It often happens that the main repo offers a new kernel, but the corresponding module is not ready on obs yet. This means upgrading to the latest rolling release cannot just happen at any time, but requires careful planning. This is a big inconvenience. 2. In the past dracut sometimes just failed to pick up the module for the initrd, causing a boot failure at the next system start. I could not figure out why, however this never happened with the first class supported ext/xfs. 3. The distro's boot/rescue media do not contain the driver. This means a third-party boot medium is required to go into a broken system, and repairing it when chroot is involved is now much more complicated because of the different distro.

weitzj 6y ago

The easiest way is using t Proxmox installer which has ZFS as a filesystem. Underneath it is a Debian 10 installation. Last time I tried you could not enable ZFS encryption. I don’t know what is the case with Openzfs 2.0 Do we have encryption,yet?

paulsmal 6y ago
Ubuntu 20.10 has an option in installer to use ZFS encryption for root partition.
- remram 6y ago
  Oh sweet, I don't think encryption was an option in 20.04. Is it still under an "experimental" option in the installer?
  - paulsmal 6y ago
    Yeah, still experimental but improved over 20.04. Added encryption and "autotrim=on" by default. I think Canonical planing is to have ZFS as an experimental option only for Desktop version of Ubuntu until next LTS release.
vetinari 6y ago
Proxmox uses their own kernels, so they build the zfs modules as well. They are bundled in the pve-kernel package.

Ericson2314 6y ago

ZFS is no extra work with NixOS! You just declare the filesystem type like any other in the config and it takes care of kernel modules and what-not.

brnt 6y ago
Is there a guide for this? Sounds interesting!
- jfb 6y ago
  For instance, I have: fileSystems."/zfs/media" = { device = "tank/media"; fsType = "zfs"; }; in my hardware-configuration.nix. tank/media is defined as using a legacy mount-point or whatever the ZFS terminology is. Done. ETA: I mean, I had to do all the gruntwork to get the pool built, yeah. But once it was defined, getting it mounted and all the kernel bits and bobs set was trivial like that.
  - evil-olive 6y ago
    In addition, if you just want to play around with it: boot.supportedFilesystems = [ "zfs" ]; Both installs the necessary kernel modules and adds zpool(1) / zfs(1) to $PATH.
pbronez 6y ago
Sure - after you figure out NixOS lol
- takeda 6y ago
  Actually no, NixOS is probably easier to use than other Linuxes. It gets more difficult when you need to package something new that it doesn't have, then you have to know the Nix language and how nixpkgs work.
  - mook 6y ago
    Interesting; when I tried it a while back it seemed like you needed to know the language to manage your configurations. Was that impression incorrect? I may have tried it far enough back that I pretty much immediately encountered packages I wished it had and tried (and failed) to package it myself, though, and got the experience mixed up…

kbumsik 6y ago

I personally use ZFS on Arch Linux. The DKMS package works almost out of the box and I haven't had any troubles. It takes a long time (but not too much) to compile though. Or you can use the latest Ubuntu that is shipped with ZFS.

nvllsvm 6y ago
Be aware of possible incompatibilities with the regular Arch kernel. I switched my NAS to linux-lts when a kernel point-release proved incompatible with openzfs.
- kbumsik 6y ago
  Right, I forgot to mention I always use the LTS kernel, since I had getrandom() booting problem last year. https://lwn.net/Articles/800509/ https://lwn.net/Articles/800509/
ed25519FUUU 6y ago
Reporting in from Ubuntu 20.04 with ZFS on root. It works great. No issues so far, other than docker requires "zfs" as the storage-driver and doesn't support overlay2. See: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1718761 https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/171...
- kbumsik 6y ago
  Interesting. I thought overlay2 can be agnostic to the base filesystems.

Jnr 6y ago

I'm using btrfs and my system still works. :)

muxator 6y ago
Me too. I have been running full btrfs since at least 5 years (single and RAID1 disks), which no data loss whatsoever. Now that there is support for swapfiles, I do not even need to create a dedicated partition for them.

briskstorm 6y ago

To mirror other comments, btrfs is pretty stable now and I run it on my server. The main problems now are with RAID5/6 profiles, the implementation has lots of issues still that can cause data loss[1]. Seems most of the core developers don't use those profiles so it hasn't been getting better. If you want to use RAID it would be safer to stick to RAID1/10[2] [1] https://lore.kernel.org/linux-btrfs/20200627032414.GX10769@hungrycats.org/ https://lore.kernel.org/linux-btrfs/20200627032414.GX10769@h... [2] https://www.man7.org/linux/man-pages/man8/mkfs.btrfs.8.html#PROFILES https://www.man7.org/linux/man-pages/man8/mkfs.btrfs.8.html#...

curt15 6y ago

On the btrfs mailing list [1], there are still sporadic reports of unrecoverable FS corruption for whatever reason. See [2], [3] for some recent examples. [1] https://lore.kernel.org/linux-btrfs/ https://lore.kernel.org/linux-btrfs/ [2] https://lore.kernel.org/linux-btrfs/CAD7Y51i=mTDnEWEJtSnUsq=xqbEtGC2NP0Yo4vAcPkSu+Wq+Rg@mail.gmail.com/T/#t https://lore.kernel.org/linux-btrfs/CAD7Y51i=mTDnEWEJtSnUsq=... [3] https://lore.kernel.org/linux-btrfs/CAMXR++KUj2L7qpR7QZeiM2Txav_EqerKKNM3ti1mK0q2-0n_-g@mail.gmail.com/T/#t https://lore.kernel.org/linux-btrfs/CAMXR++KUj2L7qpR7QZeiM2T...

viraptor 6y ago
[3] is interesting, thanks for linking that. [2] has so many moving parts, I wouldn't expect it to be related to btrfs without more information. I mean, there's both the fs and the cache layer being resized down, with unknown method.

rodgerd 6y ago

> but having to mess with the kernel has kept me from attempting it so far. Maybe I'm worrying about nothing. For the most part, yes. Occasionally a kernel developer who seems to be bitter about a company that doesn't exist any more tries to break compat with ZFS, but it's generally smooth sailing on Fedora, Debian, and CentOS, with dkms handling the building of modules seamlessly.

linsomniac 6y ago

I've been running encrypted ZFS on 20.04 on my main workstation since it came out and it's worked great. Wrote up details here, it's a slight hack for encryption, no hack if you don't want crypto. https://linsomniac.gitlab.io/post/2020-04-09-ubuntu-2004-encrypted-zfs/ https://linsomniac.gitlab.io/post/2020-04-09-ubuntu-2004-enc... A friend did a video based on my blog: https://www.youtube.com/watch?v=PILrUcXYwmc https://www.youtube.com/watch?v=PILrUcXYwmc

koolba 6y ago
Why ZFS encryption vs unencrypted ZFS atop LUKS?
- yjftsjthsd-h 6y ago
  Why add another separate layer?
  - koolba 6y ago
    Because traditionally, programs do one thing well. And in my experience LUKS works great.
    - yjftsjthsd-h 6y ago
      If you're using ZFS, you're already okay with some level of ignoring that; ZFS is a inherently huge layering violation. It's a filesystem and a volume manager with encryption and compression, its own user access system (zfs allow), and its own NFS implementation.
      - p_l 6y ago
        Umm, that's untrue. It's a very cleanly layered system, it just doesn't bother end user with details (as implementor, you can play with them, thus LustreZFS): there's separate SPA (block), DMU (OSD) and ZPL (FS) & ZVOL (emulated block device) layers. Compression and encryption are integrated at DMU level because that's a logical place for them. NFS actually calls OS nfs server.

heavyset_go 6y ago

I use btrfs on both servers and laptops without a problem these days. This wasn't the case almost a decade ago, though, when I got bit by its then-instability.

plmu 6y ago

Just use something else for your root file system, and zfs for the rest. I've been running Zol for 10 years (on arch) and had to recover a few times, but it was never difficult because of the totally standard setup except for the data disks.