4 ms·

Used ZFS on Debian in production for 8 years, yet to experience rough edges but always interested to learn.

by tomxor 2y ago

turtledragonfly 2y ago
Do you use ZFS for root, on Debian? (enabling "boot environments") I've recently switched my FreeBSD setups to use that scheme, and it's been nice. Would be interested to hear if it's similarly straightforward on Debian (my second-favorite OS :) Obviously requires support in the bootcode; I'm not sure of the state of that for Linux.
- Numerlor 2y ago
  I've been running zfs on root on my Debian home server, only the install was a tiny bit more involved but it was done in maybe half an hour going off of the guide and trying to understand everything it was doing with 0 experience (... And then did it again after I broke networking after an hour lol)
- tomxor 2y ago
  No, to be fair I use it for everything except root (which I prefer to backup as IaC, not that I wouldn't use ZFS, it's just not default so requires more effort). I use ZFS datasets to back everything except the OS, including DBs which has been the most satisfying aspect of using ZFS (being able to snapshot live DBs together with all other relevant state rather than a as a separate replication process). While I can't comment on using it for root, I can say for other use, the only annoyance I have is compiling the kernel module on every single update. This is automatically handled by APT, but for very small servers it can be slow... at most ~15 mins. Could be solved with private distribution for a large fleet but I can't be bother with that stuff. Hoping at some point Debian will relax their strict "to the letter of the GPL" attitude at some point like they did with install media drivers. But it's not the worst experience, at least installing is automated.
- free652 2y ago
  zfsbootmenu https://docs.zfsbootmenu.org/en/v3.0.x/ https://docs.zfsbootmenu.org/en/v3.0.x/ Also nice way to recover zfs if anything goes wrong. yea, it's a linux image for just booting. But you put it as an EFI image, and works great.
asveikau 2y ago
I have never had a good experience with out of tree kernel modules on linux, so I am skeptical of this report. Usually for Linux Debian is my distro of choice.
- tomxor 2y ago
  I have little experience with kernel modules. ZFS is the only one I have long term experience with on Debian. The only issue I've had with the kernel module aspect is VPS configuration, I use Linode and they used to automatically set everything to use the hypervisor kernel, but the kernel module is obviously built against the versions installed on your OS and these will probably be different. All you have to do is make sure the boot configuration is set to use GRUB if using a VPS. Linode seem to have switched to this by default for Debian now.
SoftTalker 2y ago
My experience with ZFS (over 10 years ago now) was awful. Still have a bad taste in my mouth though no doubt it's much improved today.
- zie 2y ago
  Sounds like maybe you were holding it wrong? This is year 20 for ZFS in production. Though Linux support a decade ago with ZFS was pretty rough around the edges. That's not true anymore for many distros of Linux.
  - SoftTalker 2y ago
    I didn't run it. But several times a year there were multi-day outages for "rebuilds" and at one point Oracle consultants were involved to get it back on its feet.
    - tomxor 2y ago
      If that's Oracle's ZFS then it's not really comparable, it's essentially a very old closed fork. So on top of using a languishing outdated version you have another reliability problem (Oracle). It's a little bit like comparing OracleDB to MariaDB. Most people use Open ZFS which is from the Illumos project, which was basically the escape hatch that the engineers who wrote ZFS used when Oracle tried to close source Solaris after the Sun acquisition. There are decades of improvements in all of the OSS versions that comprise Illumos (which Oracle has denied themselves by attempting to close source it, since they cannot feed off of downstream OSS code). i.e most of the people who wrote ZFS immediately left Oracle and worked on Open ZFS. Open ZFS is for both FreeBSD and Linux, and is what most people are referring to when discussing ZFS. I've never used Oracle ZFS and never will.
      - zie 2y ago
        Agreed on the larger picture. Oracle ZFS is calcified ZFS. Their last stable release it looks like was in January of 2023. OpenZFS released 2.3.1 a few weeks ago: https://github.com/openzfs/zfs/releases https://github.com/openzfs/zfs/releases It has direct support for FreeBSD and Linux. macOS, Windows and other ports work great, but are not (yet) upstream: https://openzfs.org/wiki/Distributions https://openzfs.org/wiki/Distributions
0x457 2y ago
Well, rough edges: - bootloader (when root is on ZFS)* - ARC vs kernel page cache and OOM Killer That's all in my opinion. *: Relative to FreeBSD, but that's because bootloader in FreeBSD is part of the base system and in linux it could be one of many.

Used ZFS on Debian in production for 8 years, yet to experience rough edges but always interested to learn.

by tomxor 2y ago

Used ZFS on Debian in production for 8 years, yet to experience rough edges but always interested to learn.

Do you use ZFS for root, on Debian? (enabling "boot environments") I've recently switched my FreeBSD setups to use that scheme, and it's been nice. Would be interested to hear if it's similarly straightforward on Debian (my second-favorite OS :) Obviously requires support in the bootcode; I'm not sure of the state of that for Linux.

Numerlor 2y ago
I've been running zfs on root on my Debian home server, only the install was a tiny bit more involved but it was done in maybe half an hour going off of the guide and trying to understand everything it was doing with 0 experience (... And then did it again after I broke networking after an hour lol)
tomxor 2y ago
No, to be fair I use it for everything except root (which I prefer to backup as IaC, not that I wouldn't use ZFS, it's just not default so requires more effort). I use ZFS datasets to back everything except the OS, including DBs which has been the most satisfying aspect of using ZFS (being able to snapshot live DBs together with all other relevant state rather than a as a separate replication process). While I can't comment on using it for root, I can say for other use, the only annoyance I have is compiling the kernel module on every single update. This is automatically handled by APT, but for very small servers it can be slow... at most ~15 mins. Could be solved with private distribution for a large fleet but I can't be bother with that stuff. Hoping at some point Debian will relax their strict "to the letter of the GPL" attitude at some point like they did with install media drivers. But it's not the worst experience, at least installing is automated.
free652 2y ago
zfsbootmenu https://docs.zfsbootmenu.org/en/v3.0.x/ https://docs.zfsbootmenu.org/en/v3.0.x/ Also nice way to recover zfs if anything goes wrong. yea, it's a linux image for just booting. But you put it as an EFI image, and works great.

asveikau 2y ago

I have never had a good experience with out of tree kernel modules on linux, so I am skeptical of this report. Usually for Linux Debian is my distro of choice.

tomxor 2y ago
I have little experience with kernel modules. ZFS is the only one I have long term experience with on Debian. The only issue I've had with the kernel module aspect is VPS configuration, I use Linode and they used to automatically set everything to use the hypervisor kernel, but the kernel module is obviously built against the versions installed on your OS and these will probably be different. All you have to do is make sure the boot configuration is set to use GRUB if using a VPS. Linode seem to have switched to this by default for Debian now.

SoftTalker 2y ago

My experience with ZFS (over 10 years ago now) was awful. Still have a bad taste in my mouth though no doubt it's much improved today.

zie 2y ago
Sounds like maybe you were holding it wrong? This is year 20 for ZFS in production. Though Linux support a decade ago with ZFS was pretty rough around the edges. That's not true anymore for many distros of Linux.
- SoftTalker 2y ago
  I didn't run it. But several times a year there were multi-day outages for "rebuilds" and at one point Oracle consultants were involved to get it back on its feet.
  - tomxor 2y ago
    If that's Oracle's ZFS then it's not really comparable, it's essentially a very old closed fork. So on top of using a languishing outdated version you have another reliability problem (Oracle). It's a little bit like comparing OracleDB to MariaDB. Most people use Open ZFS which is from the Illumos project, which was basically the escape hatch that the engineers who wrote ZFS used when Oracle tried to close source Solaris after the Sun acquisition. There are decades of improvements in all of the OSS versions that comprise Illumos (which Oracle has denied themselves by attempting to close source it, since they cannot feed off of downstream OSS code). i.e most of the people who wrote ZFS immediately left Oracle and worked on Open ZFS. Open ZFS is for both FreeBSD and Linux, and is what most people are referring to when discussing ZFS. I've never used Oracle ZFS and never will.
    - zie 2y ago
      Agreed on the larger picture. Oracle ZFS is calcified ZFS. Their last stable release it looks like was in January of 2023. OpenZFS released 2.3.1 a few weeks ago: https://github.com/openzfs/zfs/releases https://github.com/openzfs/zfs/releases It has direct support for FreeBSD and Linux. macOS, Windows and other ports work great, but are not (yet) upstream: https://openzfs.org/wiki/Distributions https://openzfs.org/wiki/Distributions

0x457 2y ago

Well, rough edges: - bootloader (when root is on ZFS)* - ARC vs kernel page cache and OOM Killer That's all in my opinion. *: Relative to FreeBSD, but that's because bootloader in FreeBSD is part of the base system and in linux it could be one of many.