4 ms·

Doesn't mention if they're running Linux/what flavor they're running. I personally would be wary of running OpenZFS in production on Linux, especially ZFS on r

Doesn't mention if they're running Linux/what flavor they're running. I personally would be wary of running OpenZFS in production on Linux, especially ZFS on root. It has bit me in the ass too many times on Debian with an update breaking DKMS and rendering my system unbootable. Also, it's very, very strange/worrying to see no mention of disk encryption anywhere in the post or the tuning guide. For a company with encrypt in the name, that is responsible for the majority of trust on the internet, WTF? That should be highlighted in their benchmarking. ZFS supports native encryption, MariaDB does encryption, how are they encrypting at rest/transit/use?

vorpalhex 3y ago
You can always boot off ext4 and then just run data off OpenZFS pools. The benefits of booting off ZFS are extremely minimal compared to having your working data on ZFS.
deleted 3y ago
[deleted]
deleted 3y ago
[deleted]
shrubble 3y ago
ZFS has been very reliable for me since 2006 or so when I first started using it on SPARC hardware with the Solaris 10 beta; I assume that since they have a backup server and a primary server, they don't update and reboot them both at the same time.
pmarreck 3y ago
> I personally would be wary of running OpenZFS in production on Linux A ton of people in the enterprise have been doing this for years without issue; https://openzfs.org/wiki/Companies https://openzfs.org/wiki/Companies > especially ZFS on root I've been running ZFS on root on NixOS since this excellent guide (https://openzfs.github.io/openzfs-docs/Getting%20Started/NixOS/Root%20on%20ZFS.html https://openzfs.github.io/openzfs-docs/Getting%20Started/Nix...) for about 2 years. Zero issues. (Actually, I see they've updated it, I need to look at that. Also, they default to encrypted root. I turn it off, because the slight hit to performance and extra risk of recovery impossibility is not worth it to me.) > It has bit me in the ass too many times on Debian with an update breaking DKMS and rendering my system unbootable Well, I think you've found your problem, then. (That also might be why FreeNAS, which I also have running as my NAS, switched to Linux from Debian when they re-branded as TrueNAS Core/Enterprise.) Come over to NixOS, where you can simply reboot into any of N previous instances after an update that borks something (which almost never happens, anyway, because you can actually specify, in your machine configuration, "use the latest kernel that is compatible with ZFS"). No USB boot key needed. Here's the magic line from my own declarative configuration: kernelPackages = config.boot.zfs.package.latestCompatibleLinuxPackages; Aaaaand... DONE. ;) > Also, it's very, very strange/worrying to see no mention of disk encryption anywhere in the post or the tuning guide. For a company with encrypt in the name, that is responsible for the majority of trust on the internet, WTF? You're assuming they're not doing it, without evidence. Also, if they're already managing the security around their certs and cert generation properly, they might not need FDE. FDE is overrated IMHO, frankly, and also incurs a performance cost, as well as an extra risk cost (try recovering an encrypted drive to know what I mean). In short, religions are bad, even in technological choices; there is no single technological configuration choice that is 100% better than all possible alternative configurations. > That should be highlighted in their benchmarking. ZFS supports native encryption, MariaDB does encryption, how are they encrypting at rest/transit/use? Multiple layers of encryption incur an extra performance cost with almost no gain in extra security.
- deleted 3y ago
  [deleted]
DistractionRect 3y ago
Usually you aren't updating production servers unless it's a security patch, fixes a problem you have, or adds a feature you want/need. Even then, usually you have a test environment to verify the upgrade won't bork the system
- sofixa 3y ago
  Disagree, you have to continuously update production servers (not daily, but ~weekly/monthly). The more often you do it, the more automated and risk-free it is, and the smaller the version gap for when that critical security vulnerability hits that needs to be patched ASAP.
Caligatio 3y ago
Given that they're using a HSM (actually several), there's really not much that needs protection via FDE. The certs are obviously public and the domains are in the transparency log. On the ZFS note: it's been rock solid for me with Ubuntu but a living nightmare with Arch. My Arch update would upgrade the kernel but then OpenZFS would semi-routinely incompatible resulting in an unbootable system.
- pmarreck 3y ago
  I had the same issue. I went from Ubuntu -> Arch -> NixOS looking for a distro with well-supported ZFS. Finally found one (the last one). This is the magic line from my declarative configuration that ensures I never get a kernel update that is incompatible with my ZFS-on-root config: kernelPackages = config.boot.zfs.package.latestCompatibleLinuxPackages; Been running it for 2 years now; quite happy with it, and this is probably my "final distro hop". Once you go declarative (and climb the learning curve), you are done with everything else I guess. Put it this way, I can switch to an entirely different window manager by changing 1 configuration line. And it works every time. I've tried that on other Linux distros, and it always borks something or other. I can also change my GRUB theme and boot animation via Plymouth, something I would have NEVER risked on ANY other Linux distro due to the risk of modifying boot configuration... but since it's declarative (and validated) on NixOS, I've had no issues (just tweaking). If I manage to bork something, which is rare, I just reboot into a previous update of my OS and fix it and then try again.
  - hamandcheese 3y ago
    +1 to this, ZFS with NixOS is quite manageable. One thing I'm not a fan of though is the guide on the NixOS wiki was removed and instead points to this guide in the OpenZFS docs: https://openzfs.github.io/openzfs-docs/Getting%20Started/NixOS/ https://openzfs.github.io/openzfs-docs/Getting%20Started/Nix... It does not have much in the way of rationale - I.e. they say you need minimum 1GB reserve space but don't say what reserve space is and why you need it. I recall the NixOS wiki used to explain options and trade offs in a lot more detail. Nice tip re: latestCompatibleLinuxPackage! Had not seen that anywhere yet (probably because I haven't had any issues yet).
curt15 3y ago
ZFS on Linux is commonly used in HPC these days. E.g. https://computing.llnl.gov/projects/openzfs https://computing.llnl.gov/projects/openzfs

vorpalhex 3y ago

You can always boot off ext4 and then just run data off OpenZFS pools. The benefits of booting off ZFS are extremely minimal compared to having your working data on ZFS.

deleted 3y ago

[deleted]

deleted 3y ago

[deleted]

shrubble 3y ago

ZFS has been very reliable for me since 2006 or so when I first started using it on SPARC hardware with the Solaris 10 beta; I assume that since they have a backup server and a primary server, they don't update and reboot them both at the same time.

pmarreck 3y ago

> I personally would be wary of running OpenZFS in production on Linux A ton of people in the enterprise have been doing this for years without issue; https://openzfs.org/wiki/Companies https://openzfs.org/wiki/Companies > especially ZFS on root I've been running ZFS on root on NixOS since this excellent guide (https://openzfs.github.io/openzfs-docs/Getting%20Started/NixOS/Root%20on%20ZFS.html https://openzfs.github.io/openzfs-docs/Getting%20Started/Nix...) for about 2 years. Zero issues. (Actually, I see they've updated it, I need to look at that. Also, they default to encrypted root. I turn it off, because the slight hit to performance and extra risk of recovery impossibility is not worth it to me.) > It has bit me in the ass too many times on Debian with an update breaking DKMS and rendering my system unbootable Well, I think you've found your problem, then. (That also might be why FreeNAS, which I also have running as my NAS, switched to Linux from Debian when they re-branded as TrueNAS Core/Enterprise.) Come over to NixOS, where you can simply reboot into any of N previous instances after an update that borks something (which almost never happens, anyway, because you can actually specify, in your machine configuration, "use the latest kernel that is compatible with ZFS"). No USB boot key needed. Here's the magic line from my own declarative configuration: kernelPackages = config.boot.zfs.package.latestCompatibleLinuxPackages; Aaaaand... DONE. ;) > Also, it's very, very strange/worrying to see no mention of disk encryption anywhere in the post or the tuning guide. For a company with encrypt in the name, that is responsible for the majority of trust on the internet, WTF? You're assuming they're not doing it, without evidence. Also, if they're already managing the security around their certs and cert generation properly, they might not need FDE. FDE is overrated IMHO, frankly, and also incurs a performance cost, as well as an extra risk cost (try recovering an encrypted drive to know what I mean). In short, religions are bad, even in technological choices; there is no single technological configuration choice that is 100% better than all possible alternative configurations. > That should be highlighted in their benchmarking. ZFS supports native encryption, MariaDB does encryption, how are they encrypting at rest/transit/use? Multiple layers of encryption incur an extra performance cost with almost no gain in extra security.

deleted 3y ago
[deleted]

DistractionRect 3y ago

Usually you aren't updating production servers unless it's a security patch, fixes a problem you have, or adds a feature you want/need. Even then, usually you have a test environment to verify the upgrade won't bork the system

sofixa 3y ago
Disagree, you have to continuously update production servers (not daily, but ~weekly/monthly). The more often you do it, the more automated and risk-free it is, and the smaller the version gap for when that critical security vulnerability hits that needs to be patched ASAP.

Caligatio 3y ago

Given that they're using a HSM (actually several), there's really not much that needs protection via FDE. The certs are obviously public and the domains are in the transparency log. On the ZFS note: it's been rock solid for me with Ubuntu but a living nightmare with Arch. My Arch update would upgrade the kernel but then OpenZFS would semi-routinely incompatible resulting in an unbootable system.

pmarreck 3y ago
I had the same issue. I went from Ubuntu -> Arch -> NixOS looking for a distro with well-supported ZFS. Finally found one (the last one). This is the magic line from my declarative configuration that ensures I never get a kernel update that is incompatible with my ZFS-on-root config: kernelPackages = config.boot.zfs.package.latestCompatibleLinuxPackages; Been running it for 2 years now; quite happy with it, and this is probably my "final distro hop". Once you go declarative (and climb the learning curve), you are done with everything else I guess. Put it this way, I can switch to an entirely different window manager by changing 1 configuration line. And it works every time. I've tried that on other Linux distros, and it always borks something or other. I can also change my GRUB theme and boot animation via Plymouth, something I would have NEVER risked on ANY other Linux distro due to the risk of modifying boot configuration... but since it's declarative (and validated) on NixOS, I've had no issues (just tweaking). If I manage to bork something, which is rare, I just reboot into a previous update of my OS and fix it and then try again.
- hamandcheese 3y ago
  +1 to this, ZFS with NixOS is quite manageable. One thing I'm not a fan of though is the guide on the NixOS wiki was removed and instead points to this guide in the OpenZFS docs: https://openzfs.github.io/openzfs-docs/Getting%20Started/NixOS/ https://openzfs.github.io/openzfs-docs/Getting%20Started/Nix... It does not have much in the way of rationale - I.e. they say you need minimum 1GB reserve space but don't say what reserve space is and why you need it. I recall the NixOS wiki used to explain options and trade offs in a lot more detail. Nice tip re: latestCompatibleLinuxPackage! Had not seen that anywhere yet (probably because I haven't had any issues yet).

curt15 3y ago

ZFS on Linux is commonly used in HPC these days. E.g. https://computing.llnl.gov/projects/openzfs https://computing.llnl.gov/projects/openzfs