7 ms·

Immutable Software Deploys Using ZFS Jails on FreeBSD

194 points by vermaden 11mo ago

fukka42 11mo ago
Isn't this just docker with extra steps?
- ssl-3 11mo ago
  No. ZFS has been stable in FreeBSD for something like 17 years, and FreeBSD jails have been around for something like 25 years. By the time Docker hit 1.0 (about 11 years ago), the use of snapshots and jails had already been normal parts of life in the FreeBSD space for over half of a decade.
  - IgorPartola 11mo ago
    I never find arguments like this compelling (but I agree with your sentiment). I don’t much care to fire up the time machine to go back 12 or more years to develop software today. If your argument is that ZFS and jails provide the same functionality but are more stable than Docker. But as is it comes off as “get off my lawn you young whipper snappers”. But at the same time, the reason Docker won was not because it was groundbreaking tech or because it was amazingly well tested or anything. Just as one example, it has a years old bug which actively gets more comments every week having to do with Docker grossly mishandling quotes in env files. No, the reason it won is because the development experience and the deploy experience is easy, especially when you are on Linux AND on macOS. I can’t run FreeBSD jails or ZFS on macOS, can I? Definitely not with one file and one command. Jails and ZFS are amazing tech but they are not accessible. Docker made simple things very simple while being cross-platform enough. Do I feel gross using it? Yeah. It’s a kludgy solution to the problem. But it gets the job done and is supported by every provider out there. I am excited that it is being ported to FreeBSD though I know it will be a very long process.
    - stingraycharles 11mo ago
      Yeah I would love to use FreeBSD jails with ZFS and everything, it’s just that the whole cloud and containerization thing happened based on Linux and FreeBSD just never made it into that ecosystem. You’ll be sacrificing a lot and have to hand-roll a lot if you want your organization to switch from Linux+docker to FreeBSD+jails
      - jonhohle 11mo ago
        What about FreeBSD+OCI containers https://freebsdfoundation.org/blog/oci-containers-on-freebsd/ https://freebsdfoundation.org/blog/oci-containers-on-freebsd...
        nesarkvechnep 11mo ago
        In my opinion this is the path forward. I can already imagine Hashicorp Nomad orchestrator, with the podman driver, running fleets of FreeBSD containers.
      - randmeerkat 11mo ago
        Hacker News serves +4M requests per day using nothing but two FreeBSD servers…
        delian66 11mo ago
        4M requests/day is ~46 requests/second , for content, that could be cached a lot. Even if you have spikes that are 100x bigger than the average, that would be 4600r/s which does not seem like much in 2025.
      - kbenson 11mo ago
        It's all just history now for all I know, but there was work in the past to make Linux containers work on a Solaris fork (SmartOS, specifically) by emulating the Linux syscall table and presenting that to the containers. Joyent did work on this (alas, and there's an excellent and entertaining talk from Bryan Cantrill[1] that goes over it. I imagine FreeBSD could do something similar if they aren't already. IIRC FreeBSD has a Linux emulation layer (but I don't know how much attention it still gets), and it's had containerization primitives longer than linux, so some amount of filling in the gaps in containerization features and syscall tables (if needed) could possibly yield an OCI compatibility later (for all I know all this already exists). The problem, and the reason if this doesn't exist why people probably weren't as interested in doing the work, is it would always be "mostly" compatible and working and there would be no guarantee that the underlying software wouldn't exhibit bugs or weird behavior from small behavior differences in the platform when emulating something else. Why open yourself up to the headache when you can just run Linux with containers or build what you want on FreeBSD with jails and their own native containerization primitives. 1: https://www.youtube.com/watch?v=coFIEH3vXPw https://www.youtube.com/watch?v=coFIEH3vXPw
  - fukka42 11mo ago
    [dead]
- soupbowl 11mo ago
  This is more like making an immutable linux container using only OS base tools. Docker is a whole stack doing the work for you.
  - fukka42 11mo ago
    [dead]
  - miladyincontrol 11mo ago
    I have to imagine systemd’s nspawn with btrfs integration took much inspiration. Combined with systemd’s service configuration it really makes a wonderful way of running distroless, immutable containers.
    - arminiusreturns 11mo ago
      I second systemd-nspawn being a hidden gem for this usecase. I use git post-recieve hooks that target it for much of my ci/cd pipelines. I also find myself using nspawn just to isolate apps like firefox, etc.
- righthand 11mo ago
  This is like using Lxc with fewer steps and less tape. FreeBsd has jail managers aka container managers aka “Docker” as well.
  - righthand 11mo ago
    Try Bastille FreeBSD!
soupbowl 11mo ago
For those unfamiliar with FreeBSD, this is using base OS tools to manually create this type of immutable jail/container. This can be done with 'less effort' by using a jail manager.
- pacija 11mo ago
  Jail managers come and go. Base OS tools stay and are getting better and better. I would definitely stay away from ezjail as it us quite old, active development or even maintenance has stopped long time ago. Author of the article seem to know what they are doing so I'm puzzled why they don't use `bsdinstall jail /path/to/jail` to implement basejail instead of manually unpacking archives. No need for separate custom rc script to start `lo1`, it can be done with `cloned_interfaces` directive in rc.conf. Updating and upgrading jails by passing `-b /path/to/jail` to `freebsd-update` works, but new recommended way has lately been `-j <jailname>`. Cool article overall, the beauty of FreeBSD is also in possibility to do things in many different ways.
  - TacticalCoder 11mo ago
    [dead]
AdieuToLogic 11mo ago
The ezjail[0] port is another option for achieving the article's stated goal: FreeBSD's native support for ZFS snapshots and jails provides a powerful foundation for immutable deployments. I have not used the article's tool(s) and am not comparing the functionality provided by each. I have used ezjail[0] and found it exceptionally useful for similar concerns. 0 - https://erdgeist.org/arts/software/ezjail/ https://erdgeist.org/arts/software/ezjail/
fsmv 11mo ago
I built a reverse proxy server for this kind of usecase specifically. I have a chroot launcher with a simple config file and the reverse proxy is auto configured over RPC and automatically acts as a CA root so internal communication is HTTPS. You can also set the API key as an env var and just run your binary on the command line for testing and it will auto configure and be available on your URL with https. https://github.com/fsmv/daemon/ https://github.com/fsmv/daemon/ It's a bad time for me to be mentioning it because I have a major update that's not quite ready to release that changes some client APIs and makes the whole thing much nicer with fully automatic lets encrypt. I haven't had the space to work on it for a while unfortunately.
nesarkvechnep 11mo ago
I hope OCI containers support make this boilerplate obsolete.
- soupbowl 11mo ago
  It already is, if you use most jail managers, this post is a manual approach.
  - nesarkvechnep 11mo ago
    I really don’t like jail managers because I need to learn a non-standard way to operate jails when I already know the manual approach. I just wouldn’t miss the boilerplate. Also, OCI containers > jail managers.
- cyphar 11mo ago
  FYI, we just merged FreeBSD jail support into the OCI runtime spec v1.3[1]. There is already at least one implementation of it[2] as well. [1]: https://github.com/opencontainers/runtime-spec/pull/1286 https://github.com/opencontainers/runtime-spec/pull/1286 [2]: https://github.com/samuelkarp/runj https://github.com/samuelkarp/runj
righthand 11mo ago
If you want to do this kind of thing but don’t like the looks of this indepth tutorial, I highly reccomend installing Bastille FreeBSD instead and let the bastille tooling do the heavy lifting of managing zpool cloning and what not. Their getting started tutorial should get you 90% of the way there and has Docker like tooling. [0] Also found this self host on a Raspberry Pi helpful.[1] [0] https://bastillebsd.org/getting-started/ https://bastillebsd.org/getting-started/ [1] https://www.sharpwriting.net/project/bastille-jail-management-on-raspberry-pi-with-adguard/ https://www.sharpwriting.net/project/bastille-jail-managemen...
cullumsmith 11mo ago
I highly recommend this approach. I run ~all of my digital footprint on FreeBSD jails. After surveying how many jail managers have come and go in the last decade, I decided to just roll my own using a single shell script called jailctl [0]. Nothing fancy, just VNET jails based on ZFS templates (vanilla FreeBSD rootfs) and epair interfaces (which I truck to various VLANs on the host's egress interface). One pattern that I've found useful is to give each jail a persistently delegated ZFS dataset called "data." This lets me reprovision the OS image for the jail without having to backup and restore its application data (such as a Postgres DB). It also allows each jail to manage its own ZFS snapshots. The only thing that was a bit hairy was generating unique interface names and MAC addresses for each jail's VNET interface. My first instinct was to derive the interface name from the jail name, but interface names on FreeBSD are limited to 15 characters, and occasionally I'd hit this limit. In the end I did some dark magic using md5 sums of the jail name / host interface MAC address. Kind of ugly but I really didn't want to introduce any dependencies besides /bin/sh. [0] https://github.com/cullumsmith/infrastructure/blob/master/files/usr/local/sbin/jailctl.freebsd_hypervisor https://github.com/cullumsmith/infrastructure/blob/master/fi...
- ritcgab 11mo ago
  Very neat!
inopinatus 11mo ago
I used to do something like this but with ZFS on an OpenSolaris / Illumos storage server, exporting copy-on-write clones of snapshots of iSCSI volumes to boot Xen VMs on neighbouring blades, with tagged VLANs from each host because there were multiple guests to launch with varying roles in the application cluster. We made the VLAN number match 12 bits of the IP addresses and numbered the clones similarly. It merely remained to create new snapshots every release and any dev could launch an bugfix/test/showcase/etc environment for that version, and connect to it via a VPN. I was always worried about scale if we hired more than 4096 developers but fortunately the company was acquired and its product discontinued before that happened. That was in 2007 so the control plane (scheduler and automation) were built from scratch and we had very few reference points for the overall design. If I was building that today I’d probably still use ZFS clones but at filesystem level instead of block devices, and serve jails over NFS if I can get away with it, the iSCSI part was always a little janky.
evanjrowley 11mo ago
Detailed writeup here: https://docs.skunkwerks.at/s/fUiAmi4pE https://docs.skunkwerks.at/s/fUiAmi4pE
dzogchen 11mo ago
You’re creating a daemon but it is not restarting when it crashes, is it?

194 points by vermaden 11mo ago

fukka42 11mo ago

Isn't this just docker with extra steps?

ssl-3 11mo ago
No. ZFS has been stable in FreeBSD for something like 17 years, and FreeBSD jails have been around for something like 25 years. By the time Docker hit 1.0 (about 11 years ago), the use of snapshots and jails had already been normal parts of life in the FreeBSD space for over half of a decade.
- IgorPartola 11mo ago
  I never find arguments like this compelling (but I agree with your sentiment). I don’t much care to fire up the time machine to go back 12 or more years to develop software today. If your argument is that ZFS and jails provide the same functionality but are more stable than Docker. But as is it comes off as “get off my lawn you young whipper snappers”. But at the same time, the reason Docker won was not because it was groundbreaking tech or because it was amazingly well tested or anything. Just as one example, it has a years old bug which actively gets more comments every week having to do with Docker grossly mishandling quotes in env files. No, the reason it won is because the development experience and the deploy experience is easy, especially when you are on Linux AND on macOS. I can’t run FreeBSD jails or ZFS on macOS, can I? Definitely not with one file and one command. Jails and ZFS are amazing tech but they are not accessible. Docker made simple things very simple while being cross-platform enough. Do I feel gross using it? Yeah. It’s a kludgy solution to the problem. But it gets the job done and is supported by every provider out there. I am excited that it is being ported to FreeBSD though I know it will be a very long process.
  - stingraycharles 11mo ago
    Yeah I would love to use FreeBSD jails with ZFS and everything, it’s just that the whole cloud and containerization thing happened based on Linux and FreeBSD just never made it into that ecosystem. You’ll be sacrificing a lot and have to hand-roll a lot if you want your organization to switch from Linux+docker to FreeBSD+jails
    - jonhohle 11mo ago
      What about FreeBSD+OCI containers https://freebsdfoundation.org/blog/oci-containers-on-freebsd/ https://freebsdfoundation.org/blog/oci-containers-on-freebsd...
      - nesarkvechnep 11mo ago
        In my opinion this is the path forward. I can already imagine Hashicorp Nomad orchestrator, with the podman driver, running fleets of FreeBSD containers.
    - randmeerkat 11mo ago
      Hacker News serves +4M requests per day using nothing but two FreeBSD servers…
      - delian66 11mo ago
        4M requests/day is ~46 requests/second , for content, that could be cached a lot. Even if you have spikes that are 100x bigger than the average, that would be 4600r/s which does not seem like much in 2025.
    - kbenson 11mo ago
      It's all just history now for all I know, but there was work in the past to make Linux containers work on a Solaris fork (SmartOS, specifically) by emulating the Linux syscall table and presenting that to the containers. Joyent did work on this (alas, and there's an excellent and entertaining talk from Bryan Cantrill[1] that goes over it. I imagine FreeBSD could do something similar if they aren't already. IIRC FreeBSD has a Linux emulation layer (but I don't know how much attention it still gets), and it's had containerization primitives longer than linux, so some amount of filling in the gaps in containerization features and syscall tables (if needed) could possibly yield an OCI compatibility later (for all I know all this already exists). The problem, and the reason if this doesn't exist why people probably weren't as interested in doing the work, is it would always be "mostly" compatible and working and there would be no guarantee that the underlying software wouldn't exhibit bugs or weird behavior from small behavior differences in the platform when emulating something else. Why open yourself up to the headache when you can just run Linux with containers or build what you want on FreeBSD with jails and their own native containerization primitives. 1: https://www.youtube.com/watch?v=coFIEH3vXPw https://www.youtube.com/watch?v=coFIEH3vXPw
- fukka42 11mo ago
  [dead]
soupbowl 11mo ago
This is more like making an immutable linux container using only OS base tools. Docker is a whole stack doing the work for you.
- fukka42 11mo ago
  [dead]
- miladyincontrol 11mo ago
  I have to imagine systemd’s nspawn with btrfs integration took much inspiration. Combined with systemd’s service configuration it really makes a wonderful way of running distroless, immutable containers.
  - arminiusreturns 11mo ago
    I second systemd-nspawn being a hidden gem for this usecase. I use git post-recieve hooks that target it for much of my ci/cd pipelines. I also find myself using nspawn just to isolate apps like firefox, etc.
righthand 11mo ago
This is like using Lxc with fewer steps and less tape. FreeBsd has jail managers aka container managers aka “Docker” as well.
- righthand 11mo ago
  Try Bastille FreeBSD!

soupbowl 11mo ago

For those unfamiliar with FreeBSD, this is using base OS tools to manually create this type of immutable jail/container. This can be done with 'less effort' by using a jail manager.

pacija 11mo ago
Jail managers come and go. Base OS tools stay and are getting better and better. I would definitely stay away from ezjail as it us quite old, active development or even maintenance has stopped long time ago. Author of the article seem to know what they are doing so I'm puzzled why they don't use `bsdinstall jail /path/to/jail` to implement basejail instead of manually unpacking archives. No need for separate custom rc script to start `lo1`, it can be done with `cloned_interfaces` directive in rc.conf. Updating and upgrading jails by passing `-b /path/to/jail` to `freebsd-update` works, but new recommended way has lately been `-j <jailname>`. Cool article overall, the beauty of FreeBSD is also in possibility to do things in many different ways.
- TacticalCoder 11mo ago
  [dead]

AdieuToLogic 11mo ago

The ezjail[0] port is another option for achieving the article's stated goal: FreeBSD's native support for ZFS snapshots and jails provides a powerful foundation for immutable deployments. I have not used the article's tool(s) and am not comparing the functionality provided by each. I have used ezjail[0] and found it exceptionally useful for similar concerns. 0 - https://erdgeist.org/arts/software/ezjail/ https://erdgeist.org/arts/software/ezjail/

fsmv 11mo ago

I built a reverse proxy server for this kind of usecase specifically. I have a chroot launcher with a simple config file and the reverse proxy is auto configured over RPC and automatically acts as a CA root so internal communication is HTTPS. You can also set the API key as an env var and just run your binary on the command line for testing and it will auto configure and be available on your URL with https. https://github.com/fsmv/daemon/ https://github.com/fsmv/daemon/ It's a bad time for me to be mentioning it because I have a major update that's not quite ready to release that changes some client APIs and makes the whole thing much nicer with fully automatic lets encrypt. I haven't had the space to work on it for a while unfortunately.

nesarkvechnep 11mo ago

I hope OCI containers support make this boilerplate obsolete.

soupbowl 11mo ago
It already is, if you use most jail managers, this post is a manual approach.
- nesarkvechnep 11mo ago
  I really don’t like jail managers because I need to learn a non-standard way to operate jails when I already know the manual approach. I just wouldn’t miss the boilerplate. Also, OCI containers > jail managers.
cyphar 11mo ago
FYI, we just merged FreeBSD jail support into the OCI runtime spec v1.3[1]. There is already at least one implementation of it[2] as well. [1]: https://github.com/opencontainers/runtime-spec/pull/1286 https://github.com/opencontainers/runtime-spec/pull/1286 [2]: https://github.com/samuelkarp/runj https://github.com/samuelkarp/runj

righthand 11mo ago

If you want to do this kind of thing but don’t like the looks of this indepth tutorial, I highly reccomend installing Bastille FreeBSD instead and let the bastille tooling do the heavy lifting of managing zpool cloning and what not. Their getting started tutorial should get you 90% of the way there and has Docker like tooling. [0] Also found this self host on a Raspberry Pi helpful.[1] [0] https://bastillebsd.org/getting-started/ https://bastillebsd.org/getting-started/ [1] https://www.sharpwriting.net/project/bastille-jail-management-on-raspberry-pi-with-adguard/ https://www.sharpwriting.net/project/bastille-jail-managemen...

cullumsmith 11mo ago

I highly recommend this approach. I run ~all of my digital footprint on FreeBSD jails. After surveying how many jail managers have come and go in the last decade, I decided to just roll my own using a single shell script called jailctl [0]. Nothing fancy, just VNET jails based on ZFS templates (vanilla FreeBSD rootfs) and epair interfaces (which I truck to various VLANs on the host's egress interface). One pattern that I've found useful is to give each jail a persistently delegated ZFS dataset called "data." This lets me reprovision the OS image for the jail without having to backup and restore its application data (such as a Postgres DB). It also allows each jail to manage its own ZFS snapshots. The only thing that was a bit hairy was generating unique interface names and MAC addresses for each jail's VNET interface. My first instinct was to derive the interface name from the jail name, but interface names on FreeBSD are limited to 15 characters, and occasionally I'd hit this limit. In the end I did some dark magic using md5 sums of the jail name / host interface MAC address. Kind of ugly but I really didn't want to introduce any dependencies besides /bin/sh. [0] https://github.com/cullumsmith/infrastructure/blob/master/files/usr/local/sbin/jailctl.freebsd_hypervisor https://github.com/cullumsmith/infrastructure/blob/master/fi...

ritcgab 11mo ago
Very neat!

inopinatus 11mo ago

I used to do something like this but with ZFS on an OpenSolaris / Illumos storage server, exporting copy-on-write clones of snapshots of iSCSI volumes to boot Xen VMs on neighbouring blades, with tagged VLANs from each host because there were multiple guests to launch with varying roles in the application cluster. We made the VLAN number match 12 bits of the IP addresses and numbered the clones similarly. It merely remained to create new snapshots every release and any dev could launch an bugfix/test/showcase/etc environment for that version, and connect to it via a VPN. I was always worried about scale if we hired more than 4096 developers but fortunately the company was acquired and its product discontinued before that happened. That was in 2007 so the control plane (scheduler and automation) were built from scratch and we had very few reference points for the overall design. If I was building that today I’d probably still use ZFS clones but at filesystem level instead of block devices, and serve jails over NFS if I can get away with it, the iSCSI part was always a little janky.

evanjrowley 11mo ago

Detailed writeup here: https://docs.skunkwerks.at/s/fUiAmi4pE https://docs.skunkwerks.at/s/fUiAmi4pE

dzogchen 11mo ago

You’re creating a daemon but it is not restarting when it crashes, is it?