8 ms·

FreeBSD Jails Containers

226 points by vermaden 3y ago

baggy_trough 3y ago
Also check out systemd-nspawn on linux. Another nice container solution, built in, that seems to be relatively little known.
- packetlost 3y ago
  I made a very serious attempt at using this awhile back on Arch and it high-key sucks. Like, it's not really meant as a general purpose container runtime solution, it's meant for developing systemd and related software. The ergonomics, particularly around networking are quite bad and it literally requires that you're all in on systemd-networkd and resolved to get any networking at all.
  - baggy_trough 3y ago
    I’ve used it as a general purpose container runtime and been very pleased. Ymmv.
wyager 3y ago
Vnet jails are awesome. Totally changed the way I do networking and application hosting. Makes it pretty trivial to do things like set up a jail that can only talk to the internet through a VPN.
wwarner 3y ago
"The FreeBSD host can run any other FreeBSD as long as its not newer then the host system version." Docker does not have this limitation. You can run a container that is based on a newer version of Linux than the host.
- kstrauser 3y ago
  Depends what you mean by "Linux". You can possibly use a newer userland, but containers share the kernel with the host. That's kind of the whole point of containers: they're just a set of processes that are grouped together and configured with certain restrictions.
- chungy 3y ago
  They have that limitation, but it doesn't manifest in quite a simple way. If the container's glibc and all other binaries require Linux 6.3, you're not running it on a 5.18 host. FreeBSD is developed as a whole unit. It makes it easier to reason that a 13.2 userland doesn't run on a 12.4 kernel.
  - LegionMammal978 3y ago
    > If the container's glibc and all other binaries require Linux 6.3, you're not running it on a 5.18 host. FWIW, it would take quite a long time for glibc itself to start requiring Linux 6.3 on its current targets. Since glibc 2.24 (Aug. 2016), the minimum supported kernel has been at most Linux 3.2 (Jan. 2012), on every target that has existed that long. And that's the smallest that the gap between the glibc and Linux releases has ever gotten (except for new targets). Extrapolating, we'd have to wait for glibc 2.47 (Feb. 2028) to require Linux 6.3 (Apr. 2023), if that gap were to be repeated, something which I don't find particularly likely. That is to say, with the exception of new targets, you'll rarely see a userland that absolutely requires the latest Linux kernel, even possibly if it suggests that kernel for the best support.
    - JdeBP 3y ago
      It's the "all other binaries" part that will catch you. You've completely overlooked that part. For example: Per its own README, systemd requires Linux 3.15 at absolute minimum, recommends Linux 4.15 for baseline functionality to at least work, and specifies Linux 5.7 for full BPF functionality.
- ori_b 3y ago
  That's incorrect. (Well. Technically it's kind of true. You can run a container based on a newer kernel, it'll just crash when you use any new features)
- bandrami 3y ago
  Unless that container uses kernel semantics that changed between kernel versions, or a feature that was added in the new version. Which invariably happens. See the RHEL 6/7 upgrade and the changes to syscall.
idatum 3y ago
I have a VNET jail for remote access to my home lab. This takes an extra NIC I wasn't using anyway. It has been solid running on FreeBSD 13.2. This jail allows me to lock things down to minimal access to other services I need remotely, such as an SSH tunnel to Home Assistant. I run my FreeBSD server mainly for ZFS. Adding a jail for remote access adds to the usefulness that hardware.
- appleflaxen 3y ago
  Can you elaborate on what this looks like?
  - idatum 3y ago
    jail.conf: ssh { exec.start = "/bin/sh /etc/rc"; exec.stop = "/bin/sh /etc/rc.shutdown"; exec.clean; mount.devfs; path = "/var/jail/ssh"; host.hostname = "ssh"; vnet; vnet.interface += "em0"; } It's IPv6 only, so this is key in rc.conf: rtsold_enable="YES" OpenBSD PF based router does the rest. IPv6 simplifies things here. PF on FreeBSD isn't the most ideal, sure, but I can limit local access on inet6. For SSH tunnel, it's straight forward.
    - inferiorhuman 3y ago
      FWIW VNET is not inherently IPv6 only. I'm using it in an IPv4 only context just fine.
- toast0 3y ago
  An extra NIC is fine if you've got slots, but you can also do an if_bridge in the main host, add the real nic and half of an if_epair; then move the other half to the vnet jail. Potentially a non-vnet jail with the ipv4/ipv6 options set may also be an option. I used to use jails with those settings to squeeze multiple environments into a single host at Yahoo.
  - yakubin 3y ago
    I use only if_epair and then pf to NAT it to the external world. I’ve only had issues with if_bridge, where I would either be able to make connections between jail and the external world, but not between host and jail or vice versa (between host and jail fine, but not between jail and external world). And it would act really flaky, like getting one ping response, followed by silence. pf is a lot simpler and more reliable.
jonhohle 3y ago
The issue I have with Jails, over something like docker, is that the kernel ABI changes between major releases so jails need to migrate with base systems upgrades. It also means a container registry would need to store container versions per release. I like the isolation for my own bespoke home network, but does it work well in production set up’s?
- inferiorhuman 3y ago
  Kernels are generally backwards compatible and major versions are explicitly supported for five (5) years. Over in Linux land you'll see builds for the different libcs and it seems to work just fine.
  - johncolanduoni 3y ago
    The difference is that the container libc version and kernel version don't have to correspond to each other in any way on Linux (unless your kernel is like 20 years older, in which case it wouldn't support Docker anyway).
    - inferiorhuman 3y ago
      The end result is that you're matching the executable to the libc variant instead of to the kernel. In practice it's not much of a difference, if at all. If you need to use older binaries, install the compatibility shims (or use a base jail based on the older version of FreeBSD). In terms of a support matrix there are two major versions of FreeBSD that are given active support, which is less than over in Linux land. All in all of all the concerns I'd have about using jails in a production environment, dealing with different major versions is not on that list.
      - monocasa 3y ago
        But the half the point of the container is per application libcs/userlands, so the kernel ABI is the only required constant.
Lammy 3y ago
Props for using the base-system jail management tools instead of iocage/ezjail/etc.
- wyager 3y ago
  IME, the base-system jail stuff is super easy to understand, use, and customize. When you start using "easy" tools on top of that, it rapidly becomes more complicated and opaque for non-trivial stuff. It's awesome that you can do everything you would want with a jail with only 3 config files: rc.conf (for networking and start-up), pf.conf (for firewalling/routing), and jail.conf (for specifying jail params like mount points, network if setup, etc.). Super clean, super easy.
- vermaden 3y ago
  Thanks.
inferiorhuman 3y ago
VNET jails are great, especially now that they're stable, but I'd strongly recommend using a higher level tool. I've settled on iocage, but there are a few other tools out there that seem to have gained traction. Nobody sets up cgroups by hand, right?
- waz0wski 3y ago
  A couple of tools which are both working on jail management & packaging - bastille https://bastillebsd.org https://bastillebsd.org - pot https://potluck.honeyguide.net https://potluck.honeyguide.net bastille is great for personal and small setups, pot is aimed at larger scale and meant to be used along with hashicorp nomad+consul
  - dilippkumar 3y ago
    What is the state of pot in summer 2023? Is it ready for production?
    - waz0wski 3y ago
      Depends on your workloads and needs. It's not as mature as k8s and there are rough edges but if you're into the Hashicorp ecosystem and can run your workloads in FreeBSD jails it's pretty slick - think golang & rust apps, web, gaming servers, etc
lambda_foo 3y ago
Shout out to https://github.com/samuelkarp/runj https://github.com/samuelkarp/runj which aims to provide OCI compatible runtime for FreeBSD jails. We are working on a Jails based sandbox implementation for running OCaml builds on FreeBSD, https://github.com/ocurrent/obuilder https://github.com/ocurrent/obuilder. If runj gets more support then it would be a good alternative to the various iocage/ezjail/etc solutions or the raw Jails implementation we have right now.
- amtamt 3y ago
  Does this OCI compatibility also translate to be able to run kubernetes on FreeBSD?
  - sofixa 3y ago
    Not necessarily, Kubernetes also needs a ton of other things which are Linux kernel specific, like cgroups. If you're just looking for an orchestrator capable of running FreeBSD jails, take a look at Nomad (Disclaimer: I work at HashiCorp, opinions my own, etc.)
    - pjmlp 3y ago
      cgroups are mapped into Jobs on Windows. There could be a similar kind of mapping on FreeBSD.
    - plagiarist 3y ago
      Thank you for the suggestion, I'm interested in checking it out. How does Nomad handle scaling? I'm thinking of a new project, I'd like to have the equivalent of HPA that easily scales to zero on custom metrics and a cluster autoscaler that can easily add heterogenous nodes as necessary. Even better if it can work with MacOS VMs.
      - sofixa 3y ago
        Nomad supports all sorts of architectures and OSes, including M-series CPUs and MacOS. It can also scale to a lot, 2 million containers being tested. [0] In terms of autoscaling there's the Nomad Autoscaler[1][2] that is super flexible, can get metrics from a bunch of sources, do custom queries, and perform actions (everything is via plugins, so even if your current stack isn't already supported, it's easy to add) like add extra nodes or add more instances of jobs. 0 - https://www.hashicorp.com/c2m https://www.hashicorp.com/c2m 1 - https://developer.hashicorp.com/nomad/tools/autoscaling https://developer.hashicorp.com/nomad/tools/autoscaling 2 - https://github.com/hashicorp/nomad-autoscaler https://github.com/hashicorp/nomad-autoscaler
efortis 3y ago
Related: A two-server three-jail network setup (firewall, vlan, tunnels) https://blog.uidrafter.com/freebsd-jails-network-setup https://blog.uidrafter.com/freebsd-jails-network-setup
sim7c00 3y ago
very nice and comprehensive guide, thanks a lot!
- vermaden 3y ago
  Thank You.
codetrotter 3y ago
I highly recommend the book “FreeBSD Mastery: Jails” by Michael W Lucas, in printed form https://bookshop.org/p/books/freebsd-mastery-specialty-filesystems-michael-w-lucas/8134720 https://bookshop.org/p/books/freebsd-mastery-specialty-files... It had everything I needed to start manually setting up jails to my liking. I eventually had to get some additional help online but you come a long, long, long way with that book in your hand.
tambourine_man 3y ago
I have a soft spot for FreeBSD. I hope someday we’ll have a lightweight container solution that works across the BSDs (macOS included), Linux and maybe even Windows. POSIX is all you need.
mikece 3y ago
Why isn't Jails supported on macOS? Seems like this would be a VERY powerful option.
- kitsunesoba 3y ago
  I might be entirely off-base with this, but I think it probably has something to do with the roots of macOS' (or really, NeXTSTEP's) BSD component (BSD 4.3) pre-dating FreeBSD 4.0 (the first version that introduced jails) by 14 years. It does seem like a feature worth porting, though.
- toast0 3y ago
  The OS X kernel is a mashup of things; it's been too long since I looked, but IIRC the tcp code comes from just before FreeBSD added syncache (released in FreeBSD 4.5), so that should be after the release of jails in 4.0. But, Apple would have needed to integrate jails throughout the NeXT/Mach part of the kernel too. And, you'd be missing out on features like VNET that came later (FreeBSD 8), unless Apple reintegrated with upstream, which is pretty rare for them. (They did update the FreeBSD userland, once)
gtewallace 3y ago
Here is a talk from SCaLE 2023 on Cloud Native FreeBSD: https://www.youtube.com/live/ReYon0HLj2k?feature=share&t=18063 https://www.youtube.com/live/ReYon0HLj2k?feature=share&t=180...

226 points by vermaden 3y ago

Also check out systemd-nspawn on linux. Another nice container solution, built in, that seems to be relatively little known.

packetlost 3y ago
I made a very serious attempt at using this awhile back on Arch and it high-key sucks. Like, it's not really meant as a general purpose container runtime solution, it's meant for developing systemd and related software. The ergonomics, particularly around networking are quite bad and it literally requires that you're all in on systemd-networkd and resolved to get any networking at all.
- baggy_trough 3y ago
  I’ve used it as a general purpose container runtime and been very pleased. Ymmv.

wyager 3y ago

Vnet jails are awesome. Totally changed the way I do networking and application hosting. Makes it pretty trivial to do things like set up a jail that can only talk to the internet through a VPN.

wwarner 3y ago

"The FreeBSD host can run any other FreeBSD as long as its not newer then the host system version." Docker does not have this limitation. You can run a container that is based on a newer version of Linux than the host.

kstrauser 3y ago
Depends what you mean by "Linux". You can possibly use a newer userland, but containers share the kernel with the host. That's kind of the whole point of containers: they're just a set of processes that are grouped together and configured with certain restrictions.
chungy 3y ago
They have that limitation, but it doesn't manifest in quite a simple way. If the container's glibc and all other binaries require Linux 6.3, you're not running it on a 5.18 host. FreeBSD is developed as a whole unit. It makes it easier to reason that a 13.2 userland doesn't run on a 12.4 kernel.
- LegionMammal978 3y ago
  > If the container's glibc and all other binaries require Linux 6.3, you're not running it on a 5.18 host. FWIW, it would take quite a long time for glibc itself to start requiring Linux 6.3 on its current targets. Since glibc 2.24 (Aug. 2016), the minimum supported kernel has been at most Linux 3.2 (Jan. 2012), on every target that has existed that long. And that's the smallest that the gap between the glibc and Linux releases has ever gotten (except for new targets). Extrapolating, we'd have to wait for glibc 2.47 (Feb. 2028) to require Linux 6.3 (Apr. 2023), if that gap were to be repeated, something which I don't find particularly likely. That is to say, with the exception of new targets, you'll rarely see a userland that absolutely requires the latest Linux kernel, even possibly if it suggests that kernel for the best support.
  - JdeBP 3y ago
    It's the "all other binaries" part that will catch you. You've completely overlooked that part. For example: Per its own README, systemd requires Linux 3.15 at absolute minimum, recommends Linux 4.15 for baseline functionality to at least work, and specifies Linux 5.7 for full BPF functionality.
ori_b 3y ago
That's incorrect. (Well. Technically it's kind of true. You can run a container based on a newer kernel, it'll just crash when you use any new features)
bandrami 3y ago
Unless that container uses kernel semantics that changed between kernel versions, or a feature that was added in the new version. Which invariably happens. See the RHEL 6/7 upgrade and the changes to syscall.

idatum 3y ago

I have a VNET jail for remote access to my home lab. This takes an extra NIC I wasn't using anyway. It has been solid running on FreeBSD 13.2. This jail allows me to lock things down to minimal access to other services I need remotely, such as an SSH tunnel to Home Assistant. I run my FreeBSD server mainly for ZFS. Adding a jail for remote access adds to the usefulness that hardware.

appleflaxen 3y ago
Can you elaborate on what this looks like?
- idatum 3y ago
  jail.conf: ssh { exec.start = "/bin/sh /etc/rc"; exec.stop = "/bin/sh /etc/rc.shutdown"; exec.clean; mount.devfs; path = "/var/jail/ssh"; host.hostname = "ssh"; vnet; vnet.interface += "em0"; } It's IPv6 only, so this is key in rc.conf: rtsold_enable="YES" OpenBSD PF based router does the rest. IPv6 simplifies things here. PF on FreeBSD isn't the most ideal, sure, but I can limit local access on inet6. For SSH tunnel, it's straight forward.
  - inferiorhuman 3y ago
    FWIW VNET is not inherently IPv6 only. I'm using it in an IPv4 only context just fine.
toast0 3y ago
An extra NIC is fine if you've got slots, but you can also do an if_bridge in the main host, add the real nic and half of an if_epair; then move the other half to the vnet jail. Potentially a non-vnet jail with the ipv4/ipv6 options set may also be an option. I used to use jails with those settings to squeeze multiple environments into a single host at Yahoo.
- yakubin 3y ago
  I use only if_epair and then pf to NAT it to the external world. I’ve only had issues with if_bridge, where I would either be able to make connections between jail and the external world, but not between host and jail or vice versa (between host and jail fine, but not between jail and external world). And it would act really flaky, like getting one ping response, followed by silence. pf is a lot simpler and more reliable.

jonhohle 3y ago

The issue I have with Jails, over something like docker, is that the kernel ABI changes between major releases so jails need to migrate with base systems upgrades. It also means a container registry would need to store container versions per release. I like the isolation for my own bespoke home network, but does it work well in production set up’s?

inferiorhuman 3y ago
Kernels are generally backwards compatible and major versions are explicitly supported for five (5) years. Over in Linux land you'll see builds for the different libcs and it seems to work just fine.
- johncolanduoni 3y ago
  The difference is that the container libc version and kernel version don't have to correspond to each other in any way on Linux (unless your kernel is like 20 years older, in which case it wouldn't support Docker anyway).
  - inferiorhuman 3y ago
    The end result is that you're matching the executable to the libc variant instead of to the kernel. In practice it's not much of a difference, if at all. If you need to use older binaries, install the compatibility shims (or use a base jail based on the older version of FreeBSD). In terms of a support matrix there are two major versions of FreeBSD that are given active support, which is less than over in Linux land. All in all of all the concerns I'd have about using jails in a production environment, dealing with different major versions is not on that list.
    - monocasa 3y ago
      But the half the point of the container is per application libcs/userlands, so the kernel ABI is the only required constant.

Lammy 3y ago

Props for using the base-system jail management tools instead of iocage/ezjail/etc.

wyager 3y ago
IME, the base-system jail stuff is super easy to understand, use, and customize. When you start using "easy" tools on top of that, it rapidly becomes more complicated and opaque for non-trivial stuff. It's awesome that you can do everything you would want with a jail with only 3 config files: rc.conf (for networking and start-up), pf.conf (for firewalling/routing), and jail.conf (for specifying jail params like mount points, network if setup, etc.). Super clean, super easy.
vermaden 3y ago
Thanks.

inferiorhuman 3y ago

VNET jails are great, especially now that they're stable, but I'd strongly recommend using a higher level tool. I've settled on iocage, but there are a few other tools out there that seem to have gained traction. Nobody sets up cgroups by hand, right?

waz0wski 3y ago
A couple of tools which are both working on jail management & packaging - bastille https://bastillebsd.org https://bastillebsd.org - pot https://potluck.honeyguide.net https://potluck.honeyguide.net bastille is great for personal and small setups, pot is aimed at larger scale and meant to be used along with hashicorp nomad+consul
- dilippkumar 3y ago
  What is the state of pot in summer 2023? Is it ready for production?
  - waz0wski 3y ago
    Depends on your workloads and needs. It's not as mature as k8s and there are rough edges but if you're into the Hashicorp ecosystem and can run your workloads in FreeBSD jails it's pretty slick - think golang & rust apps, web, gaming servers, etc

lambda_foo 3y ago

Shout out to https://github.com/samuelkarp/runj https://github.com/samuelkarp/runj which aims to provide OCI compatible runtime for FreeBSD jails. We are working on a Jails based sandbox implementation for running OCaml builds on FreeBSD, https://github.com/ocurrent/obuilder https://github.com/ocurrent/obuilder. If runj gets more support then it would be a good alternative to the various iocage/ezjail/etc solutions or the raw Jails implementation we have right now.

amtamt 3y ago
Does this OCI compatibility also translate to be able to run kubernetes on FreeBSD?
- sofixa 3y ago
  Not necessarily, Kubernetes also needs a ton of other things which are Linux kernel specific, like cgroups. If you're just looking for an orchestrator capable of running FreeBSD jails, take a look at Nomad (Disclaimer: I work at HashiCorp, opinions my own, etc.)
  - pjmlp 3y ago
    cgroups are mapped into Jobs on Windows. There could be a similar kind of mapping on FreeBSD.
  - plagiarist 3y ago
    Thank you for the suggestion, I'm interested in checking it out. How does Nomad handle scaling? I'm thinking of a new project, I'd like to have the equivalent of HPA that easily scales to zero on custom metrics and a cluster autoscaler that can easily add heterogenous nodes as necessary. Even better if it can work with MacOS VMs.
    - sofixa 3y ago
      Nomad supports all sorts of architectures and OSes, including M-series CPUs and MacOS. It can also scale to a lot, 2 million containers being tested. [0] In terms of autoscaling there's the Nomad Autoscaler[1][2] that is super flexible, can get metrics from a bunch of sources, do custom queries, and perform actions (everything is via plugins, so even if your current stack isn't already supported, it's easy to add) like add extra nodes or add more instances of jobs. 0 - https://www.hashicorp.com/c2m https://www.hashicorp.com/c2m 1 - https://developer.hashicorp.com/nomad/tools/autoscaling https://developer.hashicorp.com/nomad/tools/autoscaling 2 - https://github.com/hashicorp/nomad-autoscaler https://github.com/hashicorp/nomad-autoscaler

efortis 3y ago

Related: A two-server three-jail network setup (firewall, vlan, tunnels) https://blog.uidrafter.com/freebsd-jails-network-setup https://blog.uidrafter.com/freebsd-jails-network-setup

sim7c00 3y ago

very nice and comprehensive guide, thanks a lot!

vermaden 3y ago
Thank You.

codetrotter 3y ago

I highly recommend the book “FreeBSD Mastery: Jails” by Michael W Lucas, in printed form https://bookshop.org/p/books/freebsd-mastery-specialty-filesystems-michael-w-lucas/8134720 https://bookshop.org/p/books/freebsd-mastery-specialty-files... It had everything I needed to start manually setting up jails to my liking. I eventually had to get some additional help online but you come a long, long, long way with that book in your hand.

tambourine_man 3y ago

I have a soft spot for FreeBSD. I hope someday we’ll have a lightweight container solution that works across the BSDs (macOS included), Linux and maybe even Windows. POSIX is all you need.

mikece 3y ago

Why isn't Jails supported on macOS? Seems like this would be a VERY powerful option.

kitsunesoba 3y ago
I might be entirely off-base with this, but I think it probably has something to do with the roots of macOS' (or really, NeXTSTEP's) BSD component (BSD 4.3) pre-dating FreeBSD 4.0 (the first version that introduced jails) by 14 years. It does seem like a feature worth porting, though.
toast0 3y ago
The OS X kernel is a mashup of things; it's been too long since I looked, but IIRC the tcp code comes from just before FreeBSD added syncache (released in FreeBSD 4.5), so that should be after the release of jails in 4.0. But, Apple would have needed to integrate jails throughout the NeXT/Mach part of the kernel too. And, you'd be missing out on features like VNET that came later (FreeBSD 8), unless Apple reintegrated with upstream, which is pretty rare for them. (They did update the FreeBSD userland, once)

gtewallace 3y ago

Here is a talk from SCaLE 2023 on Cloud Native FreeBSD: https://www.youtube.com/live/ReYon0HLj2k?feature=share&t=18063 https://www.youtube.com/live/ReYon0HLj2k?feature=share&t=180...