6 ms·

Whenever an article about FreeBSD's jails gets posted here, the comments tend to drift toward an argument about jails vs Docker containers. People call them "Li

Whenever an article about FreeBSD's jails gets posted here, the comments tend to drift toward an argument about jails vs Docker containers. People call them "Linux containers", but really they're talking about Docker (or its clones like Podman). This may be me shouting into the void, but I wish there were an article directly comparing jails with namespaces, which is the Linux functionality that Docker uses. I can totally believe that FreeBSD jails provide a better / more unified / more secure experience than Docker, but to extend that into saying "FreeBSD jails are better than Linux namespaces" feels like a category error. Questions I would like to see answered in that article: * Can jails be used to run subprocesses in the normal filesystem, but with a different network environment (for example making a given command run its net traffic through TAP)? * Can jails be used to limit memory/cpu/IO/network for subprocesses? For threads within a process? * Can live processes be moved into or out of a jail? * Can jails be used to make a process think it's running as a different user? I feel like the answer to these questions is generally "no, that's not what jails are for", which is (1) a fine answer given the apparent goal of being a better chroot(), and (2) reinforces that jails and namespaces are addressing different problem domains.

throw0101c 3y ago
> Can jails be used to run subprocesses in the normal filesystem, but with a different network environment (for example making a given command run its net traffic through TAP)? Jails are basically like a VM-like/light: it appears like jail is its own system. Not quite as heavy as an actual VM under (e.g.) Qemu with virtualized hardware. A jail can have its own network stack: * https://klarasystems.com/articles/virtualize-your-network-on-freebsd-with-vnet/ https://klarasystems.com/articles/virtualize-your-network-on... You can then create a 'virtual patch cable' between the host and the jail and send one side of the 'cable' to the jail and do routing and stuff on the host: * https://man.freebsd.org/cgi/man.cgi?epair https://man.freebsd.org/cgi/man.cgi?epair But a (sub)process cannot be 'sent' to a jail: jails 'boot up' like a normal system does, and so you'd have your PID 1 run your regular daemon startup. > Can jails be used to limit memory/cpu/IO/network for subprocesses? For threads within a process? Yes: * https://wiki.freebsd.org/JailResourceLimits https://wiki.freebsd.org/JailResourceLimits > Can live processes be moved into or out of a jail? No: jails are VM-like in functionality. > Can jails be used to make a process think it's running as a different user? The jail, being VM-like, would have its own passwd.
- the8472 3y ago
  Calling "VM-like" is not helpful because containers have also been called that, it also doesn't explain anything. Is it running a separate kernal image? Does it do hypercalls to the host kernel? If not then I wouldn't call it VM-like. The low-level details and fiddly knobs matter, not how it looks like from userspace when all is done. Based on the jail(8) manpage I get the impression that jails are more like all the namespace and cgroup things rolled into a single entity. Which I guess makes it more difficult to use them incorrectly. But it also prevents other uses such as the one that jmillikin has in mind.
  - 0x457 3y ago
    It's called VM-like because: - Often boots user land like a VM (i.e. PID 1 is not the process you want to run, but whatever metal or VM freebsd runs as PID 1) - Normally has its own network stack (VIMAGE) - Normally runs a bunch of background services like a regular VM or on-metal would. - Normally has the entire copy of user-land - I've never seen short-lived jails either - you make a base dataset, clone it, but after that, you just run `freebsd-update` like you would in VM. The majority of jail users treat them like lightweight VMs, only difference between VM and Jail from consumer standpoint: shared kernel, access to a subset of host's FS (sharing host FS to VMs in freebsd is not as easy as it is on linux). To be clear, it's possible to use Jails like Docker, there is just no good tooling to do it. People would yell that whatever we have for jails is all we need with foam from their mouths every time you mention it.
    - the8472 3y ago
      That's just saying how people tend to use it, not what the technology does. Namespaces/Cgroups can be used both ways, to isolate a single process (or even partially-isolate individual threads) or entire userland stacks. The question that was asked what the technology can do, not how it's being used. What are the primitives, the atoms its made of.
      - fragmede 3y ago
toast0 3y ago
> * Can jails be used to run subprocesses in the normal filesystem, but with a different network environment (for example making a given command run its net traffic through TAP)? Yes, you can run a vnet jail chrooted to /; same filesystem as the host, but a separate network system (you have to setup the network for the jail at this point; but I imagine the jail tools help with that). This is a fine use for jails. I currently run a vnet jail chrooted to / to do some crazy network stuff, but I just need a separate network, no other separation. > * Can jails be used to limit memory/cpu/IO/network for subprocesses? For threads within a process? I haven't used it, but it looks like yes/maybe with the rctl subsystem. This allows limits on lots of things, include memory of several types, cpu of a few types, filesystem io in bytes per second and operations per second, number of threads. Valid subjects for resource limits are process, user, loginclass and jail. Based on the manual, this doesn't let you limit network as you wanted, or place limits on threads within a process like you wanted. If you run a vnet jail, you can potentially set networking limits in other ways, but only if you pass virtual interfaces (such as epair, or taptun) to the jail rather than physical interfaces. This is a reasonable use for jails, but you might not need a jail for this? > * Can live processes be moved into or out of a jail? A process can move itself into a jail, but only if it's superuser. This is generally used administratively. Jailed processes can not be moved out of the jail, afaik; when the jail is destroyed, all processes within are killed. This is not within the design scope of jails. > * Can jails be used to make a process think it's running as a different user? I don't think so, but I'd use something with LD_PRELOAD to override getuid/geteuid for this; but I think I must not understand the question. You can certainly have entirely different users inside jail vs on the host?
heavyset_go 3y ago
Hate to nitpick, but a comparison to bare namespaces wouldn't be apt, either, as Docker et al use more features than just namespaces in their container implementations, like control groups, union/overlay/snapshot file systems, nftables, etc. For several of those questions about jails, the answer would be the same if you were just evaluating the questions against namespaces by itself.
assimpleaspossi 3y ago
The article is about how to set up jails on FreeBSD. The article has nothing to do with anything you are talking or asking about.
vermaden 3y ago
First of all - the Namespaces and Cgroups on Linux are not about security. They are about features. FreeBSD Jails provide both secure isolation from the host and also provide the separated network namespace and resources with rctl(8) when needed. To have Linux containers separated and secured you need additional layer for that - like SELinux or AppArmor. Only then Docker/Podman/other/... Linux containers are isolated and secure. Regards.
- icedchai 3y ago
  You're telling those namespace / cgroup features have nothing to do with security? Not in any way, shape, or form? I'm a little skeptical.
  - jmillikin 3y ago
    Namespaces and cgroups provide resource accounting and some limited isolation between trusted workloads. They're not considered a sandbox or security boundary because the processes have full access to the Linux kernel APIs, which are not well-hardened. To provide a security boundary between Linux processes, it's currently considered best practice to use something like Firecracker or gVisor.
    - icedchai 3y ago
      I agree containers are more for ease of deployment, not security. But security is about shades of gray… limited isolation is one of those.
- user6723 3y ago
  Underrated post.
craftkiller 3y ago
> Can live processes be moved into or out of a jail? That might be possible on DragonflyBSD. They have support for freezing a running process to disk and restoring the process again (potentially on another machine) [1], and they have support for jails, so I don't see why you wouldn't be able to freeze a running process on the host and restoring it in the jail or vice versa. [1] https://www.dragonflybsd.org/features/#index9h2 https://www.dragonflybsd.org/features/#index9h2

throw0101c 3y ago

> Can jails be used to run subprocesses in the normal filesystem, but with a different network environment (for example making a given command run its net traffic through TAP)? Jails are basically like a VM-like/light: it appears like jail is its own system. Not quite as heavy as an actual VM under (e.g.) Qemu with virtualized hardware. A jail can have its own network stack: * https://klarasystems.com/articles/virtualize-your-network-on-freebsd-with-vnet/ https://klarasystems.com/articles/virtualize-your-network-on... You can then create a 'virtual patch cable' between the host and the jail and send one side of the 'cable' to the jail and do routing and stuff on the host: * https://man.freebsd.org/cgi/man.cgi?epair https://man.freebsd.org/cgi/man.cgi?epair But a (sub)process cannot be 'sent' to a jail: jails 'boot up' like a normal system does, and so you'd have your PID 1 run your regular daemon startup. > Can jails be used to limit memory/cpu/IO/network for subprocesses? For threads within a process? Yes: * https://wiki.freebsd.org/JailResourceLimits https://wiki.freebsd.org/JailResourceLimits > Can live processes be moved into or out of a jail? No: jails are VM-like in functionality. > Can jails be used to make a process think it's running as a different user? The jail, being VM-like, would have its own passwd.

the8472 3y ago
Calling "VM-like" is not helpful because containers have also been called that, it also doesn't explain anything. Is it running a separate kernal image? Does it do hypercalls to the host kernel? If not then I wouldn't call it VM-like. The low-level details and fiddly knobs matter, not how it looks like from userspace when all is done. Based on the jail(8) manpage I get the impression that jails are more like all the namespace and cgroup things rolled into a single entity. Which I guess makes it more difficult to use them incorrectly. But it also prevents other uses such as the one that jmillikin has in mind.
- 0x457 3y ago
  It's called VM-like because: - Often boots user land like a VM (i.e. PID 1 is not the process you want to run, but whatever metal or VM freebsd runs as PID 1) - Normally has its own network stack (VIMAGE) - Normally runs a bunch of background services like a regular VM or on-metal would. - Normally has the entire copy of user-land - I've never seen short-lived jails either - you make a base dataset, clone it, but after that, you just run `freebsd-update` like you would in VM. The majority of jail users treat them like lightweight VMs, only difference between VM and Jail from consumer standpoint: shared kernel, access to a subset of host's FS (sharing host FS to VMs in freebsd is not as easy as it is on linux). To be clear, it's possible to use Jails like Docker, there is just no good tooling to do it. People would yell that whatever we have for jails is all we need with foam from their mouths every time you mention it.
  - the8472 3y ago
    That's just saying how people tend to use it, not what the technology does. Namespaces/Cgroups can be used both ways, to isolate a single process (or even partially-isolate individual threads) or entire userland stacks. The question that was asked what the technology can do, not how it's being used. What are the primitives, the atoms its made of.
    - fragmede 3y ago

toast0 3y ago

> * Can jails be used to run subprocesses in the normal filesystem, but with a different network environment (for example making a given command run its net traffic through TAP)? Yes, you can run a vnet jail chrooted to /; same filesystem as the host, but a separate network system (you have to setup the network for the jail at this point; but I imagine the jail tools help with that). This is a fine use for jails. I currently run a vnet jail chrooted to / to do some crazy network stuff, but I just need a separate network, no other separation. > * Can jails be used to limit memory/cpu/IO/network for subprocesses? For threads within a process? I haven't used it, but it looks like yes/maybe with the rctl subsystem. This allows limits on lots of things, include memory of several types, cpu of a few types, filesystem io in bytes per second and operations per second, number of threads. Valid subjects for resource limits are process, user, loginclass and jail. Based on the manual, this doesn't let you limit network as you wanted, or place limits on threads within a process like you wanted. If you run a vnet jail, you can potentially set networking limits in other ways, but only if you pass virtual interfaces (such as epair, or taptun) to the jail rather than physical interfaces. This is a reasonable use for jails, but you might not need a jail for this? > * Can live processes be moved into or out of a jail? A process can move itself into a jail, but only if it's superuser. This is generally used administratively. Jailed processes can not be moved out of the jail, afaik; when the jail is destroyed, all processes within are killed. This is not within the design scope of jails. > * Can jails be used to make a process think it's running as a different user? I don't think so, but I'd use something with LD_PRELOAD to override getuid/geteuid for this; but I think I must not understand the question. You can certainly have entirely different users inside jail vs on the host?

heavyset_go 3y ago

Hate to nitpick, but a comparison to bare namespaces wouldn't be apt, either, as Docker et al use more features than just namespaces in their container implementations, like control groups, union/overlay/snapshot file systems, nftables, etc. For several of those questions about jails, the answer would be the same if you were just evaluating the questions against namespaces by itself.

assimpleaspossi 3y ago

The article is about how to set up jails on FreeBSD. The article has nothing to do with anything you are talking or asking about.

vermaden 3y ago

First of all - the Namespaces and Cgroups on Linux are not about security. They are about features. FreeBSD Jails provide both secure isolation from the host and also provide the separated network namespace and resources with rctl(8) when needed. To have Linux containers separated and secured you need additional layer for that - like SELinux or AppArmor. Only then Docker/Podman/other/... Linux containers are isolated and secure. Regards.

icedchai 3y ago
You're telling those namespace / cgroup features have nothing to do with security? Not in any way, shape, or form? I'm a little skeptical.
- jmillikin 3y ago
  Namespaces and cgroups provide resource accounting and some limited isolation between trusted workloads. They're not considered a sandbox or security boundary because the processes have full access to the Linux kernel APIs, which are not well-hardened. To provide a security boundary between Linux processes, it's currently considered best practice to use something like Firecracker or gVisor.
  - icedchai 3y ago
    I agree containers are more for ease of deployment, not security. But security is about shades of gray… limited isolation is one of those.
user6723 3y ago
Underrated post.

craftkiller 3y ago

> Can live processes be moved into or out of a jail? That might be possible on DragonflyBSD. They have support for freezing a running process to disk and restoring the process again (potentially on another machine) [1], and they have support for jails, so I don't see why you wouldn't be able to freeze a running process on the host and restoring it in the jail or vice versa. [1] https://www.dragonflybsd.org/features/#index9h2 https://www.dragonflybsd.org/features/#index9h2