5 ms·

> Can jails be used to run subprocesses in the normal filesystem, but with a different network environment (for example making a given command run its net traff

> Can jails be used to run subprocesses in the normal filesystem, but with a different network environment (for example making a given command run its net traffic through TAP)? Jails are basically like a VM-like/light: it appears like jail is its own system. Not quite as heavy as an actual VM under (e.g.) Qemu with virtualized hardware. A jail can have its own network stack: * https://klarasystems.com/articles/virtualize-your-network-on-freebsd-with-vnet/ https://klarasystems.com/articles/virtualize-your-network-on... You can then create a 'virtual patch cable' between the host and the jail and send one side of the 'cable' to the jail and do routing and stuff on the host: * https://man.freebsd.org/cgi/man.cgi?epair https://man.freebsd.org/cgi/man.cgi?epair But a (sub)process cannot be 'sent' to a jail: jails 'boot up' like a normal system does, and so you'd have your PID 1 run your regular daemon startup. > Can jails be used to limit memory/cpu/IO/network for subprocesses? For threads within a process? Yes: * https://wiki.freebsd.org/JailResourceLimits https://wiki.freebsd.org/JailResourceLimits > Can live processes be moved into or out of a jail? No: jails are VM-like in functionality. > Can jails be used to make a process think it's running as a different user? The jail, being VM-like, would have its own passwd.

the8472 3y ago
Calling "VM-like" is not helpful because containers have also been called that, it also doesn't explain anything. Is it running a separate kernal image? Does it do hypercalls to the host kernel? If not then I wouldn't call it VM-like. The low-level details and fiddly knobs matter, not how it looks like from userspace when all is done. Based on the jail(8) manpage I get the impression that jails are more like all the namespace and cgroup things rolled into a single entity. Which I guess makes it more difficult to use them incorrectly. But it also prevents other uses such as the one that jmillikin has in mind.
- 0x457 3y ago
  It's called VM-like because: - Often boots user land like a VM (i.e. PID 1 is not the process you want to run, but whatever metal or VM freebsd runs as PID 1) - Normally has its own network stack (VIMAGE) - Normally runs a bunch of background services like a regular VM or on-metal would. - Normally has the entire copy of user-land - I've never seen short-lived jails either - you make a base dataset, clone it, but after that, you just run `freebsd-update` like you would in VM. The majority of jail users treat them like lightweight VMs, only difference between VM and Jail from consumer standpoint: shared kernel, access to a subset of host's FS (sharing host FS to VMs in freebsd is not as easy as it is on linux). To be clear, it's possible to use Jails like Docker, there is just no good tooling to do it. People would yell that whatever we have for jails is all we need with foam from their mouths every time you mention it.
  - the8472 3y ago
    That's just saying how people tend to use it, not what the technology does. Namespaces/Cgroups can be used both ways, to isolate a single process (or even partially-isolate individual threads) or entire userland stacks. The question that was asked what the technology can do, not how it's being used. What are the primitives, the atoms its made of.
    - fragmede 3y ago
      > That's just saying how people tend to use it People tend to use words that describe how they use things to describe what they are.
      - righttoolforjob 3y ago
        Well, take a car as an example. The question is not what you use a car for, but rather how the car is built.
        0x457 3y ago
        Car is a carriage-like mode of transportation. The question was "Calling "VM-like" is not helpful because containers have also been called that, it also doesn't explain anything." I've provided examples why "VM-like" is used to describe jails.
    - 0x457 3y ago
      I think you need to look into usage of "-like" suffix.
0x457 3y ago
> But a (sub)process cannot be 'sent' to a jail: jails 'boot up' like a normal system does, and so you'd have your PID 1 run your regular daemon startup. Technically, you don't have to. It's just because jails don't have convenient tooling around them (like docker or podman) it's easier to just boot it up like a normal system.
toast0 3y ago
> But a (sub)process cannot be 'sent' to a jail: jails 'boot up' like a normal system does, and so you'd have your PID 1 run your regular daemon startup. It's an option to run jails with a whole init and everything; then it's like a separate host. But you can also just run stuff in the jail, without a whole everything. I currently run two daemons in a vnet jail to get a separate network for those daemons, but chrooted to /, because they don't need a separate filesystem. At my last job, we ran a TLS termination proxy chrooted to a very limited directory, because we didn't trust OpenSSL after Heartbleed; the chroot had just the executable, ld-elf and the libraries it loaded, unix sockets to communicate with the origin server, and logfiles, most of which was chflags schg.
- tptacek 3y ago
  You can run Linux containers with their own systemd's (obviously, they have their own PID 1). And, of course, that doesn't make them VMs. If you don't trust a component running inside a Jail or a Linux container, you shouldn't be running it there; the kernel attack surface is big. At that point you've adopted the same security model as a phone jailbreaker has (except you don't get to dedicate hardware to the anti-jailbreaking problem the way the phone vendors do).
  - toast0 3y ago
    Well what are you going to do when you have to run something you don't trust? Can't run TLS termination on an isolated host, because then the traffic to the origin goes over the network, which you also don't trust. Don't want to run OpenSSL integrated into the daemon, because OpenSSL is garbage. Couldn't run anything else in the immediate aftermath of Heartbleed, because GnuTLS was worse, and LibreSSL and BoringSSL hadn't been released yet.
    - tptacek 3y ago
      If you really don't trust it, you run it inside a virtual machine. The truth is, you really do trust OpenSSL. You just trust it less than other things. That's fine! Layer controls on top of it. In Linux-land, this is the point where you'd start thinking about things like seccomp-bpf. A truly untrusted workload is, like, a compute job you've accepted from a SAAS customer; it's arms-length multitenancy. You can't share kernels in that situation.
tptacek 3y ago
Jails are not VM-like. The defining attribute of a VM is that it runs its own kernel. FreeBSD has a virtualization system: it's called bhyve.
- heavyset_go 3y ago
  I believe I recall there being a period quite a while ago when things like LXC and OpenVZ were described/marketed as being "VM-like" around the same time that jails were popularized. It feels like jails are still stuck with being described using older terms even after Linux containers and the distinction between them and VMs were mainstreamed.
  - tptacek 3y ago
    I don't know about any of that. LXC containers are also not VM-like.
- fragmede 3y ago
  > The defining attribute of a VM is that it runs its own kernel. of a VM. The difference between "VM" and "VM-like" is the trailing modifier "-like". This means something is similar, but not the same thing. If it looks like a duck, but is made of rubber, we call it a "rubber ducky." If it looks like a duck, is wearing a blue shirt, has a speech impediment, and is not wearing any pants, we call it Donald Duck. Thus the fact that jails, and docker look like VMs because they have their own PID 1, their own file system, their own slice of memory/cpu/IO/network; that is why people are, accurately, using the descriptor "VM-like" to describe them. People understand that docker is not a true virtual machine because it's not running its own kernel. In cases where it matters, pedantry between true VM vs fake VM, or VM-like (for eg security isolation between VM and docker container) is crucial, but most discussions where "VM-like" is used to help people understand OpenVZ, LXC, jails, docker, cgroups, etc aren't focused on the possibility of an RCE in the container escaping the container, but helping people understand what a container even is in the first place. Or to put it another way, strcmp("VM", "VM-like") != 0.
  - insanitybit 3y ago
bear8642 3y ago
>> Can live processes be moved into or out of a jail? >No: jails are VM-like in functionality. Could this be implemented though if moving between sufficiently similar operating systems? Seems like might be useful to imprison a suspicious acting process or to release once seen to be safe.
inkyoto 3y ago
> Jails are basically like a VM-like/light […] With all due respect, they are not. The definition of a «virtual machine» is a settled matter, and the «M» in «VM» is important and is the differentiator: it allows one to run a different operating system kernel on the same host under the auspices of a hardware or a software supervisor (somewhat less of a defining feature). Neither jails nor cgroups possess such a property, and both restrict users to the same host operating system kernel and its version, so none of them are «VM-like/light» irregardless of the semantic interpretation of the «-like» suffix. VM-like functionality is provided by a different OS subsystem in both, Linux and FreeBSD, kernels.

the8472 3y ago

Calling "VM-like" is not helpful because containers have also been called that, it also doesn't explain anything. Is it running a separate kernal image? Does it do hypercalls to the host kernel? If not then I wouldn't call it VM-like. The low-level details and fiddly knobs matter, not how it looks like from userspace when all is done. Based on the jail(8) manpage I get the impression that jails are more like all the namespace and cgroup things rolled into a single entity. Which I guess makes it more difficult to use them incorrectly. But it also prevents other uses such as the one that jmillikin has in mind.

0x457 3y ago
It's called VM-like because: - Often boots user land like a VM (i.e. PID 1 is not the process you want to run, but whatever metal or VM freebsd runs as PID 1) - Normally has its own network stack (VIMAGE) - Normally runs a bunch of background services like a regular VM or on-metal would. - Normally has the entire copy of user-land - I've never seen short-lived jails either - you make a base dataset, clone it, but after that, you just run `freebsd-update` like you would in VM. The majority of jail users treat them like lightweight VMs, only difference between VM and Jail from consumer standpoint: shared kernel, access to a subset of host's FS (sharing host FS to VMs in freebsd is not as easy as it is on linux). To be clear, it's possible to use Jails like Docker, there is just no good tooling to do it. People would yell that whatever we have for jails is all we need with foam from their mouths every time you mention it.
- the8472 3y ago
  That's just saying how people tend to use it, not what the technology does. Namespaces/Cgroups can be used both ways, to isolate a single process (or even partially-isolate individual threads) or entire userland stacks. The question that was asked what the technology can do, not how it's being used. What are the primitives, the atoms its made of.
  - fragmede 3y ago
    > That's just saying how people tend to use it People tend to use words that describe how they use things to describe what they are.
    - righttoolforjob 3y ago
      Well, take a car as an example. The question is not what you use a car for, but rather how the car is built.
      - 0x457 3y ago
        Car is a carriage-like mode of transportation. The question was "Calling "VM-like" is not helpful because containers have also been called that, it also doesn't explain anything." I've provided examples why "VM-like" is used to describe jails.
  - 0x457 3y ago
    I think you need to look into usage of "-like" suffix.

0x457 3y ago

> But a (sub)process cannot be 'sent' to a jail: jails 'boot up' like a normal system does, and so you'd have your PID 1 run your regular daemon startup. Technically, you don't have to. It's just because jails don't have convenient tooling around them (like docker or podman) it's easier to just boot it up like a normal system.

toast0 3y ago

> But a (sub)process cannot be 'sent' to a jail: jails 'boot up' like a normal system does, and so you'd have your PID 1 run your regular daemon startup. It's an option to run jails with a whole init and everything; then it's like a separate host. But you can also just run stuff in the jail, without a whole everything. I currently run two daemons in a vnet jail to get a separate network for those daemons, but chrooted to /, because they don't need a separate filesystem. At my last job, we ran a TLS termination proxy chrooted to a very limited directory, because we didn't trust OpenSSL after Heartbleed; the chroot had just the executable, ld-elf and the libraries it loaded, unix sockets to communicate with the origin server, and logfiles, most of which was chflags schg.

tptacek 3y ago
You can run Linux containers with their own systemd's (obviously, they have their own PID 1). And, of course, that doesn't make them VMs. If you don't trust a component running inside a Jail or a Linux container, you shouldn't be running it there; the kernel attack surface is big. At that point you've adopted the same security model as a phone jailbreaker has (except you don't get to dedicate hardware to the anti-jailbreaking problem the way the phone vendors do).
- toast0 3y ago
  Well what are you going to do when you have to run something you don't trust? Can't run TLS termination on an isolated host, because then the traffic to the origin goes over the network, which you also don't trust. Don't want to run OpenSSL integrated into the daemon, because OpenSSL is garbage. Couldn't run anything else in the immediate aftermath of Heartbleed, because GnuTLS was worse, and LibreSSL and BoringSSL hadn't been released yet.
  - tptacek 3y ago
    If you really don't trust it, you run it inside a virtual machine. The truth is, you really do trust OpenSSL. You just trust it less than other things. That's fine! Layer controls on top of it. In Linux-land, this is the point where you'd start thinking about things like seccomp-bpf. A truly untrusted workload is, like, a compute job you've accepted from a SAAS customer; it's arms-length multitenancy. You can't share kernels in that situation.

tptacek 3y ago

Jails are not VM-like. The defining attribute of a VM is that it runs its own kernel. FreeBSD has a virtualization system: it's called bhyve.

heavyset_go 3y ago
I believe I recall there being a period quite a while ago when things like LXC and OpenVZ were described/marketed as being "VM-like" around the same time that jails were popularized. It feels like jails are still stuck with being described using older terms even after Linux containers and the distinction between them and VMs were mainstreamed.
- tptacek 3y ago
  I don't know about any of that. LXC containers are also not VM-like.
fragmede 3y ago
> The defining attribute of a VM is that it runs its own kernel. of a VM. The difference between "VM" and "VM-like" is the trailing modifier "-like". This means something is similar, but not the same thing. If it looks like a duck, but is made of rubber, we call it a "rubber ducky." If it looks like a duck, is wearing a blue shirt, has a speech impediment, and is not wearing any pants, we call it Donald Duck. Thus the fact that jails, and docker look like VMs because they have their own PID 1, their own file system, their own slice of memory/cpu/IO/network; that is why people are, accurately, using the descriptor "VM-like" to describe them. People understand that docker is not a true virtual machine because it's not running its own kernel. In cases where it matters, pedantry between true VM vs fake VM, or VM-like (for eg security isolation between VM and docker container) is crucial, but most discussions where "VM-like" is used to help people understand OpenVZ, LXC, jails, docker, cgroups, etc aren't focused on the possibility of an RCE in the container escaping the container, but helping people understand what a container even is in the first place. Or to put it another way, strcmp("VM", "VM-like") != 0.
- insanitybit 3y ago

bear8642 3y ago

>> Can live processes be moved into or out of a jail? >No: jails are VM-like in functionality. Could this be implemented though if moving between sufficiently similar operating systems? Seems like might be useful to imprison a suspicious acting process or to release once seen to be safe.

inkyoto 3y ago

> Jails are basically like a VM-like/light […] With all due respect, they are not. The definition of a «virtual machine» is a settled matter, and the «M» in «VM» is important and is the differentiator: it allows one to run a different operating system kernel on the same host under the auspices of a hardware or a software supervisor (somewhat less of a defining feature). Neither jails nor cgroups possess such a property, and both restrict users to the same host operating system kernel and its version, so none of them are «VM-like/light» irregardless of the semantic interpretation of the «-like» suffix. VM-like functionality is provided by a different OS subsystem in both, Linux and FreeBSD, kernels.