5 ms·

You can use Bhyve and run Docker in a Linux virtual machine. Docker works similarly under MacOS using the Hypervisor Framework. FreeBSD would need to radically

by athms 6y ago

You can use Bhyve and run Docker in a Linux virtual machine. Docker works similarly under MacOS using the Hypervisor Framework. FreeBSD would need to radically change in order to implement Docker natively.

pmlnr 6y ago
FreeBSD had "containers" - jails - a decade before linux: https://docs.freebsd.org/en/books/handbook/jails/ https://docs.freebsd.org/en/books/handbook/jails/ I still don't understand why Docker was not made to run on jails as well.
- jsmith45 6y ago
  I'm not sure the default jail utility is quite as flexible as what Linux namespaces+cgroups can do. It does look like most of what really matters does exist in some form, and I'd guess any important cases that don't exist could be fixed with a few new simple sysctl options. However, BSD's do not guarantee that their userlands will work with a mismatched kernel. Sure, it often does work, hence why jails only give a warning on mismatch rather than refuse to run at all. Also containers are most useful when most containers you want are actually available for your platform. Unless you use the Linux Emulation features, I'd suspect that relatively few containers would be mode available to run on FreeBSD. And the problem with Linux personality systems is that while many programs will work fine with them, there will always be some Linux syscalls that are unimplemented, or have important limitations/differences. So while many programs may work, some will not work. Even if the system call is fully supported, not all use cases will be. For example, not every file system Linux supports will be mountable, and programs could be using loopback mounts that need such support for weird reasons (I'd bet has made an app-bundle system for linux that relies on loopback mounting ext4). There is a reason why Microsoft abandoned the personality like implementation of WSL1 in favor of virtualization for WSL2. Now admittedly, things are not nearly as bad on FreeBSD, since implementing one Unix-like personality in a different Unix is going to be easier and work better than trying to implement it on a decidedly non-Unix kernel. But even so, there will always be some programs (however obscure) that won't work right, while virtualization can largely avoid that. (Albeit with new limitations like not being able to easily access host hardware). Noth of the above is at all a dealbreaker. containerd and docker support windows containers which have many of the above mentioned concerns, and many additional ones like the restrictions on distributing the windows base images. What really needs to happen for jail support for docker is to come up with the FreeBSD specific options for the OCI spec, implement an OCI runtime based on jails, add support for setting the OS specific options in containerd, and implement needed network support in dockerd. (containerd leaves networking setup to its caller, as docker has different opinions than kubernetes for example). The containerd people will almost certainly not object to the needed patches. If I had to guess, the docker maintainer's big concerns over a moby patch will be the overhead of supporting the needed patches (since FreeBSD will rightfully be seen as far more niche than Linux), and that the end-user experience of various docker command lines work more or less as users expect. (I.e. not more different from docker-on-linux than docker-for-windows-containers is). None of this is at all insurmountable.
  - kevans91 6y ago
    > However, BSD's do not guarantee that their userlands will work with a mismatched kernel. Sure, it often does work, hence why jails only give a warning on mismatch rather than refuse to run at all. FWIW, FreeBSD tends to go to pretty great lengths to ensure newer kernel with older userland works. A stock GENERIC kernel comes with COMPAT_FREEBSD* options back to COMPAT_FREEBSD4, and parts of the project's infrastructure tend to explicitly rely on at least supported releases to be functional in a jail on a -CURRENT kernel.
    - jsmith45 6y ago
      Interesting, and good to hear. I know the other BSDs have a very different view of things. I had heard that Linux was the only OS with a stable kernel ABI guarantee. If FreeBSD does too, that certainly is better. Windows for example makes zero guarantees there. There are a lot of syscalls that they won't renumber because some applications have taken a dependency on using them directly, but officially using a syscall without going through NTDLL (or wherever the stub is located for private syscalls) is unsupported. Those syscalls they are not keeping fixed for compatibility can and do change from version to version. Mostly in numbering, but changes to semantics or arguments can happen too. Hence Windows Containers can only run in separate namespaces on a matching kernel version, and the hyper-v isolation (a.k.a. virtualization) option for containers is needed for mismatched versions. So creating an OCI runtime that wraps jails, adding any needed support for FreeBSD specific OCI container settings to containerd, and adding the needed code for things like networking to moby/moby (a.k.a. docker) sounds very feasible to me if some FreeBSD hacker wanted to get proper docker support. Offering Linux Emulation as an experimental option top be able to run more containers would be an added bonus, and should be feasible, since they once had that working with their old unofficial (presumably pre-containerd) builds of docker.

by athms 6y ago

pmlnr 6y ago

FreeBSD had "containers" - jails - a decade before linux: https://docs.freebsd.org/en/books/handbook/jails/ https://docs.freebsd.org/en/books/handbook/jails/ I still don't understand why Docker was not made to run on jails as well.

jsmith45 6y ago
I'm not sure the default jail utility is quite as flexible as what Linux namespaces+cgroups can do. It does look like most of what really matters does exist in some form, and I'd guess any important cases that don't exist could be fixed with a few new simple sysctl options. However, BSD's do not guarantee that their userlands will work with a mismatched kernel. Sure, it often does work, hence why jails only give a warning on mismatch rather than refuse to run at all. Also containers are most useful when most containers you want are actually available for your platform. Unless you use the Linux Emulation features, I'd suspect that relatively few containers would be mode available to run on FreeBSD. And the problem with Linux personality systems is that while many programs will work fine with them, there will always be some Linux syscalls that are unimplemented, or have important limitations/differences. So while many programs may work, some will not work. Even if the system call is fully supported, not all use cases will be. For example, not every file system Linux supports will be mountable, and programs could be using loopback mounts that need such support for weird reasons (I'd bet has made an app-bundle system for linux that relies on loopback mounting ext4). There is a reason why Microsoft abandoned the personality like implementation of WSL1 in favor of virtualization for WSL2. Now admittedly, things are not nearly as bad on FreeBSD, since implementing one Unix-like personality in a different Unix is going to be easier and work better than trying to implement it on a decidedly non-Unix kernel. But even so, there will always be some programs (however obscure) that won't work right, while virtualization can largely avoid that. (Albeit with new limitations like not being able to easily access host hardware). Noth of the above is at all a dealbreaker. containerd and docker support windows containers which have many of the above mentioned concerns, and many additional ones like the restrictions on distributing the windows base images. What really needs to happen for jail support for docker is to come up with the FreeBSD specific options for the OCI spec, implement an OCI runtime based on jails, add support for setting the OS specific options in containerd, and implement needed network support in dockerd. (containerd leaves networking setup to its caller, as docker has different opinions than kubernetes for example). The containerd people will almost certainly not object to the needed patches. If I had to guess, the docker maintainer's big concerns over a moby patch will be the overhead of supporting the needed patches (since FreeBSD will rightfully be seen as far more niche than Linux), and that the end-user experience of various docker command lines work more or less as users expect. (I.e. not more different from docker-on-linux than docker-for-windows-containers is). None of this is at all insurmountable.
- kevans91 6y ago
  > However, BSD's do not guarantee that their userlands will work with a mismatched kernel. Sure, it often does work, hence why jails only give a warning on mismatch rather than refuse to run at all. FWIW, FreeBSD tends to go to pretty great lengths to ensure newer kernel with older userland works. A stock GENERIC kernel comes with COMPAT_FREEBSD* options back to COMPAT_FREEBSD4, and parts of the project's infrastructure tend to explicitly rely on at least supported releases to be functional in a jail on a -CURRENT kernel.
  - jsmith45 6y ago
    Interesting, and good to hear. I know the other BSDs have a very different view of things. I had heard that Linux was the only OS with a stable kernel ABI guarantee. If FreeBSD does too, that certainly is better. Windows for example makes zero guarantees there. There are a lot of syscalls that they won't renumber because some applications have taken a dependency on using them directly, but officially using a syscall without going through NTDLL (or wherever the stub is located for private syscalls) is unsupported. Those syscalls they are not keeping fixed for compatibility can and do change from version to version. Mostly in numbering, but changes to semantics or arguments can happen too. Hence Windows Containers can only run in separate namespaces on a matching kernel version, and the hyper-v isolation (a.k.a. virtualization) option for containers is needed for mismatched versions. So creating an OCI runtime that wraps jails, adding any needed support for FreeBSD specific OCI container settings to containerd, and adding the needed code for things like networking to moby/moby (a.k.a. docker) sounds very feasible to me if some FreeBSD hacker wanted to get proper docker support. Offering Linux Emulation as an experimental option top be able to run more containers would be an added bonus, and should be feasible, since they once had that working with their old unofficial (presumably pre-containerd) builds of docker.