5 ms·
Jails are a general mechanism to run multiple userlands on one kernel. They started with a change root directory, an IPv4 address and no access to anything dang
by crest 10y ago
Jails are a general mechanism to run multiple userlands on one kernel. They started with a change root directory, an IPv4 address and no access to anything dangerous (e.g. /dev/mem). Jails grew a lot features with the biggest bump in FreeBSD 8. Starting with FreeBSD 8 jails can be nested, have multiple IP addresses etc. but they still lacked System V IPC. This changed with FreeBSD 11.
Now the only thing jails can't have is their own IP stack. Jails share the hosts IP stack which improves efficiency and simplifies most deployments, but it prevents them having administrative access to the IP stack. There is an experimental kernel feature (VIMAGE) to jails to run their own instance of the IP stack but there are still some nasty bugs hiding in this code, because nobody thought about how to tear down the IP stack. After all it was initialised once during the boot process and kept running until the power went out.
The largest difference is in the mindset behind jails. Jails are designed as secure operating system level virtualisation. Docker on the other hand is fairly fragile and offers neither secure isolation between containers nor between containers and the host. Jails can contain a complete userland and this a very common setup.
A full FreeBSD userland + some ports/packages to make it useful is about one 1GB. This used to be a lot 15+ years ago when jails where created and the older jail management tools like ezjail reduce the per jail storage requirements with nullfs and unionfs hacks. These days 1GB isn't that much for a simple container and most FreeBSD servers run on ZFS. ZFS offers a much simpler and cleaner way to reduce storage requirements: just clone a snapshot (the template) create a new jail and copy a few config files into the clone. The only problem is that you can't rebase your clone.
Docker is designed around the idea of single purpose containers without stable storage. You can use jails to implement this idea, but FreeBSD jails support more than that. Also all the FreeBSD jail managers I used try to stay out of your network configuration as far as possible and at most configure alias IP addresses on existing interfaces.
Docker is very opinionated software fighting against limitations imposed on it by the Linux kernel. Jails are FreeBSD kernel feature touching multiple parts of the kernel with a minimal userland interface in the FreeBSD base system . Multiple higher level jail managers are available in the FreeBSD ports tree.
There is no reason why you couldn't implement a docker like jail manager and the jetpack projects started doing exactly this. Keep in mind that docket images are the worlds new statically linked binaries for people who can't figure out how to define and reproduce the relevant parts of their development environment in their production environment. Executing existing docker images with their Linux binaries would probably require a massive update to the Linux compatibility layer (a reimplementation the Linux syscall ABI).
- tachion 10y agoActually, Bjoern Zeeb (bz@) did a lot of work around VIMAGE (sometimes also called VNET) in order to track down these bugs and fix them for 11.0-RELEASE. If you haven't, you should give it a try, because it's now the way to use per-jail IP stack.
- crest 10y agoI did and during my tests i found no more panics, but I ran into a bug in the IPFW tables logic. The fix hasn't been backported to FreeBSD 11.0 as an errata. Also the epair pseudo-interface used for connecting IP stacks tops out at about 4Gb/s while the loopback interface can handle >40Gb/s on the same hardware, because it uses just one kernel thread (I don't remember the pps). I hope we'll get VIMAGE into GENERIC for FreeBSD 11.1. He completely refactored the IP stack shutdown logic. Now the layer are shut down from top to bottom instead of the other way around. This avoids most of the nasty locking problems draining the higher levels with their pointers to the lower layer resources (e.g. routes, interfaces) first. The remaining bugs won't be found without a lot more exposure.
- voltagex_ 10y agoI did get panics but I haven't been able to reproduce them. The only thing I can think of is create a jail without an assigned IP, start it, assign one, shut it down and start it up again.
- crest 10y agoThe bug I ran into was https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=212649 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=212649 which turned out to be an other symptom of https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=212576 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=212576.
- shshhdhs 10y agoThis is a fantastic summary, thank you!
- 10y ago