8 ms·

Jails on FreeBSD

143 points by rodrigo975 3y ago

jmclnx 3y ago
Very nice I used jails on FreeBSD and nothing in Linux comes close. Yes, it is not a pointy-clicky setup like Linux likes to do. But IMHO Jails are far more secure, in a way: you get what you 'pay' for.
- betaby 3y ago
  Please provide technical details of `Jails are far more secure`.
  - throw0101c 3y ago
    Multiple CVEs for escaping? E.g.: * https://blog.aquasec.com/cve-2022-0185-linux-kernel-container-escape-in-kubernetes https://blog.aquasec.com/cve-2022-0185-linux-kernel-containe... * https://securitylabs.datadoghq.com/articles/dirty-pipe-container-escape-poc/ https://securitylabs.datadoghq.com/articles/dirty-pipe-conta... * https://snyk.io/learn/docker-security/top-5-vulnerabilities/ https://snyk.io/learn/docker-security/top-5-vulnerabilities/ * https://www.container-security.site/attackers/container_breakout_vulnerabilities.html https://www.container-security.site/attackers/container_brea... The only escaping of jails that I've heard of in the last ~20 years is one not in the jails code, but tunnelling out through devfs: * https://www.freebsd.org/security/advisories/FreeBSD-SA-14:07.devfs.asc https://www.freebsd.org/security/advisories/FreeBSD-SA-14:07...
    - betaby 3y ago
      No? Because many of them Docker specific. Neither I mentioned Docker anywhere, not Docker is a good example of software with a good security record. Another link lumps together k8s, runc and so on. I specifically mentioned systemd-nspaw in the beginning as contender to jails, not other solutions.
    - tptacek 3y ago
      Misleading: the most important vulnerabilities that break shared-kernel isolation systems like Jails, Zones, and containers are kernel LPEs, and they aren’t reported as “escapes”.
- gtirloni 3y ago
  What makes Jails so far ahead of containers in your opinion?
  - doublerabbit 3y ago
    Not op, but for me: You can setup Linux in a jail. You can nest bhyve virtual machines in a jail. You can assign individual NICs to a jail. You can encrypt with ZFS a jail. You can run browsers in a jail with full set of features.
    - betaby 3y ago
      All of those available on Linux and even more, see https://www.man7.org/linux/man-pages/man1/systemd-nspawn.1.html https://www.man7.org/linux/man-pages/man1/systemd-nspawn.1.h...
      - doublerabbit 3y ago
        I don't see the guide to run FreeBSD in a container from that man page. And what about if my Linux Distro doesn't use system-md, like Slackware.
        betaby 3y ago
        > And what about if my Linux Distro doesn't use system-md, like Slackware. It's a choice of Slackware not use systemd infra. However on Linuxes you have more that one choice to run containers, your next option would be LXC/LDX https://docs.slackware.com/howtos:misc:lxc https://docs.slackware.com/howtos:misc:lxc
        doublerabbit 3y ago
        Sure and my choice not to use it too. Well aware of LXC too. I'm not claiming Linux can't. I was stating why. I know Linux can do what FreeBSD can do, but BSD does it better. However both are pretty much mimicked concept from Solaris Zones
- irusensei 3y ago
  I don’t know about the low level stuff but LXC is pretty much the same end result as FreeBSD jails. At least for my very basic needs that is.
- johnisgood 3y ago
  What about `firejail` on Linux? Is it not much more secure than FreeBSD Jails which is just based on chroot?
  - bell-cot 3y ago
    FreeBSD Jails are conceptually just based on chroot. Their actual design and implementation are a few orders of magnitude more secure.
    - mardifoufs 3y ago
      A few order of magnitudes? Can you be more specific? I'm genuinely wondering what you could be referring to that I've missed.
      - tptacek 3y ago
        chroot more or less isn't secure at all. All they're really saying is that Jails aren't built on top of chroot, but rather are a ground-up reimplementation motivated by improving chroot.
        mardifoufs 3y ago
        Ah! For some reason I thought jails were built on something similar to chroot at the kernel level, meaning that it also came with the same security caveats. (to be more accurate, I knew that it was not "on top" in the same way as Linux containers are built on top of chroot. )
      - bell-cot 3y ago
        https://en.wikipedia.org/wiki/Operating_system-level_virtualization#Implementations https://en.wikipedia.org/wiki/Operating_system-level_virtual... or https://en.wikipedia.org/wiki/Freebsd_jail https://en.wikipedia.org/wiki/Freebsd_jail
        NexRebular 3y ago
- tptacek 3y ago
  Jails are not, in fact, far more secure than Linux containers. Arguably they're a better, more coherent design, but both are shared-kernel isolation schemes, and the high-order bit of their security is exactly the same. The best argument you'll come up with is that FreeBSD kernel LPEs are rarer than those of Linux, but now we're just back to the standard Linux vs. FreeBSD security debate, which never goes anywhere.
  - zie 3y ago
    I agree with all of this, with perhaps one tiny addendum. Defaults matter, and last I checked Docker/container defaults are miserable for security. I'm guessing if you randomly inspected production container vs jails out in the world, way more container solutions would have giant holes in them compared to jails.
    - cpuguy83 3y ago
      Defaults on Docker specifically are pretty locked down. Even if if the container process has the root uid it is not able to do most things normal root can do, and is pretty much limited to being able to access root owned files and binding to privileged ports (which in a container doesn't tend to matter).
      - yjftsjthsd-h 3y ago
        > Defaults on Docker specifically are pretty locked down. That is strongly dependent on your threat model. The default docker configuration completely bypasses the firewall, making it trivial for containers to be exposed to the open internet with no way for admins to prevent it[0]. Likewise, I hesitate to call docker's default of running as root safe since it means anyone with access to the docker socket immediately has root on the host. [0] It is quite easy for someone even slightly inexperienced to accidentally write, say `-p 1234:1234` instead of `-p 127.0.0.1:1234:1234` and thereby cause a security incident or near-miss; ask me how I know.
        cpuguy83 3y ago
1over137 3y ago
Instruction are for FreeBSD 12, I wonder if it's still current for 13 and 14...?
- BanazirGalbasi 3y ago
  In my limited experience with BSD over the years (I tinker with it every year or so), a lot of the core experience remains the same. Additions to rc.conf should keep working, as well as configuration files for things like jails.
- UI_at_80x24 3y ago
  Not really. Specifically the suggestion to use lo# instead of vnet.
st3fan 3y ago
I upvoted this because I have a lot of love for FreeBSD and I think Jails are great. However, I really wish the FreeBSD folks would educate themselves a bit more about what is actually available on Linux and how those options compare to FreeBSD Jails.
- doublerabbit 3y ago
  And vice versa. I really wish the Linux folk would educate themselves a bit more about what is actually available on FreeBSD.
  - tptacek 3y ago
    Why would they? What difference would it make to them?
    - adr1an 3y ago
      Who are you referring to? Linux or FreeBsd people? I guess both should look to each others'. But it's common to see this in FLOSS, right? Goes beyond namespaces/ isolation.
      - tptacek 3y ago
        I don't mean to sound snide, I'm just saying: if your stack is Linux and the container ecosystem (for instance, if your stack is K8s) what would be in it for you to investigate Jails?
        mnd999 3y ago
        If your stack is K8s and you actually need K8s then none. Most people using K8s don’t actually need it and are practicing resume driven development.
        tptacek 3y ago
        You get that resume-driven development is a real thing, right? It's not irrational. Improving your future prospects is a very legitimate reason to invest in K8s development. I'm speaking against my own interests here; I don't like K8s.
thehigherlife 3y ago
Since this is gaining some traction. I'm curious if you are using jails and a jail manager, what are you're using now? I'm still using IOCAGE (Version 1.2) on freeBSD 13, but it seems like its not the best way to do it anymore? Documentation is all over the place on what is the best way to manage jails now.
- kevans91 3y ago
  iocage as well... I've been trying to sucker someone into writing a new jail manager in flua, since we have that in base. It hasn't worked yet.
- UI_at_80x24 3y ago
  I'm in the same boat. Started with ezjail, switched to iocage, now thinking about bastille or roll-my-own.
- kitsunesoba 3y ago
  Still on ezjail myself. Set my NAS/home server up several years ago with a blog post tutorial centered around ezjail and have just carried that forward. Eventually will need to do a fresh reinstall though and would be interested to know what's considered best practice these days.
- soupbowl 3y ago
  I use BastilleBSD, it has a simple template system which I really like and decent docs. https://github.com/BastilleBSD/bastille https://github.com/BastilleBSD/bastille Bastille also has a sister project 'rocinante' which allows you to use Bastille templates on the host. I converted my ansible scripts to bastille templates and it works a lot better for *ME*. I found I spend more time updating ansible scripts whenever I needed to use them, it costed more time then just using a setup.sh script, which rocianate basically is. https://github.com/BastilleBSD/rocinante https://github.com/BastilleBSD/rocinante Another new kid on the block for jails is AppJail, it has some interesting features. I have not played with it enough to say how stable it is. https://github.com/DtxdF/AppJail https://github.com/DtxdF/AppJail
soupbowl 3y ago
There are a lot of jail tools these days, it would be nice if we posted more modern FreeBSD guides. Myself I use BastilleBSD heavily for personal and small business use. https://github.com/BastilleBSD/bastille https://github.com/BastilleBSD/bastille
codetrotter 3y ago
Let me chime in and say, Michael W Lucas has an awesome book about FreeBSD Jails. I bought it recently and I read the whole thing and it helped me a lot. I still had to figure some things out by myself because the book is for a slightly older version of FreeBSD. But it is an awesome book. I use only the tools included in base system for setting up my jails. No “ezjail” or anything. If you read his whole book you will see how it might be the correct choice to just do it yourself. Depending on what you want to do etc. For me I am definitely much better off having set it up myself with the help of mwl’s book. https://www.freebsdmall.com/cgi-bin/fm/bsdmjails https://www.freebsdmall.com/cgi-bin/fm/bsdmjails Buy the physical copy of the book. PS: Use vnet interfaces for most of your jails.
- chunk_waffle 3y ago
  All of Michael W Lucas' books are great, they're so easy to digest.
mikece 3y ago
Is there a "best article" out there comparing BSD Jails to Docker and the strengths and weaknesses of choosing one or the other? (I'm also curious if BSD Jails are the same thing as Solaris Zones but with a different name or if there is significant nuance making them different).
- tptacek 3y ago
  Use whichever one you're more comfortable with and that suits the rest of your stack best. There's nothing Jails are going to allow you to do that Linux containers can't do. All the shared-kernel workload isolation schemes are comparably safe.
- whalesalad 3y ago
  https://blog.jessfraz.com/post/containers-zones-jails-vms/ https://blog.jessfraz.com/post/containers-zones-jails-vms/
- paulddraper 3y ago
  For starters, Docker runs as a daemon.
  - __turbobrew__ 3y ago
    For starters starters, Docker is a few layers up in the stack from jails. You would be better off comparing runc to jails or just cgroups+namespaces to jails.
    - paulddraper 3y ago
      No. Docker and runc are very similar. Docker run Docker containers. runc runs OCI containers. And jails operate at a similar level, though without an image format.
      - __turbobrew__ 3y ago
        Right, so jails do not have images or any of the tooling around building and distributing images which is the vast majority of what makes up docker. This is why cgroups+namespaces are much closer to jails than docker is. There are systems which build upon jails such as iocage and ezjail which are more similar to docker.
jmillikin 3y ago
Whenever an article about FreeBSD's jails gets posted here, the comments tend to drift toward an argument about jails vs Docker containers. People call them "Linux containers", but really they're talking about Docker (or its clones like Podman). This may be me shouting into the void, but I wish there were an article directly comparing jails with namespaces, which is the Linux functionality that Docker uses. I can totally believe that FreeBSD jails provide a better / more unified / more secure experience than Docker, but to extend that into saying "FreeBSD jails are better than Linux namespaces" feels like a category error. Questions I would like to see answered in that article: * Can jails be used to run subprocesses in the normal filesystem, but with a different network environment (for example making a given command run its net traffic through TAP)? * Can jails be used to limit memory/cpu/IO/network for subprocesses? For threads within a process? * Can live processes be moved into or out of a jail? * Can jails be used to make a process think it's running as a different user? I feel like the answer to these questions is generally "no, that's not what jails are for", which is (1) a fine answer given the apparent goal of being a better chroot(), and (2) reinforces that jails and namespaces are addressing different problem domains.
- throw0101c 3y ago
  > Can jails be used to run subprocesses in the normal filesystem, but with a different network environment (for example making a given command run its net traffic through TAP)? Jails are basically like a VM-like/light: it appears like jail is its own system. Not quite as heavy as an actual VM under (e.g.) Qemu with virtualized hardware. A jail can have its own network stack: * https://klarasystems.com/articles/virtualize-your-network-on-freebsd-with-vnet/ https://klarasystems.com/articles/virtualize-your-network-on... You can then create a 'virtual patch cable' between the host and the jail and send one side of the 'cable' to the jail and do routing and stuff on the host: * https://man.freebsd.org/cgi/man.cgi?epair https://man.freebsd.org/cgi/man.cgi?epair But a (sub)process cannot be 'sent' to a jail: jails 'boot up' like a normal system does, and so you'd have your PID 1 run your regular daemon startup. > Can jails be used to limit memory/cpu/IO/network for subprocesses? For threads within a process? Yes: * https://wiki.freebsd.org/JailResourceLimits https://wiki.freebsd.org/JailResourceLimits > Can live processes be moved into or out of a jail? No: jails are VM-like in functionality. > Can jails be used to make a process think it's running as a different user? The jail, being VM-like, would have its own passwd.
alberth 3y ago
Off topic: I miss the days when web pages looked like this link. High info density, consistent look, “responsive” without being responsive.
layer8 3y ago
In the aughts (2000s) I rented a "managed" VPS that was based on FreeBSD jails and an overlay filesystem. It was great in that you didn't have to maintain most of the software yourself (e.g. security updates) because it was maintained by the hoster on the base filesystem layer, while at the same time you were still root on your own filesystem overlay, where you could add or modify any file from the base filesystem. If you messed up somehow, you could simply revert to the base filesystem version of the respective files. Furthermore only the files on your overlay counted against your filesystem quota. Unfortunately the hoster discontinued that offering after a couple of years.
- porker 3y ago
  Verio VPS FTW! That was the first proper hosting I had.
- codetrotter 3y ago
  > Unfortunately the hoster discontinued that offering after a couple of years. I’m planning on starting a FreeBSD based hosting service, but I am not sure if anyone actually wants what I have in mind. The thing is that I have some couple of very specific ideas in mind. I will offer a small amount of storage, and not general internet connectivity. Users will have 25GB space and the idea is that they will be able to connect over Wireguard, but they cannot make outbound connections from the host. It’ll be like a /home/user in the cloud. So you can keep some files there and connect from wherever in the world and use the cli tools we all know and love. But idk if there is an actual market for that or not. The selling point is that I will be focusing on the storage of that data. ZFS with redundancy and offsite backups – the works. That’s why the amount of data should be low. Only keep the most important files there. Also I’m gonna accept payments in Bitcoin only, and people will have to sign up for many years upfront. I think maybe the number of people that want this can be counted on one hand heh
  - gtirloni 3y ago
    One question you may want to answer is why anyone would use your solution vs something like rsync.net or a small VM in DigitalOcean, etc. You said the selling point would be storage. What would be the differentiator in that area? Would price be it? Just trying to help with question to validate the idea. Best of luck.
jedberg 3y ago
Just one nit, FreeBSD had jails before March of 2000. I know this because I was using them in 1999. And in fact they were merged into CURRENT in early 1999 and STABLE in mid/late 1999. It was merged into RELEASE in March 2000.
- erk__ 3y ago
  They were actually developed a year before, but the person who paid for development had a year of exclusive use. http://freebsd.dk/sagas/jails/ http://freebsd.dk/sagas/jails/
gumballindie 3y ago
Every time I think of freebsd and how cool it was and is I wondering what caused linux to overtake it. I like linux as well, but I just don't understand the mechanics behind freebsd not gaining traction.
- jbverschoor 3y ago
  For me it was Java. I think there was something with mmap(). I ran FreeBSD servers with jails around ‘00. They worked fine. Not for me, but in hindsight, maybe the FreeBSD daemon mascotte had something to do with it. But around that time bea weblogic and ibm websphere also became popular, and I don’t think Java was officially supported. It was also a period where people tended to run oracle, but I’m not sure if that tab on Linux or something else. I think it was some custom Linux “unbreakable oracle”
- chungy 3y ago
  AT&T v BSDi had a large hand in that outcome of fates. It put the legality of BSD code in all its places into doubt for many years. Ironically, just a few years later, SCO thought "hey that was a good idea" and did it to Linux too.
Detrytus 3y ago
Offtopic, but isn't it funny that the system that calls itself FreeBSD has a concept of "jails"? Couldn't they come up with better name? :)

143 points by rodrigo975 3y ago

Very nice I used jails on FreeBSD and nothing in Linux comes close. Yes, it is not a pointy-clicky setup like Linux likes to do. But IMHO Jails are far more secure, in a way: you get what you 'pay' for.

betaby 3y ago
Please provide technical details of `Jails are far more secure`.
- throw0101c 3y ago
  Multiple CVEs for escaping? E.g.: * https://blog.aquasec.com/cve-2022-0185-linux-kernel-container-escape-in-kubernetes https://blog.aquasec.com/cve-2022-0185-linux-kernel-containe... * https://securitylabs.datadoghq.com/articles/dirty-pipe-container-escape-poc/ https://securitylabs.datadoghq.com/articles/dirty-pipe-conta... * https://snyk.io/learn/docker-security/top-5-vulnerabilities/ https://snyk.io/learn/docker-security/top-5-vulnerabilities/ * https://www.container-security.site/attackers/container_breakout_vulnerabilities.html https://www.container-security.site/attackers/container_brea... The only escaping of jails that I've heard of in the last ~20 years is one not in the jails code, but tunnelling out through devfs: * https://www.freebsd.org/security/advisories/FreeBSD-SA-14:07.devfs.asc https://www.freebsd.org/security/advisories/FreeBSD-SA-14:07...
  - betaby 3y ago
    No? Because many of them Docker specific. Neither I mentioned Docker anywhere, not Docker is a good example of software with a good security record. Another link lumps together k8s, runc and so on. I specifically mentioned systemd-nspaw in the beginning as contender to jails, not other solutions.
  - tptacek 3y ago
    Misleading: the most important vulnerabilities that break shared-kernel isolation systems like Jails, Zones, and containers are kernel LPEs, and they aren’t reported as “escapes”.
gtirloni 3y ago
What makes Jails so far ahead of containers in your opinion?
- doublerabbit 3y ago
  Not op, but for me: You can setup Linux in a jail. You can nest bhyve virtual machines in a jail. You can assign individual NICs to a jail. You can encrypt with ZFS a jail. You can run browsers in a jail with full set of features.
  - betaby 3y ago
    All of those available on Linux and even more, see https://www.man7.org/linux/man-pages/man1/systemd-nspawn.1.html https://www.man7.org/linux/man-pages/man1/systemd-nspawn.1.h...
    - doublerabbit 3y ago
      I don't see the guide to run FreeBSD in a container from that man page. And what about if my Linux Distro doesn't use system-md, like Slackware.
      - betaby 3y ago
        > And what about if my Linux Distro doesn't use system-md, like Slackware. It's a choice of Slackware not use systemd infra. However on Linuxes you have more that one choice to run containers, your next option would be LXC/LDX https://docs.slackware.com/howtos:misc:lxc https://docs.slackware.com/howtos:misc:lxc
        doublerabbit 3y ago
        Sure and my choice not to use it too. Well aware of LXC too. I'm not claiming Linux can't. I was stating why. I know Linux can do what FreeBSD can do, but BSD does it better. However both are pretty much mimicked concept from Solaris Zones
irusensei 3y ago
I don’t know about the low level stuff but LXC is pretty much the same end result as FreeBSD jails. At least for my very basic needs that is.
johnisgood 3y ago
What about `firejail` on Linux? Is it not much more secure than FreeBSD Jails which is just based on chroot?
- bell-cot 3y ago
  FreeBSD Jails are conceptually just based on chroot. Their actual design and implementation are a few orders of magnitude more secure.
  - mardifoufs 3y ago
    A few order of magnitudes? Can you be more specific? I'm genuinely wondering what you could be referring to that I've missed.
    - tptacek 3y ago
      chroot more or less isn't secure at all. All they're really saying is that Jails aren't built on top of chroot, but rather are a ground-up reimplementation motivated by improving chroot.
      - mardifoufs 3y ago
        Ah! For some reason I thought jails were built on something similar to chroot at the kernel level, meaning that it also came with the same security caveats. (to be more accurate, I knew that it was not "on top" in the same way as Linux containers are built on top of chroot. )
    - bell-cot 3y ago
      https://en.wikipedia.org/wiki/Operating_system-level_virtualization#Implementations https://en.wikipedia.org/wiki/Operating_system-level_virtual... or https://en.wikipedia.org/wiki/Freebsd_jail https://en.wikipedia.org/wiki/Freebsd_jail
      - NexRebular 3y ago
tptacek 3y ago
Jails are not, in fact, far more secure than Linux containers. Arguably they're a better, more coherent design, but both are shared-kernel isolation schemes, and the high-order bit of their security is exactly the same. The best argument you'll come up with is that FreeBSD kernel LPEs are rarer than those of Linux, but now we're just back to the standard Linux vs. FreeBSD security debate, which never goes anywhere.
- zie 3y ago
  I agree with all of this, with perhaps one tiny addendum. Defaults matter, and last I checked Docker/container defaults are miserable for security. I'm guessing if you randomly inspected production container vs jails out in the world, way more container solutions would have giant holes in them compared to jails.
  - cpuguy83 3y ago
    Defaults on Docker specifically are pretty locked down. Even if if the container process has the root uid it is not able to do most things normal root can do, and is pretty much limited to being able to access root owned files and binding to privileged ports (which in a container doesn't tend to matter).
    - yjftsjthsd-h 3y ago
      > Defaults on Docker specifically are pretty locked down. That is strongly dependent on your threat model. The default docker configuration completely bypasses the firewall, making it trivial for containers to be exposed to the open internet with no way for admins to prevent it[0]. Likewise, I hesitate to call docker's default of running as root safe since it means anyone with access to the docker socket immediately has root on the host. [0] It is quite easy for someone even slightly inexperienced to accidentally write, say `-p 1234:1234` instead of `-p 127.0.0.1:1234:1234` and thereby cause a security incident or near-miss; ask me how I know.
      - cpuguy83 3y ago

1over137 3y ago

Instruction are for FreeBSD 12, I wonder if it's still current for 13 and 14...?

BanazirGalbasi 3y ago
In my limited experience with BSD over the years (I tinker with it every year or so), a lot of the core experience remains the same. Additions to rc.conf should keep working, as well as configuration files for things like jails.
UI_at_80x24 3y ago
Not really. Specifically the suggestion to use lo# instead of vnet.

st3fan 3y ago

I upvoted this because I have a lot of love for FreeBSD and I think Jails are great. However, I really wish the FreeBSD folks would educate themselves a bit more about what is actually available on Linux and how those options compare to FreeBSD Jails.

doublerabbit 3y ago
And vice versa. I really wish the Linux folk would educate themselves a bit more about what is actually available on FreeBSD.
- tptacek 3y ago
  Why would they? What difference would it make to them?
  - adr1an 3y ago
    Who are you referring to? Linux or FreeBsd people? I guess both should look to each others'. But it's common to see this in FLOSS, right? Goes beyond namespaces/ isolation.
    - tptacek 3y ago
      I don't mean to sound snide, I'm just saying: if your stack is Linux and the container ecosystem (for instance, if your stack is K8s) what would be in it for you to investigate Jails?
      - mnd999 3y ago
        If your stack is K8s and you actually need K8s then none. Most people using K8s don’t actually need it and are practicing resume driven development.
        tptacek 3y ago
        You get that resume-driven development is a real thing, right? It's not irrational. Improving your future prospects is a very legitimate reason to invest in K8s development. I'm speaking against my own interests here; I don't like K8s.

thehigherlife 3y ago

Since this is gaining some traction. I'm curious if you are using jails and a jail manager, what are you're using now? I'm still using IOCAGE (Version 1.2) on freeBSD 13, but it seems like its not the best way to do it anymore? Documentation is all over the place on what is the best way to manage jails now.

kevans91 3y ago
iocage as well... I've been trying to sucker someone into writing a new jail manager in flua, since we have that in base. It hasn't worked yet.
UI_at_80x24 3y ago
I'm in the same boat. Started with ezjail, switched to iocage, now thinking about bastille or roll-my-own.
kitsunesoba 3y ago
Still on ezjail myself. Set my NAS/home server up several years ago with a blog post tutorial centered around ezjail and have just carried that forward. Eventually will need to do a fresh reinstall though and would be interested to know what's considered best practice these days.
soupbowl 3y ago
I use BastilleBSD, it has a simple template system which I really like and decent docs. https://github.com/BastilleBSD/bastille https://github.com/BastilleBSD/bastille Bastille also has a sister project 'rocinante' which allows you to use Bastille templates on the host. I converted my ansible scripts to bastille templates and it works a lot better for *ME*. I found I spend more time updating ansible scripts whenever I needed to use them, it costed more time then just using a setup.sh script, which rocianate basically is. https://github.com/BastilleBSD/rocinante https://github.com/BastilleBSD/rocinante Another new kid on the block for jails is AppJail, it has some interesting features. I have not played with it enough to say how stable it is. https://github.com/DtxdF/AppJail https://github.com/DtxdF/AppJail

soupbowl 3y ago

There are a lot of jail tools these days, it would be nice if we posted more modern FreeBSD guides. Myself I use BastilleBSD heavily for personal and small business use. https://github.com/BastilleBSD/bastille https://github.com/BastilleBSD/bastille

codetrotter 3y ago

Let me chime in and say, Michael W Lucas has an awesome book about FreeBSD Jails. I bought it recently and I read the whole thing and it helped me a lot. I still had to figure some things out by myself because the book is for a slightly older version of FreeBSD. But it is an awesome book. I use only the tools included in base system for setting up my jails. No “ezjail” or anything. If you read his whole book you will see how it might be the correct choice to just do it yourself. Depending on what you want to do etc. For me I am definitely much better off having set it up myself with the help of mwl’s book. https://www.freebsdmall.com/cgi-bin/fm/bsdmjails https://www.freebsdmall.com/cgi-bin/fm/bsdmjails Buy the physical copy of the book. PS: Use vnet interfaces for most of your jails.

chunk_waffle 3y ago
All of Michael W Lucas' books are great, they're so easy to digest.

mikece 3y ago

Is there a "best article" out there comparing BSD Jails to Docker and the strengths and weaknesses of choosing one or the other? (I'm also curious if BSD Jails are the same thing as Solaris Zones but with a different name or if there is significant nuance making them different).

tptacek 3y ago
Use whichever one you're more comfortable with and that suits the rest of your stack best. There's nothing Jails are going to allow you to do that Linux containers can't do. All the shared-kernel workload isolation schemes are comparably safe.
whalesalad 3y ago
https://blog.jessfraz.com/post/containers-zones-jails-vms/ https://blog.jessfraz.com/post/containers-zones-jails-vms/
paulddraper 3y ago
For starters, Docker runs as a daemon.
- __turbobrew__ 3y ago
  For starters starters, Docker is a few layers up in the stack from jails. You would be better off comparing runc to jails or just cgroups+namespaces to jails.
  - paulddraper 3y ago
    No. Docker and runc are very similar. Docker run Docker containers. runc runs OCI containers. And jails operate at a similar level, though without an image format.
    - __turbobrew__ 3y ago
      Right, so jails do not have images or any of the tooling around building and distributing images which is the vast majority of what makes up docker. This is why cgroups+namespaces are much closer to jails than docker is. There are systems which build upon jails such as iocage and ezjail which are more similar to docker.

jmillikin 3y ago

Whenever an article about FreeBSD's jails gets posted here, the comments tend to drift toward an argument about jails vs Docker containers. People call them "Linux containers", but really they're talking about Docker (or its clones like Podman). This may be me shouting into the void, but I wish there were an article directly comparing jails with namespaces, which is the Linux functionality that Docker uses. I can totally believe that FreeBSD jails provide a better / more unified / more secure experience than Docker, but to extend that into saying "FreeBSD jails are better than Linux namespaces" feels like a category error. Questions I would like to see answered in that article: * Can jails be used to run subprocesses in the normal filesystem, but with a different network environment (for example making a given command run its net traffic through TAP)? * Can jails be used to limit memory/cpu/IO/network for subprocesses? For threads within a process? * Can live processes be moved into or out of a jail? * Can jails be used to make a process think it's running as a different user? I feel like the answer to these questions is generally "no, that's not what jails are for", which is (1) a fine answer given the apparent goal of being a better chroot(), and (2) reinforces that jails and namespaces are addressing different problem domains.

throw0101c 3y ago
> Can jails be used to run subprocesses in the normal filesystem, but with a different network environment (for example making a given command run its net traffic through TAP)? Jails are basically like a VM-like/light: it appears like jail is its own system. Not quite as heavy as an actual VM under (e.g.) Qemu with virtualized hardware. A jail can have its own network stack: * https://klarasystems.com/articles/virtualize-your-network-on-freebsd-with-vnet/ https://klarasystems.com/articles/virtualize-your-network-on... You can then create a 'virtual patch cable' between the host and the jail and send one side of the 'cable' to the jail and do routing and stuff on the host: * https://man.freebsd.org/cgi/man.cgi?epair https://man.freebsd.org/cgi/man.cgi?epair But a (sub)process cannot be 'sent' to a jail: jails 'boot up' like a normal system does, and so you'd have your PID 1 run your regular daemon startup. > Can jails be used to limit memory/cpu/IO/network for subprocesses? For threads within a process? Yes: * https://wiki.freebsd.org/JailResourceLimits https://wiki.freebsd.org/JailResourceLimits > Can live processes be moved into or out of a jail? No: jails are VM-like in functionality. > Can jails be used to make a process think it's running as a different user? The jail, being VM-like, would have its own passwd.

alberth 3y ago

Off topic: I miss the days when web pages looked like this link. High info density, consistent look, “responsive” without being responsive.

layer8 3y ago

In the aughts (2000s) I rented a "managed" VPS that was based on FreeBSD jails and an overlay filesystem. It was great in that you didn't have to maintain most of the software yourself (e.g. security updates) because it was maintained by the hoster on the base filesystem layer, while at the same time you were still root on your own filesystem overlay, where you could add or modify any file from the base filesystem. If you messed up somehow, you could simply revert to the base filesystem version of the respective files. Furthermore only the files on your overlay counted against your filesystem quota. Unfortunately the hoster discontinued that offering after a couple of years.

porker 3y ago
Verio VPS FTW! That was the first proper hosting I had.
codetrotter 3y ago
> Unfortunately the hoster discontinued that offering after a couple of years. I’m planning on starting a FreeBSD based hosting service, but I am not sure if anyone actually wants what I have in mind. The thing is that I have some couple of very specific ideas in mind. I will offer a small amount of storage, and not general internet connectivity. Users will have 25GB space and the idea is that they will be able to connect over Wireguard, but they cannot make outbound connections from the host. It’ll be like a /home/user in the cloud. So you can keep some files there and connect from wherever in the world and use the cli tools we all know and love. But idk if there is an actual market for that or not. The selling point is that I will be focusing on the storage of that data. ZFS with redundancy and offsite backups – the works. That’s why the amount of data should be low. Only keep the most important files there. Also I’m gonna accept payments in Bitcoin only, and people will have to sign up for many years upfront. I think maybe the number of people that want this can be counted on one hand heh
- gtirloni 3y ago
  One question you may want to answer is why anyone would use your solution vs something like rsync.net or a small VM in DigitalOcean, etc. You said the selling point would be storage. What would be the differentiator in that area? Would price be it? Just trying to help with question to validate the idea. Best of luck.

jedberg 3y ago

Just one nit, FreeBSD had jails before March of 2000. I know this because I was using them in 1999. And in fact they were merged into CURRENT in early 1999 and STABLE in mid/late 1999. It was merged into RELEASE in March 2000.

erk__ 3y ago
They were actually developed a year before, but the person who paid for development had a year of exclusive use. http://freebsd.dk/sagas/jails/ http://freebsd.dk/sagas/jails/

gumballindie 3y ago

Every time I think of freebsd and how cool it was and is I wondering what caused linux to overtake it. I like linux as well, but I just don't understand the mechanics behind freebsd not gaining traction.

jbverschoor 3y ago
For me it was Java. I think there was something with mmap(). I ran FreeBSD servers with jails around ‘00. They worked fine. Not for me, but in hindsight, maybe the FreeBSD daemon mascotte had something to do with it. But around that time bea weblogic and ibm websphere also became popular, and I don’t think Java was officially supported. It was also a period where people tended to run oracle, but I’m not sure if that tab on Linux or something else. I think it was some custom Linux “unbreakable oracle”
chungy 3y ago
AT&T v BSDi had a large hand in that outcome of fates. It put the legality of BSD code in all its places into doubt for many years. Ironically, just a few years later, SCO thought "hey that was a good idea" and did it to Linux too.

Detrytus 3y ago

Offtopic, but isn't it funny that the system that calls itself FreeBSD has a concept of "jails"? Couldn't they come up with better name? :)