32 ms·

FreeBSD Jails for Fun and Profit (2020)

280 points by kettunen 5y ago

freemint 5y ago
I never understood the appeal of BSD jails over Solaris zones which seem to be more hardened and seem to vitualize more of the OS.
- fsflover 5y ago
  Same here, but I never understood how BSD jails or Solaris zones are better over normal hardware virtualization which is used in Qubes OS. In addition, you get a great UX in the latter.
  - nix23 5y ago
    > Same here, but I never understood how BSD jails or Solaris zones are better over normal hardware virtualization which is used in Qubes OS. In addition, you get a great UX in the latter. Tells me you don't know the difference between HW-Virtualisation and OS-Virtualisation.
    - fsflover 5y ago
      Of course I do know it. Which is why I'm asking why you would use OS-virtualization, if HW- is possible. (Unless your hardware does not allow that of course).
      - nix23 5y ago
        Since you know the difference you know the pro and contra.
  - laumars 5y ago
    Containerisation makes much more sense than virtualisation in a lot of areas: * lower runtime overhead * quicker deployment time * smaller image foot print Plus most of the advantages of VMs can still be applied. However it does massively depend on individual use cases. Qubes aims to be ultra secure and containers in Linux weren’t up to that task at the time (the situation has since improved massively).
    - StreamBright 5y ago
      - lower runtime overhead - quicker deployment time - smaller image foot print Firecracker beg to differ. I think you can do virtualisation and have all three you listed.
      - laumars 5y ago
        Sure. But for every edge case where virtualisation bridges the gap, there will be containerisation solutions that further the gap too. Also I’m not really in the mood to engage in a dumb flame war. I’m just answering the question as to why some people favour containers for some workflows. If firecracker works for you then keep at it.
    - throwawayboise 5y ago
      With modern hardware support for virtualization, is runtime overhead still a big deal (assuming your virtual environments are not emulating some other CPU?)
      - laumars 5y ago
        Depends on the industry. For general purpose computing it’s generally not an issue but it does matter for some specific workflows. I tend to look at it the other way though: given containerisation is so easy these days, what are the compelling reasons to run virtual machines. Neither is a wrong answer though.
        throwawayboise 5y ago
        VMs still provide better isolation and security, or is that no longer true?
        laumars 5y ago
        They do but not by a significant margin these days. The kind of domains where this is an issue isn’t what you’re average engineer would be concerned with. Where VMs do help the average engineer is they have more secure defaults. It’s pretty easy to accidentally run containers insecurely on Linux (FreeBSD jails are a better story though)
  - gaadd33 5y ago
    I don't think Qubes OS existed around 2000, did it?
    - fsflover 5y ago
      I was asking about what to use now.
      - nix23 5y ago
        For what? That would be the first thing to know.....
        fsflover 5y ago
        For security and workflow separation. What else is there?
        nix23 5y ago
        On your phone, a desktop a server? For real-time applications, HPC, analytics, ML/AI? Hardware virtualization has tradeoffs...sometimes massive ones, and then you trust proprietary hardware to make it faster...again.
        fsflover 5y ago
        Are you using open hardware for software virtualization? I'm curious which. > On your phone, a desktop a server? Phone doesn't support hardware virtualization of course. On my laptop I'm using Qubes OS.
        nix23 5y ago
        >Phone doesn't support hardware virtualization of course. Every CPU "supports" hardware virtualization. >software virtualization? That's a "VM" like JVM, we talk about OS-Virtualisation.
  - freemint 5y ago
    Also OS level emulation. Linux system calls can be translated to Solaris system calls allowing to run Linux specific workloads without having to emulate hardware and a full Linux kernels. Furthermore before Dtrace was ported Linux it allowed to debug Linux workloads under DTrace.
    - trasz 5y ago
      You can do both with FreeBSD in the exact same way.
      - freemint 5y ago
        I am fully aware. Parent was probably not.
    - NexRebular 5y ago
      Unless the software is "vendor locked" with linuxisms. I wish I could run everything only on zones but alas, still have to mix in some bhyves as well for e.g. docker support...
- laumars 5y ago
  FreeBSD is the appeal. Back in the earlier days of containerisation Linux had no options (Linux was pretty late to that particular game) and Solaris wasn’t free. So FreeBSD made a lot of sense. These days the tooling around Linux is better and there are open source forks of Solaris so FreeBSD might seem like an odd choice for some. However I still think FreeBSD is a rock solid operating system and one that doesn’t get taken as seriously these days as it should do.
  - AndrewUnmuted 5y ago
    FreeBSD's appeal for me is the easy maintenance. Everything related to config is in /etc/rc.conf; tunables in /boot/loader.conf. The ZFS implementation is rock-solid. I therefore only have to fiddle with my FreeBSD server, running 80TB ZFS storage pool, once a year or so. No other OS gives me this kind of comfort and stability.
    - freemint 5y ago
      I wonder if Solaris has bigger userbase it would also appeal to you.
      - AndrewUnmuted 5y ago
        Perhaps it would! But the userbase is definitely what finally got me into FreeBSD. The fact that hobbyists were contributing such excellent cli tools as `iocage` [0] and `vm-bhyve` [1] really is what got me over my fears. [0] https://iocage.io/ https://iocage.io/ [1] https://github.com/churchers/vm-bhyve https://github.com/churchers/vm-bhyve
      - deltarholamda 5y ago
        Way back when OpenSolaris was around, I put it on a server to experiment with. There was a lot to like about it, but one thing I didn't like was there was a lot of obvious cruft that has built up over the years. Things were in places that I didn't expect, but they had been there since 1904 when people wrote C with a quill pen, so SUN couldn't just move it around, because it would break MasterCard or something. The base FreeBSD system is well thought out, and because it had the ability to, it coalesced into a very coherent system. Man pages are great, everything is where it should be.
- drclau 5y ago
  Note that FreeBSD Jails were introduced in 1999, while Solaris Containers and Zones were introduces in 2004. At the time FreeBSD Jails were introduced, probably the only alternative that was wildly available was chroot, which is really far from what Jails offer. Full virtualization was too slow to be practical for most scenarios, back then. 1999 is the year when Pentium III was released.
  - betaby 5y ago
    > probably the only alternative that was wildly available was chroot Opensource Linux-vserver https://en.wikipedia.org/wiki/Linux-VServer https://en.wikipedia.org/wiki/Linux-VServer was available in 2001. At the same era commercial Virtuozzo was ubiquitous amount hosting providers.
  - pjmlp 5y ago
    Bringing other UNIX into the party, HP-UX Virtual Vaults, which were introduced in HP-UX 11, available in 1997.
  - freemint 5y ago
    I've seen nothing which would suggest that Solaris zones are more taxing than BSD jails. Did you imply that claim?
    - drclau 5y ago
      > Did you imply that claim? No, I didn't. I was comparing full virtualization to Jails. However, I realize now that there may not even have been a full virtualization solution available back then anyway, at least not for consumer hardware. I'll have to dig a bit on Wikipedia. I've never used Solaris Containers / Zones, but my understanding is that the implementation was similar to FreeBSD Jails, so I have no reason to believe the performance was different.
      - int_19h 5y ago
        Virtual PC and VMware did full virtualization on commodity hardware at the time (Macs and Windows PCs, respectively). But it was slow enough that it was mostly used for development and testing back then, and certainly not viable for any kind of routine sandboxing.
- shrubble 5y ago
  The issue with zones was the lack in the early days of GNU software and the difficulty in compiling certain things. For instance, awk can act differently and if you don't know the 25 year old decisions that led to the differences, it can be very confusing. (There are different behaviors and command line switches between GNU's AWK and the version from SVR4/XPG)
  - eddieh 5y ago
    Lack of GNU stuff would be a selling point.
    - zokula 5y ago
      - eddieh 5y ago
        Wow. Name calling, especially such wholly inappropriate and inflammatory name calling, is way outside the norms here on HN. With brand ambassadors like yourself is there any wonder why I and many others want nothing to do with GNU or the FSF?
luto 5y ago
(FreeBSD) jails are amazing. I just wish there were easier ways to use them more "cattle"-like, so I can augment or replace Docker/Podman. At the moment tooling and many of the real-world setups remind me a lot of "pet" LXC containers or even VMs in the Linux world. The tooling is slowly moving in a direction I like, though :)
- kettunen 5y ago
  Author here! This is an old post of mine which I happened to find useful. Orchestration of jails moved quite bit forward lately! For example, you can manage your jails quite nicely with containerd today! See great post from Samuel Karp about the topic: https://samuel.karp.dev/blog/2021/05/running-freebsd-jails-with-containerd-1-5/ https://samuel.karp.dev/blog/2021/05/running-freebsd-jails-w...
- milofeynman 5y ago
  I use LXC on Proxmox and I do everything with Ansible scripts. Is there something moving towards docker-like repository in LXC land? Would love to just run the latest pihole or nginx or what have you on LXC
- heresie-dabord 5y ago
  On a Debian GNU/Linux or derivative: apt-cache search jail firejail - sandbox to restrict the application environment firejail-profiles - profiles for the firejail application sandbox firetools - Qt frontend for the Firejail application sandbox
- stjohnswarts 5y ago
  Are they superior to firejail on linux? I kind of always figured they were similar level of "sandboxing" but I never had enough interest in BSD to dig in myself.
  - trasz 5y ago
    They are completely different mechanism for doing different kind of stuff. Firejail sounds like something closer to Capsicum, but without the security model.
okokwhatever 5y ago
I am amazed at how many interesting things I still have to learn in this life. Too many tools, too little time to see them all... :(
- moeris 5y ago
  ars longa vita brevis https://en.wikipedia.org/wiki/Ars_longa%2C_vita_brevis https://en.wikipedia.org/wiki/Ars_longa%2C_vita_brevis
drclau 5y ago
FreeBSD Jails were so much better than everything else out there, for a long time. I'll just copy&paste part of a comment I wrote on another HN thread some time ago, since it's relevant here: [...] In fact, many years ago, when FreeBSD was my main OS (including on notebook) I went as far as to isolate each app that used internet into its own custom-setup jail [0][1]. I had Firefox, Thunderbird, Pidgin and a few others running in complete isolation from the base system, and from each other. I even had a separate Firefox jail that was only allowed to get out via a Tor socks proxy to avoid leaks (more of an experiment than a necessity, to be fair). Communication between jails was done via commonly mounted nullfs. I have also setup QoS via PF for each of them. They were all running on the host’s Xorg, which was probably also the weakness of this setup. It was a pretty sweet setup, but required quite a bit of effort to maintain, even tho I automated most of the stuff. [...] The original comment is here: https://news.ycombinator.com/item?id=27709256 https://news.ycombinator.com/item?id=27709256
- ajross 5y ago
  Yes, but in a sense that's the essence of why the technology got left behind. Jails were a mechanism for expert admins to play with container ideas. What the market actually wanted was Docker. And what Docker needed was Linux containers (complicated, flexible, piecewise technology) and not jails, which were higher level abstractions (but yet not high enough) with jargon and framework assumptions that didn't match Docker's needs 1:1.
  - pjmlp 5y ago
    And slowly the only thing left from Docker will be the image format.
    - GordonS 5y ago
      And the Registry API too, I expect, but yeah, the untimely demise of Docker as a company feels inevitable at this point.
  - StreamBright 5y ago
    The "market" did not want Docker. Docker as a product failed. There are many reasons why FreeBSD jails count not get out for FreeBSD land, one, very important thing is the Linux community's NIH attitude.
qqumut 5y ago
Are Jails really that safe & secure?
- bigodanktime 5y ago
  Counts what you are afraid against. There's always some side channel attack that could possibly used to gain information, even on VM's this is true. Off the top of my head there could be some timing attack to gain information on which libraries others are using by reading in libraries and seeing if they are warm in the buffer cache, counts if you care about sharing the same kernel. I generally find them secure enough considering how fast they can be brought up and down.
  - antihero 5y ago
    Can a process in them exec on the rest of your system?
    - marcosdumay 5y ago
      The fundamental thing about those features (and the equivalent on every system except Windows) is that you can never get more capabilities, only less. Once you are in a jail, there is no API for getting out of it. You can't even see a binary from the rest of your system, and exec won't get you out.
- tptacek 5y ago
  They're approximately as safe as modern Docker is. Upside to Docker: more security knobs (eBPF, kernel MAC, &c); upside to jails: probably easier to get right out of the box, fewer footguns. Both jails and containers (and Solaris Zones) share a fundamental security weakness, which is a kernel shared between tenants.
shrubble 5y ago
I have been playing around with the bastillebsd.org scripts for creating and managing jails. I think it aims to be more 'Docker like'. Worked well from the limited testing I have done so far
tambourine_man 5y ago
I always hoped for macOS to borrow FreeBSD jails for itself. A Docker-like solution with a pretty UI could be really useful for pros. For novices, it could mean a less cumbersome security measure than the restrictions we’ve been experiencing since Catalina.
- eddieh 5y ago
  macOS would definitely benefit from adding support for jails and a whole host other features like procfs, unionfs, bindfs, tmpfs, etc.
  - trasz 5y ago
    Erm, why would anyone want procfs? It's obsoleted in FreeBSD for a good reason.
    - eddieh 5y ago
      It doesn't have to be procfs, but it would be nice to have a /proc like interface.
      - trasz 5y ago
        How it would be better than existing sysctl(3)-based one?
        eddieh 5y ago
        Okay, I'll admit I just want it to be more like Plan 9. sysctl, ioctl, fcntl, sockets, could all have a file based interface. Maybe it isn't any better.
- jbverschoor 5y ago
  I would like an auto-jail, based on the directory you’re in (similar to asdf/rbenv). There’s just too much complexity in software, and so many things run on your machine these days. You can’t really trust any application anymore
bigodanktime 5y ago
A great wrapper UI I have used for FreeBSD Jails is iocage (https://iocage.readthedocs.io/en/latest/ https://iocage.readthedocs.io/en/latest/). Its a great project.
- Timothycquinn 5y ago
  I was about to say the same. iocage is now the default jails wrapper on FreeNAS, which means that there is good documentation and support. The previous jails wrapper used by FreeNAS was not very well documented, but was written by some smart folks. One thing I like about iocage is how easy it is to grant the jail access to the host ZFS datasets. On a Jails note, I have had issues creating a jail that can do network inspection. I believe this is an issue with network restrictions of the jails subsystem itself. Eg, I could never run nmap or get mac addresses of remote hosts from within a jail.
dgellow 5y ago
Just curious, is there an equivalent (or at least similar in spirit) to FreeBSD jails in the Windows world?
- hestefisk 5y ago
  Windows 10 kernel supports “Sandbox”, which might be what comes closest.
  - dgellow 5y ago
    Do you mean the graphical Windows Sandbox tool? Can it also be used headless to run a service for example? That would come pretty close.
- mastax 5y ago
  There's Windows Containers[0], which are analogous to Docker containers. They can either be run shared-kernel or in a Hyper-V container. Also Virtualization-based security[1], which is supposed to be completely transparent so it's not really a developer or deployment tool. [0]: https://docs.microsoft.com/en-us/virtualization/windowscontainers/about/ https://docs.microsoft.com/en-us/virtualization/windowsconta... [1]: https://docs.microsoft.com/en-us/windows-hardware/design/device-experiences/oem-vbs https://docs.microsoft.com/en-us/windows-hardware/design/dev...
  - deleted 5y ago
    [deleted]
- VTimofeenko 5y ago
  There used to be Virtuozzo containers for Windows and looks like they are bringing it back as a technical preview: https://docs.virtuozzo.com/virtuozzo_hybrid_server_7_users_guide/advanced-tasks/managing-windows-containers.html https://docs.virtuozzo.com/virtuozzo_hybrid_server_7_users_g...
- erkserkserks 5y ago
  There’s Sandboxie, which works decently, but does have some compatibility issues: https://sandboxie-plus.com/ https://sandboxie-plus.com/
rsync 5y ago
One thing I find so valuable about jails is the ability to jail a single command with no other userland than what that binary requires. Here's an example from my personal name server: /usr/sbin/jail /jails/www www 10.10.10.36 /lighttpd -f conf/lighttpd.conf ... and although this jail has a lot of content files in it, the actual UNIX userland is only what is required to run 'lighttpd': # find /jails/www/usr | wc -l 43 So it's an extremely lightweight environment with very little attack surface. You can also share a lightweight environment with multiple commands - here are two other jail commands: /usr/sbin/jail /jails/dns ns1 10.10.10.30 /nsd/nsd -c /nsd/nsd.conf /usr/sbin/jail /jails/dns dns 10.10.10.37 /unbound/unbound -c /unbound/unbound.conf ... see how both jailings of 'nsd' and 'unbound' point to the same '/jails/dns' userland ? Once again, that userland is very, very compact: # find /jails/dns/|wc -l 97 ... so, 97 files total to run both name servers. No 'make world' necessary, no building and maintaining of a full FreeBSD system - just the lightest skeleton required for both 'nsd' and 'unbound'.
- cpuguy83 5y ago
  Totally feasible on Linux, btw. Packaging an entire system is more about convenience than anything else. It's also pretty difficult to package just the libs one needs when you are dependent on libc and other C libs. I suspect that if one was really ok with it, some tooling could be built to copy/link in system libs into the rootfs automatically from the host.
  - chriswarbo 5y ago
    > It's also pretty difficult to package just the libs one needs when you are dependent on libc and other C libs. It's pretty easy using Nix, e.g. this example defines a container for running a shell script: https://ryantm.github.io/nixpkgs/builders/images/ocitools https://ryantm.github.io/nixpkgs/builders/images/ocitools That script depends on bash, bash depends on libc, etc. so those dependencies (and only those dependencies) will be put in the container. (See https://nixos.org/guides/nix-pills/enter-environment.html#idm140737320659552 https://nixos.org/guides/nix-pills/enter-environment.html#id... for an example of what dependencies look like in Nix). > I suspect that if one was really ok with it, some tooling could be built to copy/link in system libs into the rootfs automatically from the host. Eww, no thanks! I want my containers to be reproducible.
SpaceInvader 5y ago
I use jails for years, the only thing which is painful are upgrades from ports for all the jails. It's time consuming. Poudriere helps but the whole thing is far from ideal :(
- trasz 5y ago
  Why not prebuilt packages? As in, prebuilt by FreeBSD.org, as opposed to the ones built you build yourself with Poudriere?
movedx 5y ago
With a community driver, HashiCorp's Nomad can handle FreeBSD Jails for you. Worth trying out if FreeBSD is your thing.
ComputerGuru 5y ago
Another cool thing about jails is that they're really easy to convert to bhyve virtual machines if your security needs or general paranoia levels increase at any point.

280 points by kettunen 5y ago

I never understood the appeal of BSD jails over Solaris zones which seem to be more hardened and seem to vitualize more of the OS.

fsflover 5y ago
Same here, but I never understood how BSD jails or Solaris zones are better over normal hardware virtualization which is used in Qubes OS. In addition, you get a great UX in the latter.
- nix23 5y ago
  > Same here, but I never understood how BSD jails or Solaris zones are better over normal hardware virtualization which is used in Qubes OS. In addition, you get a great UX in the latter. Tells me you don't know the difference between HW-Virtualisation and OS-Virtualisation.
  - fsflover 5y ago
    Of course I do know it. Which is why I'm asking why you would use OS-virtualization, if HW- is possible. (Unless your hardware does not allow that of course).
    - nix23 5y ago
      Since you know the difference you know the pro and contra.
- laumars 5y ago
  Containerisation makes much more sense than virtualisation in a lot of areas: * lower runtime overhead * quicker deployment time * smaller image foot print Plus most of the advantages of VMs can still be applied. However it does massively depend on individual use cases. Qubes aims to be ultra secure and containers in Linux weren’t up to that task at the time (the situation has since improved massively).
  - StreamBright 5y ago
    - lower runtime overhead - quicker deployment time - smaller image foot print Firecracker beg to differ. I think you can do virtualisation and have all three you listed.
    - laumars 5y ago
      Sure. But for every edge case where virtualisation bridges the gap, there will be containerisation solutions that further the gap too. Also I’m not really in the mood to engage in a dumb flame war. I’m just answering the question as to why some people favour containers for some workflows. If firecracker works for you then keep at it.
  - throwawayboise 5y ago
    With modern hardware support for virtualization, is runtime overhead still a big deal (assuming your virtual environments are not emulating some other CPU?)
    - laumars 5y ago
      Depends on the industry. For general purpose computing it’s generally not an issue but it does matter for some specific workflows. I tend to look at it the other way though: given containerisation is so easy these days, what are the compelling reasons to run virtual machines. Neither is a wrong answer though.
      - throwawayboise 5y ago
        VMs still provide better isolation and security, or is that no longer true?
        laumars 5y ago
        They do but not by a significant margin these days. The kind of domains where this is an issue isn’t what you’re average engineer would be concerned with. Where VMs do help the average engineer is they have more secure defaults. It’s pretty easy to accidentally run containers insecurely on Linux (FreeBSD jails are a better story though)
- gaadd33 5y ago
  I don't think Qubes OS existed around 2000, did it?
  - fsflover 5y ago
    I was asking about what to use now.
    - nix23 5y ago
      For what? That would be the first thing to know.....
      - fsflover 5y ago
        For security and workflow separation. What else is there?
        nix23 5y ago
        On your phone, a desktop a server? For real-time applications, HPC, analytics, ML/AI? Hardware virtualization has tradeoffs...sometimes massive ones, and then you trust proprietary hardware to make it faster...again.
        fsflover 5y ago
        Are you using open hardware for software virtualization? I'm curious which. > On your phone, a desktop a server? Phone doesn't support hardware virtualization of course. On my laptop I'm using Qubes OS.
        nix23 5y ago
        >Phone doesn't support hardware virtualization of course. Every CPU "supports" hardware virtualization. >software virtualization? That's a "VM" like JVM, we talk about OS-Virtualisation.
- freemint 5y ago
  Also OS level emulation. Linux system calls can be translated to Solaris system calls allowing to run Linux specific workloads without having to emulate hardware and a full Linux kernels. Furthermore before Dtrace was ported Linux it allowed to debug Linux workloads under DTrace.
  - trasz 5y ago
    You can do both with FreeBSD in the exact same way.
    - freemint 5y ago
      I am fully aware. Parent was probably not.
  - NexRebular 5y ago
    Unless the software is "vendor locked" with linuxisms. I wish I could run everything only on zones but alas, still have to mix in some bhyves as well for e.g. docker support...
laumars 5y ago
FreeBSD is the appeal. Back in the earlier days of containerisation Linux had no options (Linux was pretty late to that particular game) and Solaris wasn’t free. So FreeBSD made a lot of sense. These days the tooling around Linux is better and there are open source forks of Solaris so FreeBSD might seem like an odd choice for some. However I still think FreeBSD is a rock solid operating system and one that doesn’t get taken as seriously these days as it should do.
- AndrewUnmuted 5y ago
  FreeBSD's appeal for me is the easy maintenance. Everything related to config is in /etc/rc.conf; tunables in /boot/loader.conf. The ZFS implementation is rock-solid. I therefore only have to fiddle with my FreeBSD server, running 80TB ZFS storage pool, once a year or so. No other OS gives me this kind of comfort and stability.
  - freemint 5y ago
    I wonder if Solaris has bigger userbase it would also appeal to you.
    - AndrewUnmuted 5y ago
      Perhaps it would! But the userbase is definitely what finally got me into FreeBSD. The fact that hobbyists were contributing such excellent cli tools as `iocage` [0] and `vm-bhyve` [1] really is what got me over my fears. [0] https://iocage.io/ https://iocage.io/ [1] https://github.com/churchers/vm-bhyve https://github.com/churchers/vm-bhyve
    - deltarholamda 5y ago
      Way back when OpenSolaris was around, I put it on a server to experiment with. There was a lot to like about it, but one thing I didn't like was there was a lot of obvious cruft that has built up over the years. Things were in places that I didn't expect, but they had been there since 1904 when people wrote C with a quill pen, so SUN couldn't just move it around, because it would break MasterCard or something. The base FreeBSD system is well thought out, and because it had the ability to, it coalesced into a very coherent system. Man pages are great, everything is where it should be.
drclau 5y ago
Note that FreeBSD Jails were introduced in 1999, while Solaris Containers and Zones were introduces in 2004. At the time FreeBSD Jails were introduced, probably the only alternative that was wildly available was chroot, which is really far from what Jails offer. Full virtualization was too slow to be practical for most scenarios, back then. 1999 is the year when Pentium III was released.
- betaby 5y ago
  > probably the only alternative that was wildly available was chroot Opensource Linux-vserver https://en.wikipedia.org/wiki/Linux-VServer https://en.wikipedia.org/wiki/Linux-VServer was available in 2001. At the same era commercial Virtuozzo was ubiquitous amount hosting providers.
- pjmlp 5y ago
  Bringing other UNIX into the party, HP-UX Virtual Vaults, which were introduced in HP-UX 11, available in 1997.
- freemint 5y ago
  I've seen nothing which would suggest that Solaris zones are more taxing than BSD jails. Did you imply that claim?
  - drclau 5y ago
    > Did you imply that claim? No, I didn't. I was comparing full virtualization to Jails. However, I realize now that there may not even have been a full virtualization solution available back then anyway, at least not for consumer hardware. I'll have to dig a bit on Wikipedia. I've never used Solaris Containers / Zones, but my understanding is that the implementation was similar to FreeBSD Jails, so I have no reason to believe the performance was different.
    - int_19h 5y ago
      Virtual PC and VMware did full virtualization on commodity hardware at the time (Macs and Windows PCs, respectively). But it was slow enough that it was mostly used for development and testing back then, and certainly not viable for any kind of routine sandboxing.
shrubble 5y ago
The issue with zones was the lack in the early days of GNU software and the difficulty in compiling certain things. For instance, awk can act differently and if you don't know the 25 year old decisions that led to the differences, it can be very confusing. (There are different behaviors and command line switches between GNU's AWK and the version from SVR4/XPG)
- eddieh 5y ago
  Lack of GNU stuff would be a selling point.
  - zokula 5y ago
    - eddieh 5y ago
      Wow. Name calling, especially such wholly inappropriate and inflammatory name calling, is way outside the norms here on HN. With brand ambassadors like yourself is there any wonder why I and many others want nothing to do with GNU or the FSF?

luto 5y ago

(FreeBSD) jails are amazing. I just wish there were easier ways to use them more "cattle"-like, so I can augment or replace Docker/Podman. At the moment tooling and many of the real-world setups remind me a lot of "pet" LXC containers or even VMs in the Linux world. The tooling is slowly moving in a direction I like, though :)

kettunen 5y ago
Author here! This is an old post of mine which I happened to find useful. Orchestration of jails moved quite bit forward lately! For example, you can manage your jails quite nicely with containerd today! See great post from Samuel Karp about the topic: https://samuel.karp.dev/blog/2021/05/running-freebsd-jails-with-containerd-1-5/ https://samuel.karp.dev/blog/2021/05/running-freebsd-jails-w...
milofeynman 5y ago
I use LXC on Proxmox and I do everything with Ansible scripts. Is there something moving towards docker-like repository in LXC land? Would love to just run the latest pihole or nginx or what have you on LXC
heresie-dabord 5y ago
On a Debian GNU/Linux or derivative: apt-cache search jail firejail - sandbox to restrict the application environment firejail-profiles - profiles for the firejail application sandbox firetools - Qt frontend for the Firejail application sandbox
stjohnswarts 5y ago
Are they superior to firejail on linux? I kind of always figured they were similar level of "sandboxing" but I never had enough interest in BSD to dig in myself.
- trasz 5y ago
  They are completely different mechanism for doing different kind of stuff. Firejail sounds like something closer to Capsicum, but without the security model.

okokwhatever 5y ago

I am amazed at how many interesting things I still have to learn in this life. Too many tools, too little time to see them all... :(

moeris 5y ago
ars longa vita brevis https://en.wikipedia.org/wiki/Ars_longa%2C_vita_brevis https://en.wikipedia.org/wiki/Ars_longa%2C_vita_brevis

drclau 5y ago

FreeBSD Jails were so much better than everything else out there, for a long time. I'll just copy&paste part of a comment I wrote on another HN thread some time ago, since it's relevant here: [...] In fact, many years ago, when FreeBSD was my main OS (including on notebook) I went as far as to isolate each app that used internet into its own custom-setup jail [0][1]. I had Firefox, Thunderbird, Pidgin and a few others running in complete isolation from the base system, and from each other. I even had a separate Firefox jail that was only allowed to get out via a Tor socks proxy to avoid leaks (more of an experiment than a necessity, to be fair). Communication between jails was done via commonly mounted nullfs. I have also setup QoS via PF for each of them. They were all running on the host’s Xorg, which was probably also the weakness of this setup. It was a pretty sweet setup, but required quite a bit of effort to maintain, even tho I automated most of the stuff. [...] The original comment is here: https://news.ycombinator.com/item?id=27709256 https://news.ycombinator.com/item?id=27709256

ajross 5y ago
Yes, but in a sense that's the essence of why the technology got left behind. Jails were a mechanism for expert admins to play with container ideas. What the market actually wanted was Docker. And what Docker needed was Linux containers (complicated, flexible, piecewise technology) and not jails, which were higher level abstractions (but yet not high enough) with jargon and framework assumptions that didn't match Docker's needs 1:1.
- pjmlp 5y ago
  And slowly the only thing left from Docker will be the image format.
  - GordonS 5y ago
    And the Registry API too, I expect, but yeah, the untimely demise of Docker as a company feels inevitable at this point.
- StreamBright 5y ago
  The "market" did not want Docker. Docker as a product failed. There are many reasons why FreeBSD jails count not get out for FreeBSD land, one, very important thing is the Linux community's NIH attitude.

qqumut 5y ago

Are Jails really that safe & secure?

bigodanktime 5y ago
Counts what you are afraid against. There's always some side channel attack that could possibly used to gain information, even on VM's this is true. Off the top of my head there could be some timing attack to gain information on which libraries others are using by reading in libraries and seeing if they are warm in the buffer cache, counts if you care about sharing the same kernel. I generally find them secure enough considering how fast they can be brought up and down.
- antihero 5y ago
  Can a process in them exec on the rest of your system?
  - marcosdumay 5y ago
    The fundamental thing about those features (and the equivalent on every system except Windows) is that you can never get more capabilities, only less. Once you are in a jail, there is no API for getting out of it. You can't even see a binary from the rest of your system, and exec won't get you out.
tptacek 5y ago
They're approximately as safe as modern Docker is. Upside to Docker: more security knobs (eBPF, kernel MAC, &c); upside to jails: probably easier to get right out of the box, fewer footguns. Both jails and containers (and Solaris Zones) share a fundamental security weakness, which is a kernel shared between tenants.

shrubble 5y ago

I have been playing around with the bastillebsd.org scripts for creating and managing jails. I think it aims to be more 'Docker like'. Worked well from the limited testing I have done so far

tambourine_man 5y ago

I always hoped for macOS to borrow FreeBSD jails for itself. A Docker-like solution with a pretty UI could be really useful for pros. For novices, it could mean a less cumbersome security measure than the restrictions we’ve been experiencing since Catalina.

eddieh 5y ago
macOS would definitely benefit from adding support for jails and a whole host other features like procfs, unionfs, bindfs, tmpfs, etc.
- trasz 5y ago
  Erm, why would anyone want procfs? It's obsoleted in FreeBSD for a good reason.
  - eddieh 5y ago
    It doesn't have to be procfs, but it would be nice to have a /proc like interface.
    - trasz 5y ago
      How it would be better than existing sysctl(3)-based one?
      - eddieh 5y ago
        Okay, I'll admit I just want it to be more like Plan 9. sysctl, ioctl, fcntl, sockets, could all have a file based interface. Maybe it isn't any better.
jbverschoor 5y ago
I would like an auto-jail, based on the directory you’re in (similar to asdf/rbenv). There’s just too much complexity in software, and so many things run on your machine these days. You can’t really trust any application anymore

bigodanktime 5y ago

A great wrapper UI I have used for FreeBSD Jails is iocage (https://iocage.readthedocs.io/en/latest/ https://iocage.readthedocs.io/en/latest/). Its a great project.

Timothycquinn 5y ago
I was about to say the same. iocage is now the default jails wrapper on FreeNAS, which means that there is good documentation and support. The previous jails wrapper used by FreeNAS was not very well documented, but was written by some smart folks. One thing I like about iocage is how easy it is to grant the jail access to the host ZFS datasets. On a Jails note, I have had issues creating a jail that can do network inspection. I believe this is an issue with network restrictions of the jails subsystem itself. Eg, I could never run nmap or get mac addresses of remote hosts from within a jail.

dgellow 5y ago

Just curious, is there an equivalent (or at least similar in spirit) to FreeBSD jails in the Windows world?

hestefisk 5y ago
Windows 10 kernel supports “Sandbox”, which might be what comes closest.
- dgellow 5y ago
  Do you mean the graphical Windows Sandbox tool? Can it also be used headless to run a service for example? That would come pretty close.
mastax 5y ago
There's Windows Containers[0], which are analogous to Docker containers. They can either be run shared-kernel or in a Hyper-V container. Also Virtualization-based security[1], which is supposed to be completely transparent so it's not really a developer or deployment tool. [0]: https://docs.microsoft.com/en-us/virtualization/windowscontainers/about/ https://docs.microsoft.com/en-us/virtualization/windowsconta... [1]: https://docs.microsoft.com/en-us/windows-hardware/design/device-experiences/oem-vbs https://docs.microsoft.com/en-us/windows-hardware/design/dev...
- deleted 5y ago
  [deleted]
VTimofeenko 5y ago
There used to be Virtuozzo containers for Windows and looks like they are bringing it back as a technical preview: https://docs.virtuozzo.com/virtuozzo_hybrid_server_7_users_guide/advanced-tasks/managing-windows-containers.html https://docs.virtuozzo.com/virtuozzo_hybrid_server_7_users_g...
erkserkserks 5y ago
There’s Sandboxie, which works decently, but does have some compatibility issues: https://sandboxie-plus.com/ https://sandboxie-plus.com/

rsync 5y ago

One thing I find so valuable about jails is the ability to jail a single command with no other userland than what that binary requires. Here's an example from my personal name server: /usr/sbin/jail /jails/www www 10.10.10.36 /lighttpd -f conf/lighttpd.conf ... and although this jail has a lot of content files in it, the actual UNIX userland is only what is required to run 'lighttpd': # find /jails/www/usr | wc -l 43 So it's an extremely lightweight environment with very little attack surface. You can also share a lightweight environment with multiple commands - here are two other jail commands: /usr/sbin/jail /jails/dns ns1 10.10.10.30 /nsd/nsd -c /nsd/nsd.conf /usr/sbin/jail /jails/dns dns 10.10.10.37 /unbound/unbound -c /unbound/unbound.conf ... see how both jailings of 'nsd' and 'unbound' point to the same '/jails/dns' userland ? Once again, that userland is very, very compact: # find /jails/dns/|wc -l 97 ... so, 97 files total to run both name servers. No 'make world' necessary, no building and maintaining of a full FreeBSD system - just the lightest skeleton required for both 'nsd' and 'unbound'.

cpuguy83 5y ago
Totally feasible on Linux, btw. Packaging an entire system is more about convenience than anything else. It's also pretty difficult to package just the libs one needs when you are dependent on libc and other C libs. I suspect that if one was really ok with it, some tooling could be built to copy/link in system libs into the rootfs automatically from the host.
- chriswarbo 5y ago
  > It's also pretty difficult to package just the libs one needs when you are dependent on libc and other C libs. It's pretty easy using Nix, e.g. this example defines a container for running a shell script: https://ryantm.github.io/nixpkgs/builders/images/ocitools https://ryantm.github.io/nixpkgs/builders/images/ocitools That script depends on bash, bash depends on libc, etc. so those dependencies (and only those dependencies) will be put in the container. (See https://nixos.org/guides/nix-pills/enter-environment.html#idm140737320659552 https://nixos.org/guides/nix-pills/enter-environment.html#id... for an example of what dependencies look like in Nix). > I suspect that if one was really ok with it, some tooling could be built to copy/link in system libs into the rootfs automatically from the host. Eww, no thanks! I want my containers to be reproducible.

SpaceInvader 5y ago

I use jails for years, the only thing which is painful are upgrades from ports for all the jails. It's time consuming. Poudriere helps but the whole thing is far from ideal :(

trasz 5y ago
Why not prebuilt packages? As in, prebuilt by FreeBSD.org, as opposed to the ones built you build yourself with Poudriere?

movedx 5y ago

With a community driver, HashiCorp's Nomad can handle FreeBSD Jails for you. Worth trying out if FreeBSD is your thing.

ComputerGuru 5y ago

Another cool thing about jails is that they're really easy to convert to bhyve virtual machines if your security needs or general paranoia levels increase at any point.