11 ms·

FreeBSD Jails were so much better than everything else out there, for a long time. I'll just copy&paste part of a comment I wrote on another HN thread some time

by drclau 5y ago

FreeBSD Jails were so much better than everything else out there, for a long time. I'll just copy&paste part of a comment I wrote on another HN thread some time ago, since it's relevant here: [...] In fact, many years ago, when FreeBSD was my main OS (including on notebook) I went as far as to isolate each app that used internet into its own custom-setup jail [0][1]. I had Firefox, Thunderbird, Pidgin and a few others running in complete isolation from the base system, and from each other. I even had a separate Firefox jail that was only allowed to get out via a Tor socks proxy to avoid leaks (more of an experiment than a necessity, to be fair). Communication between jails was done via commonly mounted nullfs. I have also setup QoS via PF for each of them. They were all running on the host’s Xorg, which was probably also the weakness of this setup. It was a pretty sweet setup, but required quite a bit of effort to maintain, even tho I automated most of the stuff. [...] The original comment is here: https://news.ycombinator.com/item?id=27709256 https://news.ycombinator.com/item?id=27709256

ajross 5y ago
Yes, but in a sense that's the essence of why the technology got left behind. Jails were a mechanism for expert admins to play with container ideas. What the market actually wanted was Docker. And what Docker needed was Linux containers (complicated, flexible, piecewise technology) and not jails, which were higher level abstractions (but yet not high enough) with jargon and framework assumptions that didn't match Docker's needs 1:1.
- pjmlp 5y ago
  And slowly the only thing left from Docker will be the image format.
  - GordonS 5y ago
    And the Registry API too, I expect, but yeah, the untimely demise of Docker as a company feels inevitable at this point.
- StreamBright 5y ago
  The "market" did not want Docker. Docker as a product failed. There are many reasons why FreeBSD jails count not get out for FreeBSD land, one, very important thing is the Linux community's NIH attitude.
  - shatteredgate 5y ago
    I don't think you can explain that as NIH. From what I have seen, Linux namespaces are a much more powerful primitive than BSD jails.
    - kazen44 5y ago
      namespaces in what sense? network namespaces? because freebsd has VNET for a while now. which seems to do the same thing.
      - shatteredgate 5y ago
        And all the other ones: https://man7.org/linux/man-pages/man7/namespaces.7.html https://man7.org/linux/man-pages/man7/namespaces.7.html BSD jails are similar but not quite the same thing.
        ori_b 5y ago
        I don't get it. How are people using this flexibility to get things done in practice, and what uses aren't allowed by the jail model?
        shatteredgate 5y ago
        You can just compare the APIs, namespaces are like the individual components of a jail. You can use them to build something like a jail, or something different that has a different security model. This was discussed a lot in an old HN thread: https://news.ycombinator.com/item?id=13982620 https://news.ycombinator.com/item?id=13982620
pjmlp 5y ago
I beg to differ, given that HP-UX Virtual Vaults were quite alright and precede Jails.
- drclau 5y ago
  (noticed your reply to my other comment too, please consider this a reply to both) I wasn't aware of HP-UX Virtual Vaults, thanks. However, I'd say FreeBSD Jails still had an advantage, due to being free and running on various hardware platforms, and more importantly on commodity hardware.
  - pjmlp 5y ago
    Eventually they were replaced by HP-UX Containers (SRP) a couple of years later. Unfortunately it is hard to still find documentation, given the troubles HP-UX has gone through at HP (which kind of plays into your remark regarding FreeBSD). Still there you go, https://support.hpe.com/hpesc/public/docDisplay?docLocale=en_US&docId=c02791255 https://support.hpe.com/hpesc/public/docDisplay?docLocale=en...
- hestefisk 5y ago
  In the same way LPARs preceded Virtual Vaults on System z? :)
  - pjmlp 5y ago
    I guess so, but I was constraining myself to UNIXes. :)
vegai_ 5y ago
Is there a reason why FreeBSD doesn't default to running all applications in jails? Seems like this would be a pretty huge advantage compared to the typical Unix system's almost complete lack of sandboxing.
- kazen44 5y ago
  what should run in a jail? you need the base system to run jails. and that is pretty much the only thing that is installed.
  - vegai_ 5y ago
    Because if I don't run in a jail, every software I run has full read/write access to all my files. Almost none of the software I run needs even read access to anything except its own files.
    - nix23 5y ago
      Jail is not a Sandbox but OS-Virtualization, those are two different things, Capsicum is the sandbox for FreeBSD...like secomp is for Linux.
rsync 5y ago
"... I even had a separate Firefox jail that was only allowed to get out via a Tor socks proxy to avoid leaks ..." I have looked into doing this many times and it's neither simple nor straightforward. Specifically: jailing a GUI app that you can interact with on your desktop. I can't remember what the most promising recipe I saw for this was but it wasn't quite promising enough to compel me to built it up ... and this discussion is always (rightly) hijacked with "just use Qubes" ...
- VTimofeenko 5y ago
  Firejail with spawning nested Xorg works fine for me, including text-only copypaste between "host" and "guest" and automatic file synchronization through bind-like mounts. For some firejails I also use Linux network namespaces to control traffic going through taps. My introduction to this approach was the alternative Gentoo handbook by Sakaki[1], but the principles would apply on any distro. There's also a very interesting read on Qubes-like experience on NixOs with Wayland and XWayland[2,3]. [1]: https://wiki.gentoo.org/wiki/User:Sakaki/Sakaki%27s_EFI_Install_Guide/Sandboxing_the_Firefox_Browser_with_Firejail https://wiki.gentoo.org/wiki/User:Sakaki/Sakaki%27s_EFI_Inst... [2]: https://roscidus.com/blog/blog/2021/03/07/qubes-lite-with-kvm-and-wayland/ https://roscidus.com/blog/blog/2021/03/07/qubes-lite-with-kv... [3]: https://roscidus.com/blog/blog/2021/10/30/xwayland/ https://roscidus.com/blog/blog/2021/10/30/xwayland/
  - tome 5y ago
    What's nested Xorg? Do you mean you run each app in a separate VNC or RDP server?
    - VTimofeenko 5y ago
      An example would be Xephyr[1]. Archwiki[2] has a decent summary. In my case, I have the standard xorg session started by my login manager. Then I start Xephyr with a separate DISPLAY, that shows up as just a window in the parent environment. It does look kinda like RDP or VNC. [1]: https://linux.die.net/man/1/xephyr https://linux.die.net/man/1/xephyr [2]: https://wiki.archlinux.org/title/Xephyr https://wiki.archlinux.org/title/Xephyr
cnst 5y ago
Sounds pretty cool! TBH, I don't understand how the "modern" app security models are not deemed to be fundamentally insecure in 2021 where any random app on any random system can simply leak your whole photo library to their servers, or delete all your files. Why do banking apps need full access to my filesystem? Why don't the reviewers of the app stores prohibit such practices of excessive permissions? Yes, there's always a question of usability, but if you're an advanced user, it's still scary that you're not afforded any control to prevent these incidents unless you go ahead and redesign the whole way all these apps are working all by yourself. It seems terribly inefficient if every engineer has to do it on their own, and in their own incompatible way. Obviously most people simply give up after a while, since maintaining such a setup might itself be a whole full-time job.

by drclau 5y ago

ajross 5y ago

Yes, but in a sense that's the essence of why the technology got left behind. Jails were a mechanism for expert admins to play with container ideas. What the market actually wanted was Docker. And what Docker needed was Linux containers (complicated, flexible, piecewise technology) and not jails, which were higher level abstractions (but yet not high enough) with jargon and framework assumptions that didn't match Docker's needs 1:1.

pjmlp 5y ago
And slowly the only thing left from Docker will be the image format.
- GordonS 5y ago
  And the Registry API too, I expect, but yeah, the untimely demise of Docker as a company feels inevitable at this point.
StreamBright 5y ago
The "market" did not want Docker. Docker as a product failed. There are many reasons why FreeBSD jails count not get out for FreeBSD land, one, very important thing is the Linux community's NIH attitude.
- shatteredgate 5y ago
  I don't think you can explain that as NIH. From what I have seen, Linux namespaces are a much more powerful primitive than BSD jails.
  - kazen44 5y ago
    namespaces in what sense? network namespaces? because freebsd has VNET for a while now. which seems to do the same thing.
    - shatteredgate 5y ago
      And all the other ones: https://man7.org/linux/man-pages/man7/namespaces.7.html https://man7.org/linux/man-pages/man7/namespaces.7.html BSD jails are similar but not quite the same thing.
      - ori_b 5y ago
        I don't get it. How are people using this flexibility to get things done in practice, and what uses aren't allowed by the jail model?
        shatteredgate 5y ago
        You can just compare the APIs, namespaces are like the individual components of a jail. You can use them to build something like a jail, or something different that has a different security model. This was discussed a lot in an old HN thread: https://news.ycombinator.com/item?id=13982620 https://news.ycombinator.com/item?id=13982620

pjmlp 5y ago

I beg to differ, given that HP-UX Virtual Vaults were quite alright and precede Jails.

drclau 5y ago
(noticed your reply to my other comment too, please consider this a reply to both) I wasn't aware of HP-UX Virtual Vaults, thanks. However, I'd say FreeBSD Jails still had an advantage, due to being free and running on various hardware platforms, and more importantly on commodity hardware.
- pjmlp 5y ago
  Eventually they were replaced by HP-UX Containers (SRP) a couple of years later. Unfortunately it is hard to still find documentation, given the troubles HP-UX has gone through at HP (which kind of plays into your remark regarding FreeBSD). Still there you go, https://support.hpe.com/hpesc/public/docDisplay?docLocale=en_US&docId=c02791255 https://support.hpe.com/hpesc/public/docDisplay?docLocale=en...
hestefisk 5y ago
In the same way LPARs preceded Virtual Vaults on System z? :)
- pjmlp 5y ago
  I guess so, but I was constraining myself to UNIXes. :)

vegai_ 5y ago

Is there a reason why FreeBSD doesn't default to running all applications in jails? Seems like this would be a pretty huge advantage compared to the typical Unix system's almost complete lack of sandboxing.

kazen44 5y ago
what should run in a jail? you need the base system to run jails. and that is pretty much the only thing that is installed.
- vegai_ 5y ago
  Because if I don't run in a jail, every software I run has full read/write access to all my files. Almost none of the software I run needs even read access to anything except its own files.
  - nix23 5y ago
    Jail is not a Sandbox but OS-Virtualization, those are two different things, Capsicum is the sandbox for FreeBSD...like secomp is for Linux.

rsync 5y ago

"... I even had a separate Firefox jail that was only allowed to get out via a Tor socks proxy to avoid leaks ..." I have looked into doing this many times and it's neither simple nor straightforward. Specifically: jailing a GUI app that you can interact with on your desktop. I can't remember what the most promising recipe I saw for this was but it wasn't quite promising enough to compel me to built it up ... and this discussion is always (rightly) hijacked with "just use Qubes" ...

VTimofeenko 5y ago
Firejail with spawning nested Xorg works fine for me, including text-only copypaste between "host" and "guest" and automatic file synchronization through bind-like mounts. For some firejails I also use Linux network namespaces to control traffic going through taps. My introduction to this approach was the alternative Gentoo handbook by Sakaki[1], but the principles would apply on any distro. There's also a very interesting read on Qubes-like experience on NixOs with Wayland and XWayland[2,3]. [1]: https://wiki.gentoo.org/wiki/User:Sakaki/Sakaki%27s_EFI_Install_Guide/Sandboxing_the_Firefox_Browser_with_Firejail https://wiki.gentoo.org/wiki/User:Sakaki/Sakaki%27s_EFI_Inst... [2]: https://roscidus.com/blog/blog/2021/03/07/qubes-lite-with-kvm-and-wayland/ https://roscidus.com/blog/blog/2021/03/07/qubes-lite-with-kv... [3]: https://roscidus.com/blog/blog/2021/10/30/xwayland/ https://roscidus.com/blog/blog/2021/10/30/xwayland/
- tome 5y ago
  What's nested Xorg? Do you mean you run each app in a separate VNC or RDP server?
  - VTimofeenko 5y ago
    An example would be Xephyr[1]. Archwiki[2] has a decent summary. In my case, I have the standard xorg session started by my login manager. Then I start Xephyr with a separate DISPLAY, that shows up as just a window in the parent environment. It does look kinda like RDP or VNC. [1]: https://linux.die.net/man/1/xephyr https://linux.die.net/man/1/xephyr [2]: https://wiki.archlinux.org/title/Xephyr https://wiki.archlinux.org/title/Xephyr

cnst 5y ago

Sounds pretty cool! TBH, I don't understand how the "modern" app security models are not deemed to be fundamentally insecure in 2021 where any random app on any random system can simply leak your whole photo library to their servers, or delete all your files. Why do banking apps need full access to my filesystem? Why don't the reviewers of the app stores prohibit such practices of excessive permissions? Yes, there's always a question of usability, but if you're an advanced user, it's still scary that you're not afforded any control to prevent these incidents unless you go ahead and redesign the whole way all these apps are working all by yourself. It seems terribly inefficient if every engineer has to do it on their own, and in their own incompatible way. Obviously most people simply give up after a while, since maintaining such a setup might itself be a whole full-time job.