5 ms·

Obviously the author put a lot of effort into this paper. Hard work shows throughout. Kudos to you sir! Granted I realize the title is "Understanding and Harde

by X86BSD 11y ago

Obviously the author put a lot of effort into this paper. Hard work shows throughout. Kudos to you sir! Granted I realize the title is "Understanding and Hardening Linux Containers". However, personally, this just illustrates my frustration and the frustration of others with the Linux world. They exist in their own little echo chamber. Linux was not the first to create "containers" that are secure. There is no inclusion of other solutions outside Linux. The reason other solutions do not have these problems is because the authors actually thought about the problem. And if the Linux camp would simply look outside their echo chamber long enough to see how others solved these problems before them this paper might not have been written. It's just simply frustrating to watch Linux reinvent some wheels, poorly, time after time. Flame on!

heroprotagonist 11y ago
I see nothing wrong with this. I think it's good to be specific. The Linux implementation is different from many other containerization solutions (eg, Solaris Zones, AIX WPAR, HPUX partitions, etc, so forth). The scope of "all container solutions for every platform" is too broad to be able to provide much depth, and the audience of people interested in container information for all platforms instead of a specific platform is fairly narrow. There's plenty of room for someone to make a document specific to other platforms for a different audience. There's even room for someone to make a comparison of all platforms. It depends on your goal, your knowledge, and your intended audience.
- X86BSD 11y ago
  It's not that the paper is wrong for only focusing on Linux "containers". It's that the paper, while describing the security short comings of the various solutions, illustrates the echo chamber I refer to. "We have these security issues, that others do not have, but we are not going to look at how they solved these issues because, well, Linux!" The paper, IMO, would have better served the audience had it included ideas and/or solutions people outside Linux-land developed to solve some of these issues that were written before Linux even attempted a "container". You are free to disagree.
  - Scarbutt 11y ago
    and how do you know the others solutions are really secure? Linux is going to get the most eyes, critics, scrutiny on this because is way more widely used than the other platforms that provide similar technologies.
    - AdieuToLogic 11y ago
      and how do you know the others solutions are really secure? Linux is going to get the most eyes, critics, scrutiny on this because is way more widely used ... This is commonly known as the "given enough eyeballs" fallacy[1]. While it might help when more people have an opportunity to review security critical subsystems, a given OS' popularity certainly does not guarantee this. Heartbleed[2] is often cited as an exemplar for this very situation. 1 - http://www.amazon.com/Facts-Fallacies-Software-Engineering-Robert/dp/0321117425 http://www.amazon.com/Facts-Fallacies-Software-Engineering-R... 2 - https://en.wikipedia.org/wiki/Heartbleed https://en.wikipedia.org/wiki/Heartbleed
    - pyvpx 11y ago
      other than something like seL4, no one _knows_ what is really secure. however it would be hard to argue that a decade+ of container systems such as FreeBSD Jail and Solaris' functional equivalent (the precise name escapes me) don't have useful lessons "Linux can learn from".
    - dap 11y ago
      Because not only did the engineers who built it consider how the architecture would ensure security, they wrote all of the considerations and design choices down.[1] Even if they were completely wrong about everything, there's no reason to simply ignore all the thought that's been put into this problem. [1] https://us-east.manta.joyent.com/bcantrill/public/ppwl-cantrill-zones.pdf https://us-east.manta.joyent.com/bcantrill/public/ppwl-cantr...
    - i4k 11y ago
      Linux being secure is a fallacy too. Spender talks about it for a long time: https://grsecurity.net/~spender/interview_notes.txt https://grsecurity.net/~spender/interview_notes.txt Linux developers fix security bugs in irresponsible ways, saying nothing about the attack vector or CVE's involved. Sometimes multiple vulnerabilities are addressed in the same patch, making hard to track the fix. For more information see the exploit below: https://grsecurity.net/~spender/exploits/64bit_dos.c https://grsecurity.net/~spender/exploits/64bit_dos.c The list of kernel vulnerabilities involving namespaces is bigger each day, as the paper shows, but the industry still uses "container" and "security" in the same quotes.
vacri 11y ago
> It's just simply frustrating to watch Linux reinvent some wheels, poorly, time after time. Out of curiosity, why do you think that Linux got such traction and the BSDs didn't? I keep hearing from the BSD camp that the BSDs are better than Linux at, well, pretty much everything, but there must be some reason why BSDs were not as performant for people than Linux distros.
- AdieuToLogic 11y ago
  Out of curiosity, why do you think that Linux got such traction and the BSDs didn't? The AT&T lawsuit[1] against BSDi. Stymieing BSD in a well publicized lawsuit was enough to put a lot of people off from entertaining an x86 BSD (can't blame anyone for being prudent). Right around the same time, one Mr. Linus Torvalds was making his own OS. 1 - https://en.wikipedia.org/wiki/UNIX_System_Laboratories,_Inc._v._Berkeley_Software_Design,_Inc https://en.wikipedia.org/wiki/UNIX_System_Laboratories,_Inc.....
  - vacri 11y ago
    Thanks for the info, folks. I was unaware of the lawsuit. Curious.
- tachion 11y ago
  The lawsuit against involving BSD back in the day was one thing, and another one is... spreading FUD. That's about the fact that BSD's haven't got traction. Of course, relatively to Linux its smaller, or, better say, less visible: Sony Play Station is FreeBSD, Netflix uses FreeBSD for content delivery servers, WhatsApp is FreeBSD, OSX is BSD user land and much, much more, simply less visible.
  - pyvpx 11y ago
    the first version of Meraki were NetBSD -- same for Apple Airport devices. Junipers management plane has historically been FreeBSD (but that is unfortunately changing)
Annatar 11y ago
Totally agree with you. I fail to see the point of attempting to harden Linux "containers" (Linux actually does not have container technology, since cgroups are not containers). Why bother with that, when illumos, and SmartOS on top of it, provide free, open source Solaris zones technology? Not even installation is required, since the thing runs from random access memory, and can run Linux at the speed of bare metal? Everybody is just flapping on Linux containers. Meanwhile illumos and SmartOS are eating Linux's lunch with techology which has been working reliably in Swiss banks since 2006, and it is free open source software. What's not to love? There are T-shirts which say "Go away or I will replace you with a very small shell script." Now it's "Go away or I will virtualize your Linux inside of SmartOS." The damn thing runs like a bandit, while Linux containers keep on flapping trying to find a working solution with papers like these. Why bother, the problem has already been solved, go run SmartOS and enjoy a working solution!
- z3t4 11y ago
  SmartOS if very nice in theory. I tried to get SmartOS working on a new Supermicro server and failed, even though I got a lot of help from their awesome community and people at Joyent. Then I tried Linux (Ubuntu) and everything just worked! It was also very easy to find guides on the web for most things Linux.
  - Annatar 11y ago
    If you want the JustWorks(SM), you have to get hardware which is on the Joyent engineering bill of materials' list, or research and buy hardware which illumos has drivers for. http://eng.joyent.com/manufacturing/bom.html http://eng.joyent.com/manufacturing/bom.html SmartOS being descended from Solaris has detailed manual pages built into the operating system, because when one runs a real UNIX, one is entitled to comprehensive documentation with lots of examples. A lot of the Oracle Solaris documentation, which is also very comprehensive, applies to and can be used on illumos derivatives. The SmartOS wiki and the smartos-discuss mailing list are good sources of SmartOS specific enhancements. Did I mention the built-in man pages yet?
  - RantyDave 11y ago
dyn 11y ago
Hi author here. Thanks! It was a lot of hard work, I'm glad it's being well received and seems like people are generally finding it useful- the main point of writing it. I would have liked to talk a lot more about other non-Linux solutions, but that's where most of the NCC Group consulting work has been, which was a main driver for why this info needs to be out there (companies not understanding container security or not "turning it up to 11" in places it needs to be). My background is also mainly Linux, so I stuck to what I know. Hopefully someone else will release a similar paper on non-Linux solutions/unikernels/hybrid platforms?
JamesSwift 11y ago
As an ignorant spectator I appreciate the balance provided by your comment. However, I wish you would have provided some concrete examples of which wheels they are reinventing, and how those wheels exist in other solutions.
- X86BSD 11y ago
  Listen to this discussion[0] which highlights the point I am making. I set the marker for the relevant "container" comment but watch the entire thing to understand more fully what I and others are saying and why. [0] https://youtu.be/Ya6h2zKlpaQ?t=1h8m11s https://youtu.be/Ya6h2zKlpaQ?t=1h8m11s
- i4k 11y ago
  Container technology is grounded on namespaces and cgroups. Both technologies existed long time ago. Linux namespace is a very limited implementation of Plan9's namespace. Solaris and FreeBSD, to cite only the most famous, have resource limits database for process groups.

by X86BSD 11y ago

heroprotagonist 11y ago

I see nothing wrong with this. I think it's good to be specific. The Linux implementation is different from many other containerization solutions (eg, Solaris Zones, AIX WPAR, HPUX partitions, etc, so forth). The scope of "all container solutions for every platform" is too broad to be able to provide much depth, and the audience of people interested in container information for all platforms instead of a specific platform is fairly narrow. There's plenty of room for someone to make a document specific to other platforms for a different audience. There's even room for someone to make a comparison of all platforms. It depends on your goal, your knowledge, and your intended audience.

X86BSD 11y ago
It's not that the paper is wrong for only focusing on Linux "containers". It's that the paper, while describing the security short comings of the various solutions, illustrates the echo chamber I refer to. "We have these security issues, that others do not have, but we are not going to look at how they solved these issues because, well, Linux!" The paper, IMO, would have better served the audience had it included ideas and/or solutions people outside Linux-land developed to solve some of these issues that were written before Linux even attempted a "container". You are free to disagree.
- Scarbutt 11y ago
  and how do you know the others solutions are really secure? Linux is going to get the most eyes, critics, scrutiny on this because is way more widely used than the other platforms that provide similar technologies.
  - AdieuToLogic 11y ago
    and how do you know the others solutions are really secure? Linux is going to get the most eyes, critics, scrutiny on this because is way more widely used ... This is commonly known as the "given enough eyeballs" fallacy[1]. While it might help when more people have an opportunity to review security critical subsystems, a given OS' popularity certainly does not guarantee this. Heartbleed[2] is often cited as an exemplar for this very situation. 1 - http://www.amazon.com/Facts-Fallacies-Software-Engineering-Robert/dp/0321117425 http://www.amazon.com/Facts-Fallacies-Software-Engineering-R... 2 - https://en.wikipedia.org/wiki/Heartbleed https://en.wikipedia.org/wiki/Heartbleed
  - pyvpx 11y ago
    other than something like seL4, no one _knows_ what is really secure. however it would be hard to argue that a decade+ of container systems such as FreeBSD Jail and Solaris' functional equivalent (the precise name escapes me) don't have useful lessons "Linux can learn from".
  - dap 11y ago
    Because not only did the engineers who built it consider how the architecture would ensure security, they wrote all of the considerations and design choices down.[1] Even if they were completely wrong about everything, there's no reason to simply ignore all the thought that's been put into this problem. [1] https://us-east.manta.joyent.com/bcantrill/public/ppwl-cantrill-zones.pdf https://us-east.manta.joyent.com/bcantrill/public/ppwl-cantr...
  - i4k 11y ago
    Linux being secure is a fallacy too. Spender talks about it for a long time: https://grsecurity.net/~spender/interview_notes.txt https://grsecurity.net/~spender/interview_notes.txt Linux developers fix security bugs in irresponsible ways, saying nothing about the attack vector or CVE's involved. Sometimes multiple vulnerabilities are addressed in the same patch, making hard to track the fix. For more information see the exploit below: https://grsecurity.net/~spender/exploits/64bit_dos.c https://grsecurity.net/~spender/exploits/64bit_dos.c The list of kernel vulnerabilities involving namespaces is bigger each day, as the paper shows, but the industry still uses "container" and "security" in the same quotes.

vacri 11y ago

> It's just simply frustrating to watch Linux reinvent some wheels, poorly, time after time. Out of curiosity, why do you think that Linux got such traction and the BSDs didn't? I keep hearing from the BSD camp that the BSDs are better than Linux at, well, pretty much everything, but there must be some reason why BSDs were not as performant for people than Linux distros.

AdieuToLogic 11y ago
Out of curiosity, why do you think that Linux got such traction and the BSDs didn't? The AT&T lawsuit[1] against BSDi. Stymieing BSD in a well publicized lawsuit was enough to put a lot of people off from entertaining an x86 BSD (can't blame anyone for being prudent). Right around the same time, one Mr. Linus Torvalds was making his own OS. 1 - https://en.wikipedia.org/wiki/UNIX_System_Laboratories,_Inc._v._Berkeley_Software_Design,_Inc https://en.wikipedia.org/wiki/UNIX_System_Laboratories,_Inc.....
- vacri 11y ago
  Thanks for the info, folks. I was unaware of the lawsuit. Curious.
tachion 11y ago
The lawsuit against involving BSD back in the day was one thing, and another one is... spreading FUD. That's about the fact that BSD's haven't got traction. Of course, relatively to Linux its smaller, or, better say, less visible: Sony Play Station is FreeBSD, Netflix uses FreeBSD for content delivery servers, WhatsApp is FreeBSD, OSX is BSD user land and much, much more, simply less visible.
- pyvpx 11y ago
  the first version of Meraki were NetBSD -- same for Apple Airport devices. Junipers management plane has historically been FreeBSD (but that is unfortunately changing)

Annatar 11y ago

Totally agree with you. I fail to see the point of attempting to harden Linux "containers" (Linux actually does not have container technology, since cgroups are not containers). Why bother with that, when illumos, and SmartOS on top of it, provide free, open source Solaris zones technology? Not even installation is required, since the thing runs from random access memory, and can run Linux at the speed of bare metal? Everybody is just flapping on Linux containers. Meanwhile illumos and SmartOS are eating Linux's lunch with techology which has been working reliably in Swiss banks since 2006, and it is free open source software. What's not to love? There are T-shirts which say "Go away or I will replace you with a very small shell script." Now it's "Go away or I will virtualize your Linux inside of SmartOS." The damn thing runs like a bandit, while Linux containers keep on flapping trying to find a working solution with papers like these. Why bother, the problem has already been solved, go run SmartOS and enjoy a working solution!

z3t4 11y ago
SmartOS if very nice in theory. I tried to get SmartOS working on a new Supermicro server and failed, even though I got a lot of help from their awesome community and people at Joyent. Then I tried Linux (Ubuntu) and everything just worked! It was also very easy to find guides on the web for most things Linux.
- Annatar 11y ago
  If you want the JustWorks(SM), you have to get hardware which is on the Joyent engineering bill of materials' list, or research and buy hardware which illumos has drivers for. http://eng.joyent.com/manufacturing/bom.html http://eng.joyent.com/manufacturing/bom.html SmartOS being descended from Solaris has detailed manual pages built into the operating system, because when one runs a real UNIX, one is entitled to comprehensive documentation with lots of examples. A lot of the Oracle Solaris documentation, which is also very comprehensive, applies to and can be used on illumos derivatives. The SmartOS wiki and the smartos-discuss mailing list are good sources of SmartOS specific enhancements. Did I mention the built-in man pages yet?
- RantyDave 11y ago

dyn 11y ago

Hi author here. Thanks! It was a lot of hard work, I'm glad it's being well received and seems like people are generally finding it useful- the main point of writing it. I would have liked to talk a lot more about other non-Linux solutions, but that's where most of the NCC Group consulting work has been, which was a main driver for why this info needs to be out there (companies not understanding container security or not "turning it up to 11" in places it needs to be). My background is also mainly Linux, so I stuck to what I know. Hopefully someone else will release a similar paper on non-Linux solutions/unikernels/hybrid platforms?

JamesSwift 11y ago

As an ignorant spectator I appreciate the balance provided by your comment. However, I wish you would have provided some concrete examples of which wheels they are reinventing, and how those wheels exist in other solutions.

X86BSD 11y ago
Listen to this discussion[0] which highlights the point I am making. I set the marker for the relevant "container" comment but watch the entire thing to understand more fully what I and others are saying and why. [0] https://youtu.be/Ya6h2zKlpaQ?t=1h8m11s https://youtu.be/Ya6h2zKlpaQ?t=1h8m11s
i4k 11y ago
Container technology is grounded on namespaces and cgroups. Both technologies existed long time ago. Linux namespace is a very limited implementation of Plan9's namespace. Solaris and FreeBSD, to cite only the most famous, have resource limits database for process groups.