7 ms·

Containers and Docker: how secure are they?

117 points by jpetazzo 13y ago

jpetazzo 13y ago
By the way, if anyone knows of a documented exploit for LXC, I would love to hear about it. People (generally advocating VMs, zones, jails, OpenVZ...) will often say that "containers are not secure", but once you've taken some basic steps (like locking down kernel caps and device access) it becomes difficult to find an actual threat.
- FooBarWidget 13y ago
  Well, there's been a number of local root exploits in recent years. That's worrying.
- foobarqux 13y ago
  Is there a guide for these "basic steps" for the uninitiated?
- contingencies 13y ago
  I would love to hear about it See http://blog.zx2c4.com/749 http://blog.zx2c4.com/749 and http://bit.ly/T9CkqJ http://bit.ly/T9CkqJ
  - ak217 13y ago
    Neither of these exploits works on stock Ubuntu 12.04 LTS, with LXC or otherwise (AppArmor kicks in). Like jpetazzo, I would love to see a working LXC exploit. In my case, "working" == "can get host root when given container root on Ubuntu 12.04 or later".
    - liuw 13y ago
      The fact is that by the time you know about the "working" exploit it's already been fixed. Unless you're security-related researchers / engineers you're not very likely to get hold of a 0-day exploit.
  - jevinskie 13y ago
    Thank you so much for the first link! The very same "very black unix domain sockets magic" has been confounding me while reverse engineering a binary. OK, it calls recvmsg and then a wild FD appears from another process!? I had no idea...
- deleted 13y ago
  [deleted]
- justincormack 13y ago
  Any local root vulnerability will also work in a container, eg this one http://www.ubuntu.com/usn/usn-1914-1/ http://www.ubuntu.com/usn/usn-1914-1/ - note that a lot of kernel vulnerabilities are never really announced, just quietly fixed.
- mjg59 13y ago
  A variant on http://grsecurity.net/~spender/msr32.c http://grsecurity.net/~spender/msr32.c would have worked up until the capability check was added. Capabilities will help you, but if your business model is built on the assumption that the kernel performs capabilities checks everywhere it should then you really ought to be actively reviewing kernel entry points yourself.
- liuw 13y ago
  Do they count? http://lwn.net/Articles/543509/ http://lwn.net/Articles/543509/ http://lwn.net/Articles/543273/ http://lwn.net/Articles/543273/
pacala 13y ago
Any 0-day Linux root vulnerability qualifies. Linux is a large system, do your own risk analysis.
- jpetazzo 13y ago
  Are you implying that gaining root access inside a LXC container means that you can escalate to the host system, or to sibling containers? If yes, I would like to see an example of that (that works on systems with very minimal lockdown, i.e. using the device control group and kernel capabilities). Otherwise, if you just mean that "0-day Linux root vulnerabilities can be used to escalate from non-root to root in a Linux Containers", that's a truism, and it also stands true for VMs or OpenVZ systems. Just like 0-day vulnerabilities will help people to escalate from non-root to root in a FreeBSD jail or Solaris zone.
  - teraflop 13y ago
    No, I think the implication is that the kinds of kernel bugs that allow you to escalate from a non-root user to root within a container (by corrupting kernel data structures, for instance) will probably also allow you to escalate to root at the host level. If you can change the UID of a process, why should it be harder to change the UID namespace as well? Plus, the user namespace functionality is fairly new and complex, and there have already been a few bugs found, e.g. [1]. I assume all the known bugs have been fixed, but that doesn't ensure that more aren't lurking somewhere. [1] http://lwn.net/Articles/543273/ http://lwn.net/Articles/543273/
  - contingencies 13y ago
    I would like to see an example of that... http://blog.zx2c4.com/749 http://blog.zx2c4.com/749 and http://bit.ly/T9CkqJ http://bit.ly/T9CkqJ
    - yebyen 13y ago
      Those are great exploits, but they don't show how to escape from a container at all.
dap 13y ago
Good post, except that it's extremely misleading to use Solaris as the canonical example of non-Linux containers and then say that non-Linux containers "haven't had as much exposure" and "the source code isn't always available for peer review and auditing". Solaris containers (in Solaris first, and then illumos when Solaris became closed-source again) have been open source since 2005 and running in hostile production environments that whole time.
- pjmlp 13y ago
  I only used containers in HP-UX back in 2000, and back then they were pretty much configured in all HP-UX boxes I had access to.
- jpetazzo 13y ago
  True, I will update the blog post so that it feels less misleading. Source code for Solaris zones is indeed available; but I wouldn't consider it widely deployed. Of course, some people are using it in public hosting environments (the most notable example is probably Joyent); but I don't think that it's significant compared to the installed base of VServer, OpenVZ, or LXC out there. I mean — it's trivially easy to get access to a Linux VPS, for a ridiculously low price (sometimes for free). Now compare with something equivalent based on Solaris zones. But yeah, I'll definitely update the blog post, thanks!
  - bcantrill 13y ago
    Source code for Solaris zones is indeed available; but I wouldn't consider it widely deployed. Why would you not consider it widely deployed? If it needs to be said: just because you don't use something doesn't mean that others aren't. Speaking only for us (I work for Joyent), we have deployed hundreds of thousands of zones into production over the years -- and Joyent was running with FreeBSD jails before that. And that's just us; there are many others in the illumos/SmartOS/OmniOS, Solaris and FreeBSD communities who have been running this technology in production -- broadly -- for years. Perhaps OS virtualization is a new technology for you, but understand that it's not new for everyone; some of us have been doing this for a while -- widely deployed and in production.
contingencies 13y ago
I support docker in its efforts. However, docker is too cute, too hyped, and too rapidly developed to trust with your security as yet. Quite frankly, you have to understand a bit more than how to call an API to have faith in your infrastructure's inherent security. For example, in this article the author links to the 'list of dropped capabilities in the Docker code'. As it happens, I wrote that list quite some time ago, and wrote it for lxc-gentoo, a guest-generation script for raw LXC against an earlier kernel version with an earlier LXC userspace. Not only is the list now out of date, it's no longer using the preferred approach. Why is this? Instead of explicit drop ("allow all, deny some") after some months of raising the issue one of the LXC devs finally added the 'lxc.keep' (ie. "deny all, allow some") which is architecturally more secure against things like kernel upgrades which add or modify kernel capabilities. Furthermore, the docker people only included this when I added https://github.com/dotcloud/docker/commits/v0.5.0/lxc_template.go?author=globalcitizen https://github.com/dotcloud/docker/commits/v0.5.0/lxc_templa... ... things as important as WARNING: procfs is a known attack vector and should probably be disabled if your userspace allows it. eg. see http://blog.zx2c4.com/749 http://blog.zx2c4.com/749 and WARNING: sysfs is a known attack vector and should probably be disabled if your userspace allows it. eg. see http://bit.ly/T9CkqJ http://bit.ly/T9CkqJ Again, I fully support docker's efforts but the article is ... misleading at best.
- shykes 13y ago
  Hi, docker maintainer here. There's a reason we keep saying docker is not yet production-ready. Right now our focus is on usability and stabilizing the management API to make deployment-centric deployment awesome. You can be sure that before we tell anyone that they can use docker to sandbox untrusted code in a shared environment (which by the way is not the only use case of docker) we will be locking down our default lxc configuration and doing a sweep of all pending security issues. For the record, we (dotCloud) have tens of thousands of lxc containers currently running untrusted code in production on shared infrastructure, and have had to monitor and maintain them 24/7 for several years. Before that we ran openvz. And before that, we ran vserver. So while docker itself may not yet be ready for production (and indeed we don't use it in production at dotcloud either), you don't need to worry about our stance on security. We care about it just as much as you do.
deleted 13y ago
[deleted]
SkyMarshal 13y ago
>Finally, if you run Docker on a server, it is recommended to run exclusively Docker in the server, and move all other services within containers controlled by Docker. This looks like what CoreOS is providing, a stripped down barebones host, with all other services not strictly necessary in the host moved to the containers. >Capabilities turn the binary “root/non-root” dichotomy into a fine-grained access control system. Processes (like web servers) that just need to bind on a port below 1024 do not have to run as root: they can just be granted the net_bind_service capability instead. And there are many other capabilities, for almost all the specific areas where root privileges are usually needed. This is awesome, has been a personal pain point in the past, trying to get JVM running as non-root in ubuntu server. Theoretically it's easy with IPTABLEs, but in practice it can be tricky to get working exactly right.
- secstate 13y ago
  To your first point, CoreOS is SmartOS for linux :)
AYBABTME 13y ago
I'm very interested in all those things, but I clearly lack a trajectory for learning them. Is there a reference I could read or a 'name' for that domain? How does one become educated on these things? So far I've grabbed knowledge by reading paper on operating systems (and misunderstanding 80% of their content), reading man pages, reading Tanenbaum's textbooks, etc. But still I don't feel like I know or understand. They say a lack of words for things render one blinds of their ignorance. Sometimes it's also that you just don't know what needs to be learnt.
gouggoug 13y ago
"No exploit has been crafted yet to demonstrate this, but it will certainly happen in the feature". But will it be considered a future? ;)

117 points by jpetazzo 13y ago

By the way, if anyone knows of a documented exploit for LXC, I would love to hear about it. People (generally advocating VMs, zones, jails, OpenVZ...) will often say that "containers are not secure", but once you've taken some basic steps (like locking down kernel caps and device access) it becomes difficult to find an actual threat.

FooBarWidget 13y ago
Well, there's been a number of local root exploits in recent years. That's worrying.
foobarqux 13y ago
Is there a guide for these "basic steps" for the uninitiated?
contingencies 13y ago
I would love to hear about it See http://blog.zx2c4.com/749 http://blog.zx2c4.com/749 and http://bit.ly/T9CkqJ http://bit.ly/T9CkqJ
- ak217 13y ago
  Neither of these exploits works on stock Ubuntu 12.04 LTS, with LXC or otherwise (AppArmor kicks in). Like jpetazzo, I would love to see a working LXC exploit. In my case, "working" == "can get host root when given container root on Ubuntu 12.04 or later".
  - liuw 13y ago
    The fact is that by the time you know about the "working" exploit it's already been fixed. Unless you're security-related researchers / engineers you're not very likely to get hold of a 0-day exploit.
- jevinskie 13y ago
  Thank you so much for the first link! The very same "very black unix domain sockets magic" has been confounding me while reverse engineering a binary. OK, it calls recvmsg and then a wild FD appears from another process!? I had no idea...
deleted 13y ago
[deleted]
justincormack 13y ago
Any local root vulnerability will also work in a container, eg this one http://www.ubuntu.com/usn/usn-1914-1/ http://www.ubuntu.com/usn/usn-1914-1/ - note that a lot of kernel vulnerabilities are never really announced, just quietly fixed.
mjg59 13y ago
A variant on http://grsecurity.net/~spender/msr32.c http://grsecurity.net/~spender/msr32.c would have worked up until the capability check was added. Capabilities will help you, but if your business model is built on the assumption that the kernel performs capabilities checks everywhere it should then you really ought to be actively reviewing kernel entry points yourself.
liuw 13y ago
Do they count? http://lwn.net/Articles/543509/ http://lwn.net/Articles/543509/ http://lwn.net/Articles/543273/ http://lwn.net/Articles/543273/

pacala 13y ago

Any 0-day Linux root vulnerability qualifies. Linux is a large system, do your own risk analysis.

jpetazzo 13y ago
Are you implying that gaining root access inside a LXC container means that you can escalate to the host system, or to sibling containers? If yes, I would like to see an example of that (that works on systems with very minimal lockdown, i.e. using the device control group and kernel capabilities). Otherwise, if you just mean that "0-day Linux root vulnerabilities can be used to escalate from non-root to root in a Linux Containers", that's a truism, and it also stands true for VMs or OpenVZ systems. Just like 0-day vulnerabilities will help people to escalate from non-root to root in a FreeBSD jail or Solaris zone.
- teraflop 13y ago
  No, I think the implication is that the kinds of kernel bugs that allow you to escalate from a non-root user to root within a container (by corrupting kernel data structures, for instance) will probably also allow you to escalate to root at the host level. If you can change the UID of a process, why should it be harder to change the UID namespace as well? Plus, the user namespace functionality is fairly new and complex, and there have already been a few bugs found, e.g. [1]. I assume all the known bugs have been fixed, but that doesn't ensure that more aren't lurking somewhere. [1] http://lwn.net/Articles/543273/ http://lwn.net/Articles/543273/
- contingencies 13y ago
  I would like to see an example of that... http://blog.zx2c4.com/749 http://blog.zx2c4.com/749 and http://bit.ly/T9CkqJ http://bit.ly/T9CkqJ
  - yebyen 13y ago
    Those are great exploits, but they don't show how to escape from a container at all.

dap 13y ago

Good post, except that it's extremely misleading to use Solaris as the canonical example of non-Linux containers and then say that non-Linux containers "haven't had as much exposure" and "the source code isn't always available for peer review and auditing". Solaris containers (in Solaris first, and then illumos when Solaris became closed-source again) have been open source since 2005 and running in hostile production environments that whole time.

pjmlp 13y ago
I only used containers in HP-UX back in 2000, and back then they were pretty much configured in all HP-UX boxes I had access to.
jpetazzo 13y ago
True, I will update the blog post so that it feels less misleading. Source code for Solaris zones is indeed available; but I wouldn't consider it widely deployed. Of course, some people are using it in public hosting environments (the most notable example is probably Joyent); but I don't think that it's significant compared to the installed base of VServer, OpenVZ, or LXC out there. I mean — it's trivially easy to get access to a Linux VPS, for a ridiculously low price (sometimes for free). Now compare with something equivalent based on Solaris zones. But yeah, I'll definitely update the blog post, thanks!
- bcantrill 13y ago
  Source code for Solaris zones is indeed available; but I wouldn't consider it widely deployed. Why would you not consider it widely deployed? If it needs to be said: just because you don't use something doesn't mean that others aren't. Speaking only for us (I work for Joyent), we have deployed hundreds of thousands of zones into production over the years -- and Joyent was running with FreeBSD jails before that. And that's just us; there are many others in the illumos/SmartOS/OmniOS, Solaris and FreeBSD communities who have been running this technology in production -- broadly -- for years. Perhaps OS virtualization is a new technology for you, but understand that it's not new for everyone; some of us have been doing this for a while -- widely deployed and in production.

contingencies 13y ago

I support docker in its efforts. However, docker is too cute, too hyped, and too rapidly developed to trust with your security as yet. Quite frankly, you have to understand a bit more than how to call an API to have faith in your infrastructure's inherent security. For example, in this article the author links to the 'list of dropped capabilities in the Docker code'. As it happens, I wrote that list quite some time ago, and wrote it for lxc-gentoo, a guest-generation script for raw LXC against an earlier kernel version with an earlier LXC userspace. Not only is the list now out of date, it's no longer using the preferred approach. Why is this? Instead of explicit drop ("allow all, deny some") after some months of raising the issue one of the LXC devs finally added the 'lxc.keep' (ie. "deny all, allow some") which is architecturally more secure against things like kernel upgrades which add or modify kernel capabilities. Furthermore, the docker people only included this when I added https://github.com/dotcloud/docker/commits/v0.5.0/lxc_template.go?author=globalcitizen https://github.com/dotcloud/docker/commits/v0.5.0/lxc_templa... ... things as important as WARNING: procfs is a known attack vector and should probably be disabled if your userspace allows it. eg. see http://blog.zx2c4.com/749 http://blog.zx2c4.com/749 and WARNING: sysfs is a known attack vector and should probably be disabled if your userspace allows it. eg. see http://bit.ly/T9CkqJ http://bit.ly/T9CkqJ Again, I fully support docker's efforts but the article is ... misleading at best.

shykes 13y ago
Hi, docker maintainer here. There's a reason we keep saying docker is not yet production-ready. Right now our focus is on usability and stabilizing the management API to make deployment-centric deployment awesome. You can be sure that before we tell anyone that they can use docker to sandbox untrusted code in a shared environment (which by the way is not the only use case of docker) we will be locking down our default lxc configuration and doing a sweep of all pending security issues. For the record, we (dotCloud) have tens of thousands of lxc containers currently running untrusted code in production on shared infrastructure, and have had to monitor and maintain them 24/7 for several years. Before that we ran openvz. And before that, we ran vserver. So while docker itself may not yet be ready for production (and indeed we don't use it in production at dotcloud either), you don't need to worry about our stance on security. We care about it just as much as you do.

deleted 13y ago

[deleted]

SkyMarshal 13y ago

>Finally, if you run Docker on a server, it is recommended to run exclusively Docker in the server, and move all other services within containers controlled by Docker. This looks like what CoreOS is providing, a stripped down barebones host, with all other services not strictly necessary in the host moved to the containers. >Capabilities turn the binary “root/non-root” dichotomy into a fine-grained access control system. Processes (like web servers) that just need to bind on a port below 1024 do not have to run as root: they can just be granted the net_bind_service capability instead. And there are many other capabilities, for almost all the specific areas where root privileges are usually needed. This is awesome, has been a personal pain point in the past, trying to get JVM running as non-root in ubuntu server. Theoretically it's easy with IPTABLEs, but in practice it can be tricky to get working exactly right.

secstate 13y ago
To your first point, CoreOS is SmartOS for linux :)

AYBABTME 13y ago

I'm very interested in all those things, but I clearly lack a trajectory for learning them. Is there a reference I could read or a 'name' for that domain? How does one become educated on these things? So far I've grabbed knowledge by reading paper on operating systems (and misunderstanding 80% of their content), reading man pages, reading Tanenbaum's textbooks, etc. But still I don't feel like I know or understand. They say a lack of words for things render one blinds of their ignorance. Sometimes it's also that you just don't know what needs to be learnt.

gouggoug 13y ago

"No exploit has been crafted yet to demonstrate this, but it will certainly happen in the feature". But will it be considered a future? ;)