19 ms·

CVE-2019-5736: runc container breakout

240 points by afshinmeh 8y ago

CaliforniaKarl 8y ago
Red Hat’s page on the vulnerability: https://access.redhat.com/security/vulnerabilities/runcescape https://access.redhat.com/security/vulnerabilities/runcescap... RH CVE page, with the vulnerability’s metrics and the list of RH packages affected (plus links to the errata pages that have details on fixed builds): https://access.redhat.com/security/cve/cve-2019-5736 https://access.redhat.com/security/cve/cve-2019-5736
gr2020 8y ago
Looks like Docker 18.09.2 was released a few minutes ago to address this: https://github.com/docker/docker-ce/releases https://github.com/docker/docker-ce/releases
- el_duderino 8y ago
  PoC here: https://github.com/feexd/pocs/blob/master/CVE-2019-5736/exploit.c https://github.com/feexd/pocs/blob/master/CVE-2019-5736/expl...
koolba 8y ago
Is this issue specific to containers running as root?
- cyphar 8y ago
  Yes. You need to be able to run a container as root (or rather, as a user which has write access to the host runc binary -- which is usually root). User namespaces protect you for this reason.
deleted 8y ago
[deleted]
- deleted 8y ago
  [deleted]
CaliforniaKarl 8y ago
Debian’s security tracker, showing the affected versions, and (when available) the fixed versions: https://security-tracker.debian.org/tracker/CVE-2019-5736 https://security-tracker.debian.org/tracker/CVE-2019-5736 And Ubuntu’s: https://people.canonical.com/~ubuntu-security/cve/2019/CVE-2019-5736.html https://people.canonical.com/~ubuntu-security/cve/2019/CVE-2... Personally, I like these vs. RHEL, since all the info is on page.
- dfc 8y ago
  In addition to the information all being on one page you can also: git clone https://salsa.debian.org/security-tracker-team/security-tracker.git git clone https://git.launchpad.net/ubuntu-cve-tracker There is a lot of interesting things you can do with the data.
darren0 8y ago
The best fix is to upgrade to 18.09.2. For those that can't do that immediately, backported versions of runc for Docker releases going back to 1.12.6 are available from Rancher at https://github.com/rancher/runc-cve https://github.com/rancher/runc-cve. But please only do that as a temporary workaround until you can properly upgrade to 18.09.2. Please patch if you don't 100% trust all users on your host.
- peterwwillis 8y ago
  For systems where you already enforce security policies, updating policy is faster than upgrading software, due to fewer build processes, quality control issues, and potential side-effects. If you are using SELinux, verify your containers are running as container_t. If not, verify you are using user namespaces that don't map host root into the container user's namespace. These should mitigate the issue. (as far as trust goes, just don't trust any local users. there's too many ways to privesc on Linux, and SELinux is the only thing that stops most of them)
  - geofft 8y ago
    I believe that if you're in an environment where users don't have (and can't gain) root inside the container, you're also fine, and if that's your theoretical policy, getting to the point where you 100% enforce that might also be easier than patching.
- justincormack 8y ago
  We also released Docker 18.06.2 with the fix, as a lot of Kubernetes users are on this release.
  - olemartinorg 8y ago
    The new package in Ubuntu Trusty seems to be broken. Not that Trusty is supported for much longer. See https://github.com/docker/for-linux/issues/591 https://github.com/docker/for-linux/issues/591
pizlonator 8y ago
Yikes that's a big patch! Just on a meta-level, security vulnerabilities fixed with big patches are usually the least fun. Also, I would bet that freshly written C code has about 1 RCE bug every 100 LoC. This patch has 236 LoCs so probably about 2.36 RCE's.
- loeg 8y ago
  1 RCE bug per 100 LoC feels like a very high bet. How did you arrive at that number? I've heard 1 bug per 100 LoC before, but RCEs are a very specific subset of bugs.
  - pizlonator 8y ago
    Yeah, I was joking. But not completely. Even if you leave a codebase alone in the sense that you only fix security bugs, you’ll end up with a slow trickle that never quite ends. There are the RCEs that get reported plus a bunch that don’t. So, if you: - project into the future, assuming that if there still has been a trickle of bugs being found then more bugs will also still be found in the future. - take into account that there are some number of unreported bugs. Maybe for every reported one there is one that isn’t. Dunno the ratio there. Put all that together and it’s not hard to imagine a 1RCE/100LoC rate. But still I’m kinda joking. But only slightly. Maybe if I had a way to bet money on this and it was a testable bet (it’s not because of the unreported RCEs) then I’d throw some cash down.
- megous 8y ago
  By your guess each new Linux release would have about 5000 new RCEs. So that's 25000-30000 new RCEs over the last year alone.
  - ebeip90 8y ago
    Probably only off by an order of magnitude. Check out Dmitry’s szykaller slides. https://events.linuxfoundation.org/wp-content/uploads/2017/11/Syzbot-and-the-Tale-of-Thousand-Kernel-Bugs-Dmitry-Vyukov-Google.pdf https://events.linuxfoundation.org/wp-content/uploads/2017/1... Edit: I missed the “RCE” context. Most of these are just privescs or memory disclosures.
tyingq 8y ago
Is this something that non-privileged containers mitigates? Curious what the big barriers are to this. I know they exist, but aren't used widely...I assume because some functionality doesn't work.
- iwalton3 8y ago
  Yes. The lxc commit[1] states that this issue only affects privileged containers. No CVE has been issued for lxc because they consider privileged containers to be insecure. In my experience unprivileged containers work for most tasks, but there is breakage in some areas. Usually the issues are simple to resolve, like disabling OOM adjustments in systemd or changing the idmap range in winbind to be within the namespace allotment. [1] https://github.com/lxc/lxc/commit/6400238d08cdf1ca20d49bafb85f4e224348bf9d https://github.com/lxc/lxc/commit/6400238d08cdf1ca20d49bafb8...
  - brauner 8y ago
    I've also written a smallish blogpost about this CVE. I'm a LX{C,D} maintainer and I've worked with Aleksa the runC maintainer together on a fix for this CVE: https://brauner.github.io/2019/02/12/privileged-containers.html https://brauner.github.io/2019/02/12/privileged-containers.h...
    - arno1 8y ago
      Thank you @brauner for writing this blogpost! IIUC, using Docker's userns-remap would protect against this CVE by making the containers run unprivileged (container's id 0 != host's id 0) and should generally be the industry's best practice.
- cpuguy83 8y ago
  In terms of mitigation, simply running as non-root is not enough as it is very difficult to prevent escalation to root (e.g. suid binaries). User namespaces which is setup such that uid's in the container are not mapped to any in-use uids on the host is needed.
miguelmota 8y ago
For better isolation check out KataContainers: https://github.com/kata-containers/runtime https://github.com/kata-containers/runtime It's a drop-in replacement for runc. With KataContainers it runs docker containers in a lightweight VM so you get all the security benefits of a VM. The downside is slightly slower container start up times and might not work in nested virtualized environments.
- justanother- 8y ago
  Alternative idea: throw away docker and katacontainers and move to freebsd, where jails were introduced on 14 Mar 2000 (no, seriously, superior technology exists for 19 years - stable, time proven, working). Some more info: https://www.freebsd.org/doc/handbook/jails.html https://www.freebsd.org/doc/handbook/jails.html And for quick start: https://github.com/iocage/iocage https://github.com/iocage/iocage
  - jeswin 8y ago
    There have been vulnerabilities in jails previously. Also, Linux gets far more attention from exploit researchers because of wider adoption - so the number of incidents isn't a good metric. Kata has hardware isolation, so will be safer. If I have misunderstood jails and it's immune to kernel exploits please do correct me.
    - justanother- 8y ago
      Sorry, I am not going into endless debates. It is waste of my time. Check documentation, read about it and technology is here for 19 years. I dont care what anyone useses. I have just stated what the most reliable technology for compartmentization is. Docker and kata are in IT time since yesterday and there are lots of dragons still hiding. Same goes for integration with ZFS (part of freebsd since 6 April 2007).
  - peterwwillis 8y ago
    The major use case for Docker is really as a massively simplified package manager and an entrypoint for distributed applictions. Other features, like quicker runtime than VMs and system isolation, are just icing on the cake. It took a massive marketing campaign to get people to use Docker and realize it made their life easier, so something like iocage would need the same push. (Also, nobody wants to start adopting additional OSes unless absolutely necessary)
achillean 8y ago
There are nearly 4,000 exposed Docker daemons: https://www.shodan.io/report/ol761bRb https://www.shodan.io/report/ol761bRb
- spydum 8y ago
  If your dockerd is exposed without something like mutual TLS auth, RCE is a feature! Some little container breakout is irrelevant (I’m sure you know, but others might be confused)
- nineteen999 8y ago
  So inexperienced developers are just as likely to reach for Docker as experienced ones? I wonder how many of those 4,000 docker daemons are running/managing containers of dubious origin.
  - achillean 8y ago
    Around 10% of them are running cryptominers so it looks like there are already people out there compromising these public Docker instances.
NathanKP 8y ago
Amazon employee here: we have released a security bulletin covering how to update to the latest patched Docker on Amazon Linux, Amazon ECS, Amazon EKS, AWS Fargate, AWS IoT Greengrass, AWS Batch, AWS Elastic Beanstalk, AWS Cloud9, AWS SageMaker, AWS RoboMaker, and AWS Deep Learning AMI. Please check out the bulletin and update if you are using one of these services. https://aws.amazon.com/security/security-bulletins/AWS-2019-002/ https://aws.amazon.com/security/security-bulletins/AWS-2019-...
- tristanz 8y ago
  As far as I understand, EKS doesn't support PodSecurityPolicy yet so any user that can launch a pod can trivially root the host via host mounts already. This surprisingly isn't clearly documented.
  - NathanKP 8y ago
    ECS doesn't have a top level resource called "PodSecurityPolicy" but we do provide task level configuration options for all the major settings that you would normally put in your pod security policy, including including adding and dropping capabilities, privileged or unprivileged mode, docker security options for controlling SELinux or AppArmor, ulimits, sysctl settings, among others. You can find all these configuration options and more documented here: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definition_parameters.html#container_definition_security https://docs.aws.amazon.com/AmazonECS/latest/developerguide/... It is definitely possible to prevent a task running in ECS from getting root access to the host. If there is something missing that you feel we need to add to ECS to better enable this, definitely reach out and let me know!
    - tristanz 8y ago
      I'm referring to EKS here not ECS. EKS doesn't yet enable the PodSecurityPolicy admission controller, so any user that can launch a pod via EKS can root the EKS cluster regardless of RBAC rules. The main ask here is to just find a way to enable PodSecurityPolicy admission controller so that secure multi-user EKS clusters are possible like ECS.
sethvargo 8y ago
Hey all - Seth from Google here. Please let us know if you have any questions regarding GKE or questions about the upgrade process. I'm also happy to escalate any feedback to our internal product and engineering teams. Here's link to our security posting with more information and upgrade procedures: https://cloud.google.com/kubernetes-engine/docs/security-bulletins#february-11-2019-runc https://cloud.google.com/kubernetes-engine/docs/security-bul...
- sethvargo 8y ago
  One thing I want to emphasize: you are only affected if you're using Ubuntu base images for your node pools. If you're using COS, you are unaffected.
  - numbsafari 8y ago
    It would be great if folks could run COS and leverage tools like Falco. I realize there is a trade-off between having a totally locked down OS and being able to flexibly use such tools. However, Google and Sysdig announced a partnership around Falco and GCSCC integration. It would make sense that such a tool would be able to be run on COS. Perhaps I'm guilty wanting to have my cake and eat it, too. But this seems like an area where GKE and COS are somewhat limited.
    - random_copy 8y ago
      Falco can read events via an ebpf program (instead of the falco-probe kernel module). https://github.com/draios/sysdig/wiki/eBPF https://github.com/draios/sysdig/wiki/eBPF So, falco will work on GKE and COS
    - markstemm 8y ago
      Hi, Falco developer here. We do have support for running falco with an ebpf program taking the place of the kernel module. You can learn more about ebpf support at https://github.com/draios/sysdig/wiki/eBPF https://github.com/draios/sysdig/wiki/eBPF, and you should be able to run falco with ebpf by setting an environment variable SYSDIG_EBPF_PROBE="".
morpheuskafka 8y ago
Yet to see an Ubuntu Security Notice released, I'm presuming an update to the docker.io package will be released?
- alexmurray 8y ago
  docker.io and runc are in universe so not officially supported by the Ubuntu Security team so won't get a Ubuntu Security Notice
wicket 8y ago
> However, it is blocked through correct use of user namespaces (where the host root is not mapped into the container's user namespace). In other words, this won't affect anyone who understands the implications of running a process as root. Unfortunately, the sad truth is that most people I've come across who have "lots of experience" with implementing Docker containers, do not even understand the basics of how they work, let alone the implications of root access. I've interviewed candidates who claim to know Docker but can't even tell me how Docker differs from traditional virtualisation or how it achieves its isolation. The best explanation that most of them come up with is, "Docker containers are more lightweight". This sort of vulnerability should have been a non-issue but it has gained attention due to the sheer amount of incorrectly configured containers in the wild. This was an accident waiting to happen, and I doubt we've heard the last of this sort of thing.
- tyingq 8y ago
  Docker runs privileged by default, doesn't it? Seems unfair to put all the blame on the end users of it.
- cpuguy83 8y ago
  The thing is, you would have to disallow running as root in the system itself, including suid binaries and other things (e.g. docker exec -u root). User namespaces forces this. It's not just about not running as root, it's making sure the container cannot map to the real UID 0.
- geofft 8y ago
  I think it's subtler than that. It is mostly safe to run a contained process as "root" because in theory the ways that root access can be exercised is highly sandbox by the use of various namespaces, as well as things like capability restrictions (you generally don't have CAP_SYS_ADMIN or a few others), limited syscall attack surface (you generally have a syscall allowlist via seccomp-bpf), etc. Yes, it's wrong to not understand that the runc process runs as root. But I think it's only very slightly less wrong to claim that a process inside the container has root access in the way that, say, ssh root@host-system has root access. It mostly does not, and this vulnerability is notable precisely because it's one of the rare ways to exercise that root privilege outside the container. We looked at this at $work and got into a serious rabbit hole about how exactly Linux capabilities work. I think if I started asking interviewees to explain permitted vs. effective capability sets and how file and process capabilities differ, I'd never hire anyone. (And I think to figure out yourself how to "correctly configure" a container, you need to have at least some understanding of that.)
deathanatos 8y ago
Not that this shouldn't be patched and all, but this seems like it is being treated with more urgency that is required. If I am understanding the CVE correctly, you need to be able to launch privileged containers with an attacker-controlled image where the container user is root and not namespaced (i.e., the same root as the outside root user). How is this not "on the wrong side of an airtight hatch[1]"? Am I missing something here? If you can start privileged containers, why not just execute evil.exe directly? [1]: https://blogs.msdn.microsoft.com/oldnewthing/20060508-22/?p=31283 https://blogs.msdn.microsoft.com/oldnewthing/20060508-22/?p=...
- tssuser 8y ago
  Privileged containers in docker have a different meaning [1]. A lot of work has gone into trying to harden the default docker container options against container escape, even when the process is running as root. This includes dropping some capabilities, blocking syscalls with seccomp, shadowing sensitive procfs and sysfs paths, hiding most devices, and some LSM hardening [2]. Even with all that it is far more effective to just run as non-root, but hopefully that gives some context for why vulnerabilities like this are treated as high severity. [1] https://docs.docker.com/engine/reference/commandline/run/#full-container-capabilities---privileged https://docs.docker.com/engine/reference/commandline/run/#fu... [2] https://docs.docker.com/engine/security/non-events/ https://docs.docker.com/engine/security/non-events/
- tinco 8y ago
  I run a privileged container, of which I am not the author. I also know many other people run this container in privileged mode. No one is paying that person for it, and if they want to or they get compromised at some point we all might get rooted when we update the image. I think my OS is on a read only filesystem though, and maybe I've got it namespaced correctly as well, but still it's pretty dangerous.
- cpuguy83 8y ago
  "Privilged" means root in the container maps to root on the host, not literal "--privileged"
geggam 8y ago
Next year this exploit will still be in thousands if not millions of containers all over There is a distinct lack of knowledge on how to manage a system in the container ecosystem
- ec109685 8y ago
  What do you mean _in_ containers?
yujie1984 8y ago
Mesosphere employee here. We have released the product advisory on this CVE. Please check out the advisory and update your software. https://support.mesosphere.com/s/article/Known-Issue-Container-Runtime-Vulnerability-MSPH-2019-0003 https://support.mesosphere.com/s/article/Known-Issue-Contain...
wodny 8y ago
The vulnerability description seems to be lacking an explanation why the /proc/$PID/exe symlink is so special and why using the #!/proc/self/exe hashbang will work while using #!/usr/sbin/runc probably won't. Am I right that the proc filesystem in proc_exe_link() fills the file_operations struct in a way that causes open() not to go through a dereferencing procedure using the filesystem but just open the file used to run the executable?
- wodny 8y ago
  So I will answer myself. Experiments suggest it is like that: https://www.reddit.com/r/linux/comments/apmptq/cve20195736_runc_vulnerability_enabling_container/egcc313/ https://www.reddit.com/r/linux/comments/apmptq/cve20195736_r...

240 points by afshinmeh 8y ago

Red Hat’s page on the vulnerability: https://access.redhat.com/security/vulnerabilities/runcescape https://access.redhat.com/security/vulnerabilities/runcescap... RH CVE page, with the vulnerability’s metrics and the list of RH packages affected (plus links to the errata pages that have details on fixed builds): https://access.redhat.com/security/cve/cve-2019-5736 https://access.redhat.com/security/cve/cve-2019-5736

gr2020 8y ago

Looks like Docker 18.09.2 was released a few minutes ago to address this: https://github.com/docker/docker-ce/releases https://github.com/docker/docker-ce/releases

el_duderino 8y ago
PoC here: https://github.com/feexd/pocs/blob/master/CVE-2019-5736/exploit.c https://github.com/feexd/pocs/blob/master/CVE-2019-5736/expl...

koolba 8y ago

Is this issue specific to containers running as root?

cyphar 8y ago
Yes. You need to be able to run a container as root (or rather, as a user which has write access to the host runc binary -- which is usually root). User namespaces protect you for this reason.

deleted 8y ago

[deleted]

deleted 8y ago
[deleted]

CaliforniaKarl 8y ago

Debian’s security tracker, showing the affected versions, and (when available) the fixed versions: https://security-tracker.debian.org/tracker/CVE-2019-5736 https://security-tracker.debian.org/tracker/CVE-2019-5736 And Ubuntu’s: https://people.canonical.com/~ubuntu-security/cve/2019/CVE-2019-5736.html https://people.canonical.com/~ubuntu-security/cve/2019/CVE-2... Personally, I like these vs. RHEL, since all the info is on page.

dfc 8y ago
In addition to the information all being on one page you can also: git clone https://salsa.debian.org/security-tracker-team/security-tracker.git git clone https://git.launchpad.net/ubuntu-cve-tracker There is a lot of interesting things you can do with the data.

darren0 8y ago

The best fix is to upgrade to 18.09.2. For those that can't do that immediately, backported versions of runc for Docker releases going back to 1.12.6 are available from Rancher at https://github.com/rancher/runc-cve https://github.com/rancher/runc-cve. But please only do that as a temporary workaround until you can properly upgrade to 18.09.2. Please patch if you don't 100% trust all users on your host.

peterwwillis 8y ago
For systems where you already enforce security policies, updating policy is faster than upgrading software, due to fewer build processes, quality control issues, and potential side-effects. If you are using SELinux, verify your containers are running as container_t. If not, verify you are using user namespaces that don't map host root into the container user's namespace. These should mitigate the issue. (as far as trust goes, just don't trust any local users. there's too many ways to privesc on Linux, and SELinux is the only thing that stops most of them)
- geofft 8y ago
  I believe that if you're in an environment where users don't have (and can't gain) root inside the container, you're also fine, and if that's your theoretical policy, getting to the point where you 100% enforce that might also be easier than patching.
justincormack 8y ago
We also released Docker 18.06.2 with the fix, as a lot of Kubernetes users are on this release.
- olemartinorg 8y ago
  The new package in Ubuntu Trusty seems to be broken. Not that Trusty is supported for much longer. See https://github.com/docker/for-linux/issues/591 https://github.com/docker/for-linux/issues/591

pizlonator 8y ago

Yikes that's a big patch! Just on a meta-level, security vulnerabilities fixed with big patches are usually the least fun. Also, I would bet that freshly written C code has about 1 RCE bug every 100 LoC. This patch has 236 LoCs so probably about 2.36 RCE's.

loeg 8y ago
1 RCE bug per 100 LoC feels like a very high bet. How did you arrive at that number? I've heard 1 bug per 100 LoC before, but RCEs are a very specific subset of bugs.
- pizlonator 8y ago
  Yeah, I was joking. But not completely. Even if you leave a codebase alone in the sense that you only fix security bugs, you’ll end up with a slow trickle that never quite ends. There are the RCEs that get reported plus a bunch that don’t. So, if you: - project into the future, assuming that if there still has been a trickle of bugs being found then more bugs will also still be found in the future. - take into account that there are some number of unreported bugs. Maybe for every reported one there is one that isn’t. Dunno the ratio there. Put all that together and it’s not hard to imagine a 1RCE/100LoC rate. But still I’m kinda joking. But only slightly. Maybe if I had a way to bet money on this and it was a testable bet (it’s not because of the unreported RCEs) then I’d throw some cash down.
megous 8y ago
By your guess each new Linux release would have about 5000 new RCEs. So that's 25000-30000 new RCEs over the last year alone.
- ebeip90 8y ago
  Probably only off by an order of magnitude. Check out Dmitry’s szykaller slides. https://events.linuxfoundation.org/wp-content/uploads/2017/11/Syzbot-and-the-Tale-of-Thousand-Kernel-Bugs-Dmitry-Vyukov-Google.pdf https://events.linuxfoundation.org/wp-content/uploads/2017/1... Edit: I missed the “RCE” context. Most of these are just privescs or memory disclosures.

tyingq 8y ago

Is this something that non-privileged containers mitigates? Curious what the big barriers are to this. I know they exist, but aren't used widely...I assume because some functionality doesn't work.

iwalton3 8y ago
Yes. The lxc commit[1] states that this issue only affects privileged containers. No CVE has been issued for lxc because they consider privileged containers to be insecure. In my experience unprivileged containers work for most tasks, but there is breakage in some areas. Usually the issues are simple to resolve, like disabling OOM adjustments in systemd or changing the idmap range in winbind to be within the namespace allotment. [1] https://github.com/lxc/lxc/commit/6400238d08cdf1ca20d49bafb85f4e224348bf9d https://github.com/lxc/lxc/commit/6400238d08cdf1ca20d49bafb8...
- brauner 8y ago
  I've also written a smallish blogpost about this CVE. I'm a LX{C,D} maintainer and I've worked with Aleksa the runC maintainer together on a fix for this CVE: https://brauner.github.io/2019/02/12/privileged-containers.html https://brauner.github.io/2019/02/12/privileged-containers.h...
  - arno1 8y ago
    Thank you @brauner for writing this blogpost! IIUC, using Docker's userns-remap would protect against this CVE by making the containers run unprivileged (container's id 0 != host's id 0) and should generally be the industry's best practice.
cpuguy83 8y ago
In terms of mitigation, simply running as non-root is not enough as it is very difficult to prevent escalation to root (e.g. suid binaries). User namespaces which is setup such that uid's in the container are not mapped to any in-use uids on the host is needed.

miguelmota 8y ago

For better isolation check out KataContainers: https://github.com/kata-containers/runtime https://github.com/kata-containers/runtime It's a drop-in replacement for runc. With KataContainers it runs docker containers in a lightweight VM so you get all the security benefits of a VM. The downside is slightly slower container start up times and might not work in nested virtualized environments.

justanother- 8y ago
Alternative idea: throw away docker and katacontainers and move to freebsd, where jails were introduced on 14 Mar 2000 (no, seriously, superior technology exists for 19 years - stable, time proven, working). Some more info: https://www.freebsd.org/doc/handbook/jails.html https://www.freebsd.org/doc/handbook/jails.html And for quick start: https://github.com/iocage/iocage https://github.com/iocage/iocage
- jeswin 8y ago
  There have been vulnerabilities in jails previously. Also, Linux gets far more attention from exploit researchers because of wider adoption - so the number of incidents isn't a good metric. Kata has hardware isolation, so will be safer. If I have misunderstood jails and it's immune to kernel exploits please do correct me.
  - justanother- 8y ago
    Sorry, I am not going into endless debates. It is waste of my time. Check documentation, read about it and technology is here for 19 years. I dont care what anyone useses. I have just stated what the most reliable technology for compartmentization is. Docker and kata are in IT time since yesterday and there are lots of dragons still hiding. Same goes for integration with ZFS (part of freebsd since 6 April 2007).
- peterwwillis 8y ago
  The major use case for Docker is really as a massively simplified package manager and an entrypoint for distributed applictions. Other features, like quicker runtime than VMs and system isolation, are just icing on the cake. It took a massive marketing campaign to get people to use Docker and realize it made their life easier, so something like iocage would need the same push. (Also, nobody wants to start adopting additional OSes unless absolutely necessary)

achillean 8y ago

There are nearly 4,000 exposed Docker daemons: https://www.shodan.io/report/ol761bRb https://www.shodan.io/report/ol761bRb

spydum 8y ago
If your dockerd is exposed without something like mutual TLS auth, RCE is a feature! Some little container breakout is irrelevant (I’m sure you know, but others might be confused)
nineteen999 8y ago
So inexperienced developers are just as likely to reach for Docker as experienced ones? I wonder how many of those 4,000 docker daemons are running/managing containers of dubious origin.
- achillean 8y ago
  Around 10% of them are running cryptominers so it looks like there are already people out there compromising these public Docker instances.

NathanKP 8y ago

Amazon employee here: we have released a security bulletin covering how to update to the latest patched Docker on Amazon Linux, Amazon ECS, Amazon EKS, AWS Fargate, AWS IoT Greengrass, AWS Batch, AWS Elastic Beanstalk, AWS Cloud9, AWS SageMaker, AWS RoboMaker, and AWS Deep Learning AMI. Please check out the bulletin and update if you are using one of these services. https://aws.amazon.com/security/security-bulletins/AWS-2019-002/ https://aws.amazon.com/security/security-bulletins/AWS-2019-...

tristanz 8y ago
As far as I understand, EKS doesn't support PodSecurityPolicy yet so any user that can launch a pod can trivially root the host via host mounts already. This surprisingly isn't clearly documented.
- NathanKP 8y ago
  ECS doesn't have a top level resource called "PodSecurityPolicy" but we do provide task level configuration options for all the major settings that you would normally put in your pod security policy, including including adding and dropping capabilities, privileged or unprivileged mode, docker security options for controlling SELinux or AppArmor, ulimits, sysctl settings, among others. You can find all these configuration options and more documented here: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definition_parameters.html#container_definition_security https://docs.aws.amazon.com/AmazonECS/latest/developerguide/... It is definitely possible to prevent a task running in ECS from getting root access to the host. If there is something missing that you feel we need to add to ECS to better enable this, definitely reach out and let me know!
  - tristanz 8y ago
    I'm referring to EKS here not ECS. EKS doesn't yet enable the PodSecurityPolicy admission controller, so any user that can launch a pod via EKS can root the EKS cluster regardless of RBAC rules. The main ask here is to just find a way to enable PodSecurityPolicy admission controller so that secure multi-user EKS clusters are possible like ECS.

sethvargo 8y ago

Hey all - Seth from Google here. Please let us know if you have any questions regarding GKE or questions about the upgrade process. I'm also happy to escalate any feedback to our internal product and engineering teams. Here's link to our security posting with more information and upgrade procedures: https://cloud.google.com/kubernetes-engine/docs/security-bulletins#february-11-2019-runc https://cloud.google.com/kubernetes-engine/docs/security-bul...

sethvargo 8y ago
One thing I want to emphasize: you are only affected if you're using Ubuntu base images for your node pools. If you're using COS, you are unaffected.
- numbsafari 8y ago
  It would be great if folks could run COS and leverage tools like Falco. I realize there is a trade-off between having a totally locked down OS and being able to flexibly use such tools. However, Google and Sysdig announced a partnership around Falco and GCSCC integration. It would make sense that such a tool would be able to be run on COS. Perhaps I'm guilty wanting to have my cake and eat it, too. But this seems like an area where GKE and COS are somewhat limited.
  - random_copy 8y ago
    Falco can read events via an ebpf program (instead of the falco-probe kernel module). https://github.com/draios/sysdig/wiki/eBPF https://github.com/draios/sysdig/wiki/eBPF So, falco will work on GKE and COS
  - markstemm 8y ago
    Hi, Falco developer here. We do have support for running falco with an ebpf program taking the place of the kernel module. You can learn more about ebpf support at https://github.com/draios/sysdig/wiki/eBPF https://github.com/draios/sysdig/wiki/eBPF, and you should be able to run falco with ebpf by setting an environment variable SYSDIG_EBPF_PROBE="".

morpheuskafka 8y ago

Yet to see an Ubuntu Security Notice released, I'm presuming an update to the docker.io package will be released?

alexmurray 8y ago
docker.io and runc are in universe so not officially supported by the Ubuntu Security team so won't get a Ubuntu Security Notice

wicket 8y ago

> However, it is blocked through correct use of user namespaces (where the host root is not mapped into the container's user namespace). In other words, this won't affect anyone who understands the implications of running a process as root. Unfortunately, the sad truth is that most people I've come across who have "lots of experience" with implementing Docker containers, do not even understand the basics of how they work, let alone the implications of root access. I've interviewed candidates who claim to know Docker but can't even tell me how Docker differs from traditional virtualisation or how it achieves its isolation. The best explanation that most of them come up with is, "Docker containers are more lightweight". This sort of vulnerability should have been a non-issue but it has gained attention due to the sheer amount of incorrectly configured containers in the wild. This was an accident waiting to happen, and I doubt we've heard the last of this sort of thing.

tyingq 8y ago
Docker runs privileged by default, doesn't it? Seems unfair to put all the blame on the end users of it.
cpuguy83 8y ago
The thing is, you would have to disallow running as root in the system itself, including suid binaries and other things (e.g. docker exec -u root). User namespaces forces this. It's not just about not running as root, it's making sure the container cannot map to the real UID 0.
geofft 8y ago
I think it's subtler than that. It is mostly safe to run a contained process as "root" because in theory the ways that root access can be exercised is highly sandbox by the use of various namespaces, as well as things like capability restrictions (you generally don't have CAP_SYS_ADMIN or a few others), limited syscall attack surface (you generally have a syscall allowlist via seccomp-bpf), etc. Yes, it's wrong to not understand that the runc process runs as root. But I think it's only very slightly less wrong to claim that a process inside the container has root access in the way that, say, ssh root@host-system has root access. It mostly does not, and this vulnerability is notable precisely because it's one of the rare ways to exercise that root privilege outside the container. We looked at this at $work and got into a serious rabbit hole about how exactly Linux capabilities work. I think if I started asking interviewees to explain permitted vs. effective capability sets and how file and process capabilities differ, I'd never hire anyone. (And I think to figure out yourself how to "correctly configure" a container, you need to have at least some understanding of that.)

deathanatos 8y ago

Not that this shouldn't be patched and all, but this seems like it is being treated with more urgency that is required. If I am understanding the CVE correctly, you need to be able to launch privileged containers with an attacker-controlled image where the container user is root and not namespaced (i.e., the same root as the outside root user). How is this not "on the wrong side of an airtight hatch[1]"? Am I missing something here? If you can start privileged containers, why not just execute evil.exe directly? [1]: https://blogs.msdn.microsoft.com/oldnewthing/20060508-22/?p=31283 https://blogs.msdn.microsoft.com/oldnewthing/20060508-22/?p=...

tssuser 8y ago
Privileged containers in docker have a different meaning [1]. A lot of work has gone into trying to harden the default docker container options against container escape, even when the process is running as root. This includes dropping some capabilities, blocking syscalls with seccomp, shadowing sensitive procfs and sysfs paths, hiding most devices, and some LSM hardening [2]. Even with all that it is far more effective to just run as non-root, but hopefully that gives some context for why vulnerabilities like this are treated as high severity. [1] https://docs.docker.com/engine/reference/commandline/run/#full-container-capabilities---privileged https://docs.docker.com/engine/reference/commandline/run/#fu... [2] https://docs.docker.com/engine/security/non-events/ https://docs.docker.com/engine/security/non-events/
tinco 8y ago
I run a privileged container, of which I am not the author. I also know many other people run this container in privileged mode. No one is paying that person for it, and if they want to or they get compromised at some point we all might get rooted when we update the image. I think my OS is on a read only filesystem though, and maybe I've got it namespaced correctly as well, but still it's pretty dangerous.
cpuguy83 8y ago
"Privilged" means root in the container maps to root on the host, not literal "--privileged"

geggam 8y ago

Next year this exploit will still be in thousands if not millions of containers all over There is a distinct lack of knowledge on how to manage a system in the container ecosystem

ec109685 8y ago
What do you mean _in_ containers?

yujie1984 8y ago

Mesosphere employee here. We have released the product advisory on this CVE. Please check out the advisory and update your software. https://support.mesosphere.com/s/article/Known-Issue-Container-Runtime-Vulnerability-MSPH-2019-0003 https://support.mesosphere.com/s/article/Known-Issue-Contain...

wodny 8y ago

The vulnerability description seems to be lacking an explanation why the /proc/$PID/exe symlink is so special and why using the #!/proc/self/exe hashbang will work while using #!/usr/sbin/runc probably won't. Am I right that the proc filesystem in proc_exe_link() fills the file_operations struct in a way that causes open() not to go through a dereferencing procedure using the filesystem but just open the file used to run the executable?

wodny 8y ago
So I will answer myself. Experiments suggest it is like that: https://www.reddit.com/r/linux/comments/apmptq/cve20195736_runc_vulnerability_enabling_container/egcc313/ https://www.reddit.com/r/linux/comments/apmptq/cve20195736_r...