27 ms·

Leaky Vessels flaws allow hackers to escape Docker, runc containers

63 points by el_duderino 3y ago

j16sdiz 3y ago
I have seen this for ages and thought it was some design limitation. Didn't attempt to report it as vulneribility ...
- rdegges 3y ago
  oO This one is a clear vulnerability -- if you're able to escape a container and access the host system (usually with root access since Docker is running as root most places), that's a big issue.
  - px43 3y ago
    Last I checked, Docker wasn't considered a security boundary and Docker escapes weren't considered a security issue. The most obvious escapes have slowly been mitigated over time, but the prevailing wisdom is that you should always assume that an attacker can work their way from the container to the host system. Obviously never put anything sensitive on a docker host, and assume anything running on the host (like the other docker containers) is running at the same privilege level.
    - nyrikki 3y ago
      I tried for a long time to even get docker to just give a config option to disable privileged mode with no success. I even resorted to posting instructions on stack exchange demonstrating how to walk /sys to find device numbers and use mknod to read the hosts root volume and now as most servers don't mount their efi partition, how you could mount it rw inside a container with little effort. Containers are namespaces and purely depend on privilege dropping for the security they provide. Part of the problem is that container breakout is narrowly defined. The fact that a privileged container can upload firmware or access private keys by reading the hosts root volume didn't count. While the ability to disable privileged mode wouldn't have solved this issue it still would have reduced the attack surface. But will the projects refusal to even take that step I gave up. Deciding that the only safe option was to consider containers as namespaces and nothing more. Unfortunately adding persistence and other functionality tends to result people running it as uid0, which means that you have to consider as anything that can launch a container as having superuser privileges.
rdegges 3y ago
If you want more info, we have a detailed blog post on the Snyk website here: https://snyk.io/blog/leaky-vessels-docker-runc-container-breakout-vulnerabilities/ https://snyk.io/blog/leaky-vessels-docker-runc-container-bre... (we also built two open source detection tools you can use to see if anyone is taking advantage of these issues in your infrastructure): a static analysis tool and a runtime tool using eBPF.
WhatIsDukkha 3y ago
Podman uses crun by default but will use runc if unavailable. Buildkit doesn't look too useful(?) with podman as some of its better features are built in to podman.
flaminHotSpeedo 3y ago
Friendly reminder that containers aren't a security control and your threat modeling should already assume the host VM running any untrusted containers is compromised
- mrweasel 3y ago
  Multi-tenancy, in any shape or form is sadly a potential security problem. I don't care if it's VM, containers, VLANs or databases, if you absolutely need to know that no one else is going to have access to your data, you need separate physical hardware. In many circumstances you can make some tradeoffs, like running multiple customers or projects on the same vmware installation, because the risk is low or the other tenants are well known. I don't think Kubernetes is quite there yet where I'd trust multiple tenants on the same cluster.
  - flaminHotSpeedo 3y ago
    Technically yes, but practically no. If you have the expertise to do it well, or cash to pay for someone else to do it well for you, you can definitely get trustworthy security boundaries without physical separation. As an example, a VM from a reputable cloud provider is secure enough for most purposes, likewise I'd trust managed containers from a reputable cloud provider (except Azure, they have a dodgy track record when it comes to container security)
  - nyrikki 3y ago
    As an example of the difference directly related to k8s "Security settings that you specify for a Container apply only to the individual Container, and they override settings made at the Pod level when there is overlap." https://kubernetes.io/docs/tasks/configure-pod-container/security-context/ https://kubernetes.io/docs/tasks/configure-pod-container/sec... While there are other Byzantine faults like this CSV, that should give you a hint. The problem is that containers depend on good actors dropping privlages on a shared kernel. Where VM's have a natural abstraction and their own kernel, containers require proper configuration of seccomp, apparmor, selinux, dropping privileges, etc... to reduce the attack surface. An easy demonstration of this is to launch a privlaged container and run insmod with some random kernel module on the host OS. In namespaces, you start with no isolation, and you add whatever you want or more realistically remember to add. They are not jails where you start with a reasonable secure baseline. In fact the k8s 'baseline' pod security standard still has MKNOD caps which can result in issues like CVE-2021-25741. While /proc now has filtering /sys doesn't and you can find the major minor numbers of the hosts root filesystem and read it from a container. It is not that containers can't be made reasonably secure, but that they aren't inherently secure. They are default allow all because they are inherently just namespaces implemented basically through value remapping.
fbdab103 3y ago
Just this week there was a story[0] with a subthread about how Docker requires you to mount a volume, so it is better than launching shell scripts directly. I know all software has bugs, but Docker did not prioritize security and is unfit for isolation of untrusted code. I will continue to stick with VMs as my security boundary. [0] https://news.ycombinator.com/item?id=39240471 https://news.ycombinator.com/item?id=39240471
- mrweasel 3y ago
  Just pointing out the most VM / virtualization platforms have also had security problems where an attacker could escape the VM and access the underlying operating system.
  - flaminHotSpeedo 3y ago
    The major difference is that VM's are intended to be a security boundary as a primary function, and give you tools like hardware virtualization or enforcing single tenancy for a given NUMA node to reduce the attack surface.
  - tptacek 3y ago
    KVM vulnerabilities are rare compared to Linux kernel LPEs, which makes sense, because KVM is a relatively small part of the Linux kernel compared to the system call and driver attack surface. Containers are great, but shared-kernel security isolation of standard Unix programs doesn't work.
- hipadev23 3y ago
  Maybe I’m in the minority here, but I use containers for ease of packaging and deployment, security isn’t really my expectation for the container as I also rely on VMs for that aspect. I have a handful of long-running VMs with docker engine installed, and that serves as my application platform.
  - flaminHotSpeedo 3y ago
    Definitely not the minority, anyone who needs secure isolation for a container should be wrapping it in a single tenant VM
akaike 3y ago
That's why it's better to run a docker container inside a micro vm I guess.
- 3abiton 3y ago
  Why not just nix it?
Wool2662 3y ago
Frankly if a user can run the docker command he is basically root anyways. That's why you do not add random users to the docker group.
- westurner 3y ago
  Podman (and Docker) do rootless containers, which would limit privilege escalation from this vuln, too IIUC Does container-selinux limit this container escape vulnerability?

63 points by el_duderino 3y ago

j16sdiz 3y ago

I have seen this for ages and thought it was some design limitation. Didn't attempt to report it as vulneribility ...

rdegges 3y ago
oO This one is a clear vulnerability -- if you're able to escape a container and access the host system (usually with root access since Docker is running as root most places), that's a big issue.
- px43 3y ago
  Last I checked, Docker wasn't considered a security boundary and Docker escapes weren't considered a security issue. The most obvious escapes have slowly been mitigated over time, but the prevailing wisdom is that you should always assume that an attacker can work their way from the container to the host system. Obviously never put anything sensitive on a docker host, and assume anything running on the host (like the other docker containers) is running at the same privilege level.
  - nyrikki 3y ago
    I tried for a long time to even get docker to just give a config option to disable privileged mode with no success. I even resorted to posting instructions on stack exchange demonstrating how to walk /sys to find device numbers and use mknod to read the hosts root volume and now as most servers don't mount their efi partition, how you could mount it rw inside a container with little effort. Containers are namespaces and purely depend on privilege dropping for the security they provide. Part of the problem is that container breakout is narrowly defined. The fact that a privileged container can upload firmware or access private keys by reading the hosts root volume didn't count. While the ability to disable privileged mode wouldn't have solved this issue it still would have reduced the attack surface. But will the projects refusal to even take that step I gave up. Deciding that the only safe option was to consider containers as namespaces and nothing more. Unfortunately adding persistence and other functionality tends to result people running it as uid0, which means that you have to consider as anything that can launch a container as having superuser privileges.

rdegges 3y ago

If you want more info, we have a detailed blog post on the Snyk website here: https://snyk.io/blog/leaky-vessels-docker-runc-container-breakout-vulnerabilities/ https://snyk.io/blog/leaky-vessels-docker-runc-container-bre... (we also built two open source detection tools you can use to see if anyone is taking advantage of these issues in your infrastructure): a static analysis tool and a runtime tool using eBPF.

WhatIsDukkha 3y ago

Podman uses crun by default but will use runc if unavailable. Buildkit doesn't look too useful(?) with podman as some of its better features are built in to podman.

flaminHotSpeedo 3y ago

Friendly reminder that containers aren't a security control and your threat modeling should already assume the host VM running any untrusted containers is compromised

mrweasel 3y ago
Multi-tenancy, in any shape or form is sadly a potential security problem. I don't care if it's VM, containers, VLANs or databases, if you absolutely need to know that no one else is going to have access to your data, you need separate physical hardware. In many circumstances you can make some tradeoffs, like running multiple customers or projects on the same vmware installation, because the risk is low or the other tenants are well known. I don't think Kubernetes is quite there yet where I'd trust multiple tenants on the same cluster.
- flaminHotSpeedo 3y ago
  Technically yes, but practically no. If you have the expertise to do it well, or cash to pay for someone else to do it well for you, you can definitely get trustworthy security boundaries without physical separation. As an example, a VM from a reputable cloud provider is secure enough for most purposes, likewise I'd trust managed containers from a reputable cloud provider (except Azure, they have a dodgy track record when it comes to container security)
- nyrikki 3y ago
  As an example of the difference directly related to k8s "Security settings that you specify for a Container apply only to the individual Container, and they override settings made at the Pod level when there is overlap." https://kubernetes.io/docs/tasks/configure-pod-container/security-context/ https://kubernetes.io/docs/tasks/configure-pod-container/sec... While there are other Byzantine faults like this CSV, that should give you a hint. The problem is that containers depend on good actors dropping privlages on a shared kernel. Where VM's have a natural abstraction and their own kernel, containers require proper configuration of seccomp, apparmor, selinux, dropping privileges, etc... to reduce the attack surface. An easy demonstration of this is to launch a privlaged container and run insmod with some random kernel module on the host OS. In namespaces, you start with no isolation, and you add whatever you want or more realistically remember to add. They are not jails where you start with a reasonable secure baseline. In fact the k8s 'baseline' pod security standard still has MKNOD caps which can result in issues like CVE-2021-25741. While /proc now has filtering /sys doesn't and you can find the major minor numbers of the hosts root filesystem and read it from a container. It is not that containers can't be made reasonably secure, but that they aren't inherently secure. They are default allow all because they are inherently just namespaces implemented basically through value remapping.

fbdab103 3y ago

Just this week there was a story[0] with a subthread about how Docker requires you to mount a volume, so it is better than launching shell scripts directly. I know all software has bugs, but Docker did not prioritize security and is unfit for isolation of untrusted code. I will continue to stick with VMs as my security boundary. [0] https://news.ycombinator.com/item?id=39240471 https://news.ycombinator.com/item?id=39240471

mrweasel 3y ago
Just pointing out the most VM / virtualization platforms have also had security problems where an attacker could escape the VM and access the underlying operating system.
- flaminHotSpeedo 3y ago
  The major difference is that VM's are intended to be a security boundary as a primary function, and give you tools like hardware virtualization or enforcing single tenancy for a given NUMA node to reduce the attack surface.
- tptacek 3y ago
  KVM vulnerabilities are rare compared to Linux kernel LPEs, which makes sense, because KVM is a relatively small part of the Linux kernel compared to the system call and driver attack surface. Containers are great, but shared-kernel security isolation of standard Unix programs doesn't work.
hipadev23 3y ago
Maybe I’m in the minority here, but I use containers for ease of packaging and deployment, security isn’t really my expectation for the container as I also rely on VMs for that aspect. I have a handful of long-running VMs with docker engine installed, and that serves as my application platform.
- flaminHotSpeedo 3y ago
  Definitely not the minority, anyone who needs secure isolation for a container should be wrapping it in a single tenant VM

akaike 3y ago

That's why it's better to run a docker container inside a micro vm I guess.

3abiton 3y ago
Why not just nix it?

Wool2662 3y ago

Frankly if a user can run the docker command he is basically root anyways. That's why you do not add random users to the docker group.

westurner 3y ago
Podman (and Docker) do rootless containers, which would limit privilege escalation from this vuln, too IIUC Does container-selinux limit this container escape vulnerability?