5 ms·

This is a really interesting add-on to GKE and I'm glad to see vendors starting to offer a variety of container runtimes on their platforms. That said, I'm rea

This is a really interesting add-on to GKE and I'm glad to see vendors starting to offer a variety of container runtimes on their platforms. That said, I'm really not a fan of the opening line where it references the old trope of "containers don't contain" The idea that it's trivial to break out of any Docker style container just doesn't reflect reality. Have there been vulns that allow for container breakout, sure there have, but every piece of software (including gVisor) has had vulnerabilities in it. What you can say about gVisor is that it likely presents a smaller attack surface in its default configuration than a runc style Docker container. However, of course, there's nothing to stop people tightening up on the defaults and still using runc. As an aside for anyone who thinks container breakouts are trivially easy, you can go to https://contained.af https://contained.af and win yourself some money :)

longtermsec 7y ago
The idea that it's trivial to break out of any Docker style container just doesn't reflect reality. -- not just being contrarian here, actually, the reality is that it might be trivial. and it was demonstratively trivial for a long time (see CVE-2019-5736) As for contained.af -- its not a good indicator, it mostly indicates that the reward doesnt meet the market price for demonstrating an escape from a set of hardened namespaces (which is going to cost more than an escape from "any docker container").
- raesene9 7y ago
  So the runc vuln, only applied if you were a) running as root in the container and b) hadn't enabled user namespacing. (Also for completeness, it didn't work on RHEL based distros that applied their standard SELinux policy (IIRC)) Also not specifically a Docker vulnerability, it was a runc issue which also affected other Linux containerization software (e.g. lxc) But despite all that, that's just an example of what I was talking about, all software has vulns, including runc, including gvisor. Stating that "containers don't contain" implies that it's not just a specific bug, but that architecturally the process is flawed (at least IMHO), which I would suggest is at the least an over-simplification. as to contained.af, well if it was indeed "trivial" then surely not a large reward would be required :)
  - longtermsec 7y ago
    so a) and b) are common in practice. these were not obscure boundary conditions or a corner case. and it was very trivial to exploit. "all software has vulns" is a slippery slope is my overarching point. you can't use that to say that the the security risks and isolation are comparable to gvisor. gvisor does away with a very significant amount of attack surface in the linux kernel and reimplements it in golang, which eliminates many bug classes. for a realistic risk assessment you should consider the linux kernel as a bottomless barrel of memory management bugs, which are exploitable from within a container, whereas gvisor will have a much more finite set of bugs On our team we've got extensive experience in finding compromises in this area, particularly in kernels, and that is why I am adamant that one should not think what docker provides meets the bar for best practices in a security critical environment. Something like gvisor would much more fit the bill.
    - raesene9 7y ago
      The original point I was making what that dismissing container isolation with the trope "containers don't contain" is overly simplistic, not that I thought that docker/runc containers with a default profile had as small an attack surface as gVisor. Generally the security of a piece of software isn't considered fundamentally flawed just because it has a security bug, otherwise pretty much every piece of software would be in that bucket by now. As such dismissing containers using that trope based on a bug which wasn't discovered when the trope was coined (by Dan Walsh IIRC) doesn't seem appropriate. There have been (AFAICR) three breakouts that would affect a default Docker installation in the last 3-4 years (Dirty C0w, WaitID, and the runc issue). That doesn't feel like a particularly high incidence, and gVisor has had at least one in it's shorter lifespan... If it's always trivial to breakout of docker/containerd/runc containers as (if I'm understanding you correctly) you appear to be implying and which is what appears to be implied by the trope, then I imagine people will be making good money from bug bounties for a long time as a lot of companies are creating platforms which execute semi or untrusted code in runc containers.
      - longtermsec 7y ago
        I'm not sure that it is overly simplistic, I think the statement that "containers do not contain" is an intentional oxymoron that points to some ground truths. These ground truths are that a process in a container is running in the same kernel, and although namespaces are meant to isolate some set of resources from other processes, and there are still very many shared resources that might not be isolated at all. This means a lot of attack surface, and exploiting the kernel will grant access to the other processes on the system. In terms of quantity, 4 is not an accurate picture. I haven't sat down to analyze CVEs (https://www.cvedetails.com/product/47/Linux-Linux-Kernel.html?vendor_id=33 https://www.cvedetails.com/product/47/Linux-Linux-Kernel.htm...), but say out of 50 practically exploitable kernel memory corruption bugs/year 4-5 new bugs every year are reachable from some common namespace configuration for a container. And this just marks what is publicly disclosed, which is a subset of the vulnerabilities attackers know about. Bounties arent the only outlet for these, see: VEP.
amscanne 7y ago
(I'm a co-author of the blog post) I generally agree re: trope, but it's useful because I'm not sure the core idea is widely understood outside security circles. Many people assume that containers provide a strong isolation boundary, and while a break-out is not trivial, providing more isolation in some cases is important, as you allude. While one option is certainly to provide a locked down policy, monitor the flow of kernel CVEs, and patch constantly, this may not be feasible for many organizations if a) they lack the technical expertise or b) don't know the workloads they're running a priori and can't apply a fixed policy. So different container runtimes are about providing additional tools for defense-in-depth. (VMs are fantastic tool for this, but it's also nice to have tools that play well in containerized infrastructure other than custom security policies.) None of these tools will be perfect of course, hopefully they can make it easier to improve on the status quo. Re: contained.af, this is a great example of the workloads problem. If you have a known workload where you can essentially disable all capabilities and access to system resources (e.g. no network), there are many options for securing that workload. They aren't all generalizable.
- raesene9 7y ago
  Oh I'd agree and gVisor provides (IMO) a smaller attack surface than a default runc container. With that said both options, and indeed hypervisor based isolation, are generally one security flaw away from a breakout vulnerability, so the only difference in that respect is the incidence of those flaws. My experience of people's expectations of container isolation is perhaps somewhat different to yours, which is what prompted my initial comment. It's all too common (in my experience) to see container isolation dismissed using that "containers don't contain" trope, and for me that feels frustrating as the real picture is much more nuanced than that. It's all about choosing the right isolation technology for a) a given workload and b) a given threat model/attack surface. There are tradeoffs (both in terms of performance, and in terms of flexibility) in replacing the runc layer with a different container runtime. Sometimes those will make sense, other times not so much :) All that said I'm very excited to see more options here, as it'll give everyone the choice of what mechanism works for them for specific workloads.

longtermsec 7y ago

The idea that it's trivial to break out of any Docker style container just doesn't reflect reality. -- not just being contrarian here, actually, the reality is that it might be trivial. and it was demonstratively trivial for a long time (see CVE-2019-5736) As for contained.af -- its not a good indicator, it mostly indicates that the reward doesnt meet the market price for demonstrating an escape from a set of hardened namespaces (which is going to cost more than an escape from "any docker container").

raesene9 7y ago
So the runc vuln, only applied if you were a) running as root in the container and b) hadn't enabled user namespacing. (Also for completeness, it didn't work on RHEL based distros that applied their standard SELinux policy (IIRC)) Also not specifically a Docker vulnerability, it was a runc issue which also affected other Linux containerization software (e.g. lxc) But despite all that, that's just an example of what I was talking about, all software has vulns, including runc, including gvisor. Stating that "containers don't contain" implies that it's not just a specific bug, but that architecturally the process is flawed (at least IMHO), which I would suggest is at the least an over-simplification. as to contained.af, well if it was indeed "trivial" then surely not a large reward would be required :)
- longtermsec 7y ago
  so a) and b) are common in practice. these were not obscure boundary conditions or a corner case. and it was very trivial to exploit. "all software has vulns" is a slippery slope is my overarching point. you can't use that to say that the the security risks and isolation are comparable to gvisor. gvisor does away with a very significant amount of attack surface in the linux kernel and reimplements it in golang, which eliminates many bug classes. for a realistic risk assessment you should consider the linux kernel as a bottomless barrel of memory management bugs, which are exploitable from within a container, whereas gvisor will have a much more finite set of bugs On our team we've got extensive experience in finding compromises in this area, particularly in kernels, and that is why I am adamant that one should not think what docker provides meets the bar for best practices in a security critical environment. Something like gvisor would much more fit the bill.
  - raesene9 7y ago
    The original point I was making what that dismissing container isolation with the trope "containers don't contain" is overly simplistic, not that I thought that docker/runc containers with a default profile had as small an attack surface as gVisor. Generally the security of a piece of software isn't considered fundamentally flawed just because it has a security bug, otherwise pretty much every piece of software would be in that bucket by now. As such dismissing containers using that trope based on a bug which wasn't discovered when the trope was coined (by Dan Walsh IIRC) doesn't seem appropriate. There have been (AFAICR) three breakouts that would affect a default Docker installation in the last 3-4 years (Dirty C0w, WaitID, and the runc issue). That doesn't feel like a particularly high incidence, and gVisor has had at least one in it's shorter lifespan... If it's always trivial to breakout of docker/containerd/runc containers as (if I'm understanding you correctly) you appear to be implying and which is what appears to be implied by the trope, then I imagine people will be making good money from bug bounties for a long time as a lot of companies are creating platforms which execute semi or untrusted code in runc containers.
    - longtermsec 7y ago
      I'm not sure that it is overly simplistic, I think the statement that "containers do not contain" is an intentional oxymoron that points to some ground truths. These ground truths are that a process in a container is running in the same kernel, and although namespaces are meant to isolate some set of resources from other processes, and there are still very many shared resources that might not be isolated at all. This means a lot of attack surface, and exploiting the kernel will grant access to the other processes on the system. In terms of quantity, 4 is not an accurate picture. I haven't sat down to analyze CVEs (https://www.cvedetails.com/product/47/Linux-Linux-Kernel.html?vendor_id=33 https://www.cvedetails.com/product/47/Linux-Linux-Kernel.htm...), but say out of 50 practically exploitable kernel memory corruption bugs/year 4-5 new bugs every year are reachable from some common namespace configuration for a container. And this just marks what is publicly disclosed, which is a subset of the vulnerabilities attackers know about. Bounties arent the only outlet for these, see: VEP.

amscanne 7y ago

(I'm a co-author of the blog post) I generally agree re: trope, but it's useful because I'm not sure the core idea is widely understood outside security circles. Many people assume that containers provide a strong isolation boundary, and while a break-out is not trivial, providing more isolation in some cases is important, as you allude. While one option is certainly to provide a locked down policy, monitor the flow of kernel CVEs, and patch constantly, this may not be feasible for many organizations if a) they lack the technical expertise or b) don't know the workloads they're running a priori and can't apply a fixed policy. So different container runtimes are about providing additional tools for defense-in-depth. (VMs are fantastic tool for this, but it's also nice to have tools that play well in containerized infrastructure other than custom security policies.) None of these tools will be perfect of course, hopefully they can make it easier to improve on the status quo. Re: contained.af, this is a great example of the workloads problem. If you have a known workload where you can essentially disable all capabilities and access to system resources (e.g. no network), there are many options for securing that workload. They aren't all generalizable.

raesene9 7y ago
Oh I'd agree and gVisor provides (IMO) a smaller attack surface than a default runc container. With that said both options, and indeed hypervisor based isolation, are generally one security flaw away from a breakout vulnerability, so the only difference in that respect is the incidence of those flaws. My experience of people's expectations of container isolation is perhaps somewhat different to yours, which is what prompted my initial comment. It's all too common (in my experience) to see container isolation dismissed using that "containers don't contain" trope, and for me that feels frustrating as the real picture is much more nuanced than that. It's all about choosing the right isolation technology for a) a given workload and b) a given threat model/attack surface. There are tradeoffs (both in terms of performance, and in terms of flexibility) in replacing the runc layer with a different container runtime. Sometimes those will make sense, other times not so much :) All that said I'm very excited to see more options here, as it'll give everyone the choice of what mechanism works for them for specific workloads.