4 ms·

Good to see some practices like default deny networking (ingress and egress) and very limited interactive production access being laid out here. A couple of ot

Good to see some practices like default deny networking (ingress and egress) and very limited interactive production access being laid out here. A couple of other areas that aren't mentioned, although perhaps they're still doing them are around container breakout risks. There's no mention of what (if any)hardening is being done on the container runtime, either restrictive seccomp, Apparmor/SELinux policies or using something like gVisor/Firecracker. With this year's number of container breakout CVEs, seems like an important area. A related one is whether container aware runtime security is being used to detect where an attacker might have got access to a single container and be trying to breakout to either the underlying platform or to other containers in the cluster.

TruthWillHurt 5y ago
Exactly. containers are not secure sandboxes by default and if one is breached all those K8s networking ACLs are worthless.
- csmpltn 5y ago
  > "Exactly. containers are not secure sandboxes by default and if one is breached all those K8s networking ACLs are worthless." Your suggestion being? Putting a sandbox inside a sandbox? How many layers deep should this be, before being considered "secure"?
  - chrisseaton 5y ago
    I don't think this is the bottomless pit that you think it is. A virtualised instance is a lot more secure than a container, and it's probably fine to stop at virtualised instances.
    - csmpltn 5y ago
      A lot more secure? In what ways?
      - chrisseaton 5y ago
        Containers are really a kind of process-isolation - you still share a kernel. You can find a lot of people saying that containers aren’t enough for running untrusted user code. If you run a fully virtualised instance you get your own kernel and aren’t relying on process isolation. Would you be happy if your cloud provider was running your containers on the same virtual I stance as someone else’s? Most people wouldn’t be.
        csmpltn 5y ago
        The only meaningful difference between breaking out of a process-isolated "container" and a full-blown VM is what's waiting for you outside once you've broken out. Whether it's kernel/OS or a bare metal hypervisor isn't really all that meaningful: exploits and vulnerabilities exist for either. There should be proper hardware-level isolation here, depending on the scenario. Most cloud companies can't afford that though, because they're not rolling out their own hardware.
        chrisseaton 5y ago
        Genuinely, would you be happy with just container isolation between you and other customers of your cloud provider? Most people absolutely would not.
        csmpltn 5y ago
        > "Genuinely, would you be happy with just container isolation between you and other customers of your cloud provider? Most people absolutely would not." But that's exactly how VPS hosting works today - you don't get your own private blade unless you're ready to pay premium prices and have the competence needed to run them yourself. The technicalities of how private resources in a VPS are isolated from each other will differ, but the concept remains the same nonetheless. People bite the bullet, only to be subject to things like rowhammer [1], or other container escape scenarios [2]. The top comment in this thread reflects the proper way of dealing with this: containers or sandboxes are may not be treated as a secure boundary. [1] https://www.usenix.org/conference/usenixsecurity16/technical-sessions/presentation/xiao https://www.usenix.org/conference/usenixsecurity16/technical... [2] https://www.intezer.com/blog/research/how-we-escaped-docker-in-azure-functions/ https://www.intezer.com/blog/research/how-we-escaped-docker-...
  - kasey_junk 5y ago
    Most serious security teams do not consider containers a security boundary. So it’s not a sandbox inside a sandbox, it’s just a sandbox. Gvisor and firecracker are the most popular sandboxes for containerized workloads.
    - staticassertion 5y ago
      I think this is outdated. Docker is a security boundary. There is no built-in way to get out of a Docker container just by asking by default (if you mount the socket into the container, it's trivial). How good of a boundary it is may be another story. There's some seccomp filters going on and namespacing is pretty sweet too. But an attacker can escape by exploiting the kernel, which I think most security people would consider to be not particularly high effort. So, suitable for internal services that you generally trust, not suitable for hostile code or highly exposed services. In an ideal world maybe we'd all use Firecracker but it's not nearly as easy to do that vs just putting something in a container.
      - TrueDuality 5y ago
        The reason that containers are not generally considered a security boundary is that many of the namespace primitives were _not designed_ as a security layer, they aren't designed to actively reduce the privileges from the current user's context. Since most containers are started as the root user, the namespace transition inherits root's permissions even if they're later dropped. Without SELinux or seccomp restrictions, root can still pretty much do anything to the host even inside the containers. For the most part this is troublesome when parts of the kernel or host userspace code are not fully aware of the different forms of namespacing (there are still portions that just check for an effective UID of 0, without checking whether they're in a namespace for example). These are the components where a lot of container breakouts happen and is largely mitigated by having internal processes in the container not running as root in the namespace. Dropping privileges to a different user still trace's it origin back to the root user on the host, so in some cases being partially aware of namespaces in a section of the kernel or host user code actively hurts the security by tracing the user back to root and using those privileges again. SELinux really tightens the potential to pull these shenanigans, but most production k8s clusters at least that I've seen are built on Ubuntu where those protections aren't available. In this case the security layer is once again SELinux not the namespacing. As long as the container runtime is performing the various namespace isolation primitives starting from the root user these container bypasses are going to be a risk. There are 'rootless' versions of containers which can only use the privileges available to lower (presumably heavily restricted) user but those aren't widely used. Once again this is relying on the security protections of the host user authorization, not on the namespaces. The networking analogy is NAT. People treat it like a security layer as it kind-of-sort-of looks like an ingress firewall since you can't directly address devices inside a NAT, but its not and can be pierced pretty easily. NAT is not a firewall. Namespaces are not a security layer.
  - TruthWillHurt 5y ago
    Opposite - don't mess with sandboxing. Use PaaS services like its > 2008, and let AWS / Google security teams harden their platform.
staticassertion 5y ago
> With this year's number of container breakout CVEs, seems like an important area. Worth noting that even basic hardening in docker will prevent a lot of them. I say "in docker" because K8s disables seccomp, which matters a lot since `unshare` is denied by docker's seccomp and is very useful for attackers in a container. If you use Docker the main thing to do is just not run as root. If you do that much, and it's not hard at all, you are in a much better place than a default k8s pod.

TruthWillHurt 5y ago

Exactly. containers are not secure sandboxes by default and if one is breached all those K8s networking ACLs are worthless.

csmpltn 5y ago
> "Exactly. containers are not secure sandboxes by default and if one is breached all those K8s networking ACLs are worthless." Your suggestion being? Putting a sandbox inside a sandbox? How many layers deep should this be, before being considered "secure"?
- chrisseaton 5y ago
  I don't think this is the bottomless pit that you think it is. A virtualised instance is a lot more secure than a container, and it's probably fine to stop at virtualised instances.
  - csmpltn 5y ago
    A lot more secure? In what ways?
    - chrisseaton 5y ago
      Containers are really a kind of process-isolation - you still share a kernel. You can find a lot of people saying that containers aren’t enough for running untrusted user code. If you run a fully virtualised instance you get your own kernel and aren’t relying on process isolation. Would you be happy if your cloud provider was running your containers on the same virtual I stance as someone else’s? Most people wouldn’t be.
      - csmpltn 5y ago
        The only meaningful difference between breaking out of a process-isolated "container" and a full-blown VM is what's waiting for you outside once you've broken out. Whether it's kernel/OS or a bare metal hypervisor isn't really all that meaningful: exploits and vulnerabilities exist for either. There should be proper hardware-level isolation here, depending on the scenario. Most cloud companies can't afford that though, because they're not rolling out their own hardware.
        chrisseaton 5y ago
        Genuinely, would you be happy with just container isolation between you and other customers of your cloud provider? Most people absolutely would not.
        csmpltn 5y ago
        > "Genuinely, would you be happy with just container isolation between you and other customers of your cloud provider? Most people absolutely would not." But that's exactly how VPS hosting works today - you don't get your own private blade unless you're ready to pay premium prices and have the competence needed to run them yourself. The technicalities of how private resources in a VPS are isolated from each other will differ, but the concept remains the same nonetheless. People bite the bullet, only to be subject to things like rowhammer [1], or other container escape scenarios [2]. The top comment in this thread reflects the proper way of dealing with this: containers or sandboxes are may not be treated as a secure boundary. [1] https://www.usenix.org/conference/usenixsecurity16/technical-sessions/presentation/xiao https://www.usenix.org/conference/usenixsecurity16/technical... [2] https://www.intezer.com/blog/research/how-we-escaped-docker-in-azure-functions/ https://www.intezer.com/blog/research/how-we-escaped-docker-...
- kasey_junk 5y ago
  Most serious security teams do not consider containers a security boundary. So it’s not a sandbox inside a sandbox, it’s just a sandbox. Gvisor and firecracker are the most popular sandboxes for containerized workloads.
  - staticassertion 5y ago
    I think this is outdated. Docker is a security boundary. There is no built-in way to get out of a Docker container just by asking by default (if you mount the socket into the container, it's trivial). How good of a boundary it is may be another story. There's some seccomp filters going on and namespacing is pretty sweet too. But an attacker can escape by exploiting the kernel, which I think most security people would consider to be not particularly high effort. So, suitable for internal services that you generally trust, not suitable for hostile code or highly exposed services. In an ideal world maybe we'd all use Firecracker but it's not nearly as easy to do that vs just putting something in a container.
    - TrueDuality 5y ago
      The reason that containers are not generally considered a security boundary is that many of the namespace primitives were _not designed_ as a security layer, they aren't designed to actively reduce the privileges from the current user's context. Since most containers are started as the root user, the namespace transition inherits root's permissions even if they're later dropped. Without SELinux or seccomp restrictions, root can still pretty much do anything to the host even inside the containers. For the most part this is troublesome when parts of the kernel or host userspace code are not fully aware of the different forms of namespacing (there are still portions that just check for an effective UID of 0, without checking whether they're in a namespace for example). These are the components where a lot of container breakouts happen and is largely mitigated by having internal processes in the container not running as root in the namespace. Dropping privileges to a different user still trace's it origin back to the root user on the host, so in some cases being partially aware of namespaces in a section of the kernel or host user code actively hurts the security by tracing the user back to root and using those privileges again. SELinux really tightens the potential to pull these shenanigans, but most production k8s clusters at least that I've seen are built on Ubuntu where those protections aren't available. In this case the security layer is once again SELinux not the namespacing. As long as the container runtime is performing the various namespace isolation primitives starting from the root user these container bypasses are going to be a risk. There are 'rootless' versions of containers which can only use the privileges available to lower (presumably heavily restricted) user but those aren't widely used. Once again this is relying on the security protections of the host user authorization, not on the namespaces. The networking analogy is NAT. People treat it like a security layer as it kind-of-sort-of looks like an ingress firewall since you can't directly address devices inside a NAT, but its not and can be pierced pretty easily. NAT is not a firewall. Namespaces are not a security layer.
- TruthWillHurt 5y ago
  Opposite - don't mess with sandboxing. Use PaaS services like its > 2008, and let AWS / Google security teams harden their platform.

staticassertion 5y ago

> With this year's number of container breakout CVEs, seems like an important area. Worth noting that even basic hardening in docker will prevent a lot of them. I say "in docker" because K8s disables seccomp, which matters a lot since `unshare` is denied by docker's seccomp and is very useful for attackers in a container. If you use Docker the main thing to do is just not run as root. If you do that much, and it's not hard at all, you are in a much better place than a default k8s pod.