6 ms·

Escaping privileged containers for fun

112 points by jordyzomer 5y ago

smbv 5y ago
I use AppArmour and use Podman rootless for everything that needs to be containerized so it's all good for me anyway :) Fun read! Maybe put an RSS button somewhere so it's easier to subscribe to?
- jordyzomer 5y ago
  Thanks very good suggestion! I'll take a look if Hugo has a feature for that otherwise I 'll edit the templates :D
- WhyNotHugo 5y ago
  > Podman rootless This is the way. Docker rootless and podman rootless. A lot less attack surface than running containers as root. I haven't tried a dedicated user for this though. I'm sure that mounting volumes would get messed up.
  - 2OEH8eoCRo0 5y ago
    I run services like Plex or DNS adblocker in this manner. Each service gets its own user to run rootless under and inside the container the app doesn't run as root. Security is layers.
paxys 5y ago
Containers are not a security mechanism. Containers are not a security mechanism. Containers are not a security mechanism. ... Every engineer making a foray into Docker or any similar tech for the first time should be made to write this line x1000 before proceeding.
- 2OEH8eoCRo0 5y ago
  https://www.redhat.com/sysadmin/basic-security-principles-containers https://www.redhat.com/sysadmin/basic-security-principles-co...
- hutrdvnj 5y ago
  How about virtual machines?
- zibzab 5y ago
  There are _some_ containers that are security mechanism. Docker is definitely not one of them. But LXD for example uses unprivileged containers by default and can be hardened with additional software for syscall filtering
  - iancarroll 5y ago
    If LXD also uses cgroups and Linux namespaces it seems like the differences would be pretty small? Docker also allows custom syscall filtering, unprivileged containers, etc. I think the reality is just that containers will always be imperfect, and you should understand that risk when building systems.
    - EE84M3i 5y ago
      I was under the understanding that docker didn't use user namespaces by default -- so "root" in the container is the same user as "root" outside the container -- but that LXD does. Has that changed?
      - raesene9 5y ago
        It doesn't but TBH user namespaces have a mixed record when it comes to security. For example CVE-2022-0185 wasn't exploitable from a standard container unless user namespaces were available allowing additional capabilities to be gained, inside that user namespace (https://blog.aquasec.com/cve-2022-0185-linux-kernel-container-escape-in-kubernetes https://blog.aquasec.com/cve-2022-0185-linux-kernel-containe... has more details.
    - MereInterest 5y ago
      My biggest concern is how docker's default behavior is to run roughshod over traditional unix security. Usually, adding a user to a group grants them some additional files or devices. The expectation is that adding a user to the "docker" group would allow that user to interact with the docker daemon, and that the docker daemon would check user-level privileges internally. Instead, adding a user to the "docker" group allows a user to have passwordless root-level privileges to access or modify any file on the system, regardless of file ownership (e.g. `docker run --volume /:/mnt ubuntu cat /mnt/etc/shadow`). There's some documentation on how only trusted users should be given access to the docker daemon, but I do not consider it sufficient. At no point in the installation or getting started documents [0,1] is it mentioned that it drives a gaping hole through existing security measures. Instead, the mention was on the security page, three sections down in a discussion about the attack surface [2]. This is the sort of issue that should be in big bold blinking letters at the top of every tutorial, that access to docker is I know that rootless docker exists and improves this situation, but it is neither the default behavior, nor the introductory example in official documentation. I know that docker's primary role is dependency management, and that it only considers escalation coming from within the container as security issues. But there's a world of difference between "I'm not a locksmith, so I don't sell locks." and "I'm not a locksmith, so I break in your windows." One is ambivalent to existing security measures, while the other, like Docker, actively subverts them. [0] https://docs.docker.com/get-started/ https://docs.docker.com/get-started/ [1] https://docs.docker.com/engine/install/ https://docs.docker.com/engine/install/ [2] https://docs.docker.com/engine/security/#docker-daemon-attack-surface https://docs.docker.com/engine/security/#docker-daemon-attac... Edit: When I was initially researching this, because I was absolutely floored that this security flaw was even a possibility in something as widely used as docker, I came across this reddit post [3], which explains the issue quite well. It is 4 years old and predates rootless docker, but is still accurate to the default and most widely used behavior. [3] https://www.reddit.com/r/docker/comments/7y2yp2/why_is_singularity_used_as_opposed_to_docker_in/duqhel4/ https://www.reddit.com/r/docker/comments/7y2yp2/why_is_singu...
- staticassertion 5y ago
  Containers are a security mechanism. We can say that now. For a long time they weren't, and then it wasn't really clear, but at this point they are. How good are they? Questionable - there are footguns (running privileged) and they rely on the Linux kernel (garbage), but they are a barrier. If you put a process into a container, even a fairly default config'd container, it will require an additional vulnerability for an attacker to escape that container. That makes it a security boundary, period. There is no universal "ask to leave the container" by default. This post is attacking a non-default configuration ie: you have to explicitly pass in "--privileged". Don't do that (the post says this repeatedly). The fact that people are using docker and we're basically getting a half decent sandbox for free, everywhere, is an incredible security win. I wouldn't rely on them for multi-tenant workloads, but as a nice way to restrict attackers given RCE, yeah, sure.
  - atoav 5y ago
    Containers increase the effort for the attacker to put into escaping that container — provided the developers actually run code that doesn't make that easy right of the bat. I have seen to much code where you can really feel the thought of "let's just use docker, then we don't have to think about complicated linux things like priviledges and permissions". Using containers as an extra layer of security is perfectly fine. Using it as an excuse to not do any other security measures is not.
    - staticassertion 5y ago
      > "let's just use docker, then we don't have to think about complicated linux things like priviledges and permissions" Honestly, that's fine. What more do you want? For developers to roll their own SELinux and Apparmor profiles? For them to manually implement seccomp filtering? No one does that. The reality is that no one is choosing between "containers or some other approach", they're choosing between "containers or nothing".
      - danuker 5y ago
        Hopefully they aren't choosing containers between "containers + insecure app" or "no containers + secure app", thinking containers will secure the app.
- maple3142 5y ago
  I wonder how hard is it to escape a unprivileged container as a non-root user in a container. It seems that many CTF organizers already host their RCE challenge this way.
- raesene9 5y ago
  Containers provide a level of isolation relative to standard Linux processes. Docker style containers were not designed specifically as a security sandbox (IMO) but instead re-use a set of Linux primitives to provide a level of isolation. One of the good parts about this is that as they use Linux primitives, the level of isolation can be increased relatively easily. Things like --cap-drop=ALL and --no-new-privileges, combined with ensuring the contained process(es) don't run as root can make things better. It's also important to ensure that the isolation put in place isn't removed. The major case where this happens is that Kubernetes disables Docker's seccomp filter which has left containers running in k8s clusters vulnerable to issues that stock Docker containers were not (CVE-2022-0185 for a recent example). If I was running truly untrusted processes from unknown people, I probably would not use Docker containers, instead I'd use gVisor or Firecracker.
- godwhoa 5y ago
  FWIW, most of the untrusted code execution "platforms" these days use Docker with some hardening (dropped capabilities, syscall filtering, readonly fs etc). Eg. Repl.it and Rust Playground.
- cle 5y ago
  These days, Docker doesn't care about the underlying container runtime. You can use Docker to run containers in VMs with Firecracker, gVisor, Kata, etc. Security and multitenancy are their raison d'être. You might have been right in 2014. In 2022, containers are most definitely a security mechanism.
ackbar03 5y ago
Traditional advice was you analyze potentially malicious binaries in a VM. I guess its more prudent now to do it on air-gapped systems?
- contravariant 5y ago
  Well, at the very least don't do it in a privileged docker container.
jwilk 5y ago
One-liner to trigger core dumping: $ sh -c 'kill -ABRT $$' Aborted (core dumped)
- cuteboy19 5y ago
  $$ would give the pid of the inner sh command right? Because the single quote would disable interpolation in the outer shell. So why would this work?
  - jwilk 5y ago
    It works by killing the inner shell. The outer shell remains alive, so that it can print the "core dumped" message. If you don't mind losing your current shell and don't care about the message, you can use just: $ kill -ABRT $$
    - deleted 5y ago
      [deleted]

112 points by jordyzomer 5y ago

smbv 5y ago

I use AppArmour and use Podman rootless for everything that needs to be containerized so it's all good for me anyway :) Fun read! Maybe put an RSS button somewhere so it's easier to subscribe to?

jordyzomer 5y ago
Thanks very good suggestion! I'll take a look if Hugo has a feature for that otherwise I 'll edit the templates :D
WhyNotHugo 5y ago
> Podman rootless This is the way. Docker rootless and podman rootless. A lot less attack surface than running containers as root. I haven't tried a dedicated user for this though. I'm sure that mounting volumes would get messed up.
- 2OEH8eoCRo0 5y ago
  I run services like Plex or DNS adblocker in this manner. Each service gets its own user to run rootless under and inside the container the app doesn't run as root. Security is layers.

paxys 5y ago

Containers are not a security mechanism. Containers are not a security mechanism. Containers are not a security mechanism. ... Every engineer making a foray into Docker or any similar tech for the first time should be made to write this line x1000 before proceeding.

2OEH8eoCRo0 5y ago
https://www.redhat.com/sysadmin/basic-security-principles-containers https://www.redhat.com/sysadmin/basic-security-principles-co...
hutrdvnj 5y ago
How about virtual machines?
zibzab 5y ago
There are _some_ containers that are security mechanism. Docker is definitely not one of them. But LXD for example uses unprivileged containers by default and can be hardened with additional software for syscall filtering
- iancarroll 5y ago
  If LXD also uses cgroups and Linux namespaces it seems like the differences would be pretty small? Docker also allows custom syscall filtering, unprivileged containers, etc. I think the reality is just that containers will always be imperfect, and you should understand that risk when building systems.
  - EE84M3i 5y ago
    I was under the understanding that docker didn't use user namespaces by default -- so "root" in the container is the same user as "root" outside the container -- but that LXD does. Has that changed?
    - raesene9 5y ago
      It doesn't but TBH user namespaces have a mixed record when it comes to security. For example CVE-2022-0185 wasn't exploitable from a standard container unless user namespaces were available allowing additional capabilities to be gained, inside that user namespace (https://blog.aquasec.com/cve-2022-0185-linux-kernel-container-escape-in-kubernetes https://blog.aquasec.com/cve-2022-0185-linux-kernel-containe... has more details.
  - MereInterest 5y ago
    My biggest concern is how docker's default behavior is to run roughshod over traditional unix security. Usually, adding a user to a group grants them some additional files or devices. The expectation is that adding a user to the "docker" group would allow that user to interact with the docker daemon, and that the docker daemon would check user-level privileges internally. Instead, adding a user to the "docker" group allows a user to have passwordless root-level privileges to access or modify any file on the system, regardless of file ownership (e.g. `docker run --volume /:/mnt ubuntu cat /mnt/etc/shadow`). There's some documentation on how only trusted users should be given access to the docker daemon, but I do not consider it sufficient. At no point in the installation or getting started documents [0,1] is it mentioned that it drives a gaping hole through existing security measures. Instead, the mention was on the security page, three sections down in a discussion about the attack surface [2]. This is the sort of issue that should be in big bold blinking letters at the top of every tutorial, that access to docker is I know that rootless docker exists and improves this situation, but it is neither the default behavior, nor the introductory example in official documentation. I know that docker's primary role is dependency management, and that it only considers escalation coming from within the container as security issues. But there's a world of difference between "I'm not a locksmith, so I don't sell locks." and "I'm not a locksmith, so I break in your windows." One is ambivalent to existing security measures, while the other, like Docker, actively subverts them. [0] https://docs.docker.com/get-started/ https://docs.docker.com/get-started/ [1] https://docs.docker.com/engine/install/ https://docs.docker.com/engine/install/ [2] https://docs.docker.com/engine/security/#docker-daemon-attack-surface https://docs.docker.com/engine/security/#docker-daemon-attac... Edit: When I was initially researching this, because I was absolutely floored that this security flaw was even a possibility in something as widely used as docker, I came across this reddit post [3], which explains the issue quite well. It is 4 years old and predates rootless docker, but is still accurate to the default and most widely used behavior. [3] https://www.reddit.com/r/docker/comments/7y2yp2/why_is_singularity_used_as_opposed_to_docker_in/duqhel4/ https://www.reddit.com/r/docker/comments/7y2yp2/why_is_singu...
staticassertion 5y ago
Containers are a security mechanism. We can say that now. For a long time they weren't, and then it wasn't really clear, but at this point they are. How good are they? Questionable - there are footguns (running privileged) and they rely on the Linux kernel (garbage), but they are a barrier. If you put a process into a container, even a fairly default config'd container, it will require an additional vulnerability for an attacker to escape that container. That makes it a security boundary, period. There is no universal "ask to leave the container" by default. This post is attacking a non-default configuration ie: you have to explicitly pass in "--privileged". Don't do that (the post says this repeatedly). The fact that people are using docker and we're basically getting a half decent sandbox for free, everywhere, is an incredible security win. I wouldn't rely on them for multi-tenant workloads, but as a nice way to restrict attackers given RCE, yeah, sure.
- atoav 5y ago
  Containers increase the effort for the attacker to put into escaping that container — provided the developers actually run code that doesn't make that easy right of the bat. I have seen to much code where you can really feel the thought of "let's just use docker, then we don't have to think about complicated linux things like priviledges and permissions". Using containers as an extra layer of security is perfectly fine. Using it as an excuse to not do any other security measures is not.
  - staticassertion 5y ago
    > "let's just use docker, then we don't have to think about complicated linux things like priviledges and permissions" Honestly, that's fine. What more do you want? For developers to roll their own SELinux and Apparmor profiles? For them to manually implement seccomp filtering? No one does that. The reality is that no one is choosing between "containers or some other approach", they're choosing between "containers or nothing".
    - danuker 5y ago
      Hopefully they aren't choosing containers between "containers + insecure app" or "no containers + secure app", thinking containers will secure the app.
maple3142 5y ago
I wonder how hard is it to escape a unprivileged container as a non-root user in a container. It seems that many CTF organizers already host their RCE challenge this way.
raesene9 5y ago
Containers provide a level of isolation relative to standard Linux processes. Docker style containers were not designed specifically as a security sandbox (IMO) but instead re-use a set of Linux primitives to provide a level of isolation. One of the good parts about this is that as they use Linux primitives, the level of isolation can be increased relatively easily. Things like --cap-drop=ALL and --no-new-privileges, combined with ensuring the contained process(es) don't run as root can make things better. It's also important to ensure that the isolation put in place isn't removed. The major case where this happens is that Kubernetes disables Docker's seccomp filter which has left containers running in k8s clusters vulnerable to issues that stock Docker containers were not (CVE-2022-0185 for a recent example). If I was running truly untrusted processes from unknown people, I probably would not use Docker containers, instead I'd use gVisor or Firecracker.
godwhoa 5y ago
FWIW, most of the untrusted code execution "platforms" these days use Docker with some hardening (dropped capabilities, syscall filtering, readonly fs etc). Eg. Repl.it and Rust Playground.
cle 5y ago
These days, Docker doesn't care about the underlying container runtime. You can use Docker to run containers in VMs with Firecracker, gVisor, Kata, etc. Security and multitenancy are their raison d'être. You might have been right in 2014. In 2022, containers are most definitely a security mechanism.

ackbar03 5y ago

Traditional advice was you analyze potentially malicious binaries in a VM. I guess its more prudent now to do it on air-gapped systems?

contravariant 5y ago
Well, at the very least don't do it in a privileged docker container.

jwilk 5y ago

One-liner to trigger core dumping: $ sh -c 'kill -ABRT $$' Aborted (core dumped)

cuteboy19 5y ago
$$ would give the pid of the inner sh command right? Because the single quote would disable interpolation in the outer shell. So why would this work?
- jwilk 5y ago
  It works by killing the inner shell. The outer shell remains alive, so that it can print the "core dumped" message. If you don't mind losing your current shell and don't care about the message, you can use just: $ kill -ABRT $$
  - deleted 5y ago
    [deleted]