10 ms·
How to Escape a Container
- yjftsjthsd-h 3y agoI find it relatively comforting that every single one of these requires disabling security features, though it is a good reminder of why certain things are dangerous to give containers and should be avoided.
- paxys 3y agoContainers are not a security boundary. Containers are not a security boundary. Containers are not a security boundary. Any system that treats them as such is inherently compromised.
- insanitybit 3y agoContainers are absolutely a security boundary, and an excellent one at that - at least, Docker containers are. 1. You get file, process, and network namespaces, which are a security boundary 2. You get a seccomp filter, which is a security boundary The "containers are not a security boundary" meme needs to die. Elsewhere you mention that "containers are not sufficient for untrusted code" but that's a very specific and very niche threat model. Most people don't say "send me a binary and I'll execute it", or have arbitrary RCE + multitenancy concerns. Containers aren't sufficient for multi-tenant RCE because the RCE is by design so 100% of your security pressure is on the container at that point. In the vast majority of cases you're dealing with servers that don't intend to allow arbitrary code execution, and containers are an extremely easy way to drive up the cost of an attack given that the attacker has already spent a lot of time and money on the RCE. SELinux is also not sufficient for the "RCE by design" threat model - is SELinux not a security boundary? Further, containers can limit the impact of remote vulnerabilities like path traversal attacks, since they have file isolation by default. edit: I see elsewhere that there's a real lack of clarity here. First off, a security boundary can be meaningfully defined as a limitation on an attacker that does not have a way around it without additional exploitation. So the main reason why people have said "containers are not a security boundary" is because: a) Very, very early on, escaping a container was trivial - like you could just ask to leave and you'd be out. b) There were some blog posts basically saying "containers aren't sufficient for multi-tenancy" where arbitrary users can run arbitrary code on the same host. This is still the case today - but it's also an extremely rare threat model. Why would containers not be sufficient for (b) ? Because the majority of the Linux kernel is still exposed to the attacker within a container - the vast majority of system call interfaces are exposed (but seccomp removes a number of these, which is nice). The Linux kernel is not at all sufficiently hardened against attackers who can make arbitrary system calls, therefor containers are not sufficient against those attackers. If you give the attacker RCE by default (ie: your service is "Send me a binary and i'll run it") then an attacker can spend all of their time and money just on a local privesc, which isn't crazy difficult. Since the cost of RCE in an RCE-aaS is 0 the consensus is that containers aren't strong enough for RCE-aaS threat models. In that case use Firecracker or gVisor or a dedicated host. Otherwise, RCE costs tend to be pretty high and having to develop an additional LPE on top of one is, at minimum, quite a pain for many attackers. Containers are extremely easy to deploy software into, something like a Firecracker VM is not. Containers are basically just processes, so you can monitor them and manage them trivially. Monitoring and managing VMs with processes inside of them is obviously harder. So I think the 'bang for your buck' with containers is extremely solid.
- rjzzleep 3y agoThey were on Solaris no?
- carbotaniuman 3y agoWhat is the security boundary then? Everywhere I read treats them as a security boundary for say, untrusted code.
- openasocket 3y agoVirtual machines can be considered a security boundary. Containers are absolutely not.
- dvt 3y agoThe distinction is getting more and more fuzzy, so this is almost a meaningless point (as is GGP). It's very vendor-specific, let alone that I'm pretty sure Spectre attacks can work across VMs.
- openasocket 3y agoI wouldn’t consider the distinction “fuzzy”. Assuming we’re talking Linux (I don’t know about the Mac and Windows world) containers are implemented using namespaces and cgroups, and always have been. Whether you are talking docker, containerd, some more minimalistic thing built on runc, it’s all Linux namespaces and cgroups. And those things were explicitly not designed to act as security boundaries when running untrusted code.
- deleted 3y ago[deleted]
- zokier 3y agoKata containers would like a word
- yjftsjthsd-h 3y agoNow in fairness, that is very specifically using a VM to add an even stronger boundry; it's not really the same thing.
- 2OEH8eoCRo0 3y agoOf course they are. There is debate over how good of a boundary they are but even caution tape is a security boundary.
- johncolanduoni 3y agoThis is true, but the exploits listed in this article are poor evidence of this. If it was as simple as not giving Linux containers any capabilities or host sockets they would be a decent security boundary. I find it super frustrating that we're stuck with kernels with inherent weaknesses to their security approach that we have to re-implement them in userspace in one way or another (gVisor, Firecracker, etc.) just to get the hardware-provided userspace/kernel boundary to work properly.
- 10000truths 3y agoThis kind of reductive dogma is meaningless FUD. A container is as secure as its underlying implementation.
- imetatroll 3y agoIs gvisor considered sufficient when running 3rd-party code? Are there other measures that should be taken in addition to using gvisor?
- brutal_chaos_ 3y agoA container is a security boundry. However, no security boundry is perfect and more boundries must be put in place as well, depending on your threat model. edit: typo
- harporoeder 3y agoAll of these escapes rely on some obvious explicit reduction of the isolation guarantees. If you know how to escape a simple docker container invoked with default parameters such as `docker run --rm -it ubuntu /bin/bash` I'm sure many people would be interested.
- paxys 3y ago"Escapes" is a vaguely defined term, but considering basic escalation of privilege: - Networking is the obvious one in this scenario. By default (on Docker/LXC/others) all containers are on the same virtual bridge and can communicate with each other. Even with some additional configuration and isolation it is possible to MITM attack other containers on the same host. - It is very easy to DDoS adjacent containers e.g. by spamming signals, forking new processes, creating files. There is again no default safeguard against this.
- charcircuit 3y agoIf you give a container access to the network, then yes it will have access to the network. That isn't an escape. cgroups can prevent containers from using too much memory or cpu
- im3w1l 3y agoI think we must be careful not to use no-true-scotsman here. Imo, an attack should be considered valid if it works against a typical config - one you would arrive at following a popular tutorial, rather than requiring that it must work against an ideal config.
- charcircuit 3y agoThe definition of escaping a container (namespace) is being able to interact with resources not in the namespaces of the process. If a process's network namespace contains a network device then it is not an escape to use it by definition. If the network namespace for a process contains no devices then being able to use a network device would be an escape.
- ajross 3y agoIs putting processes in separate memory spaces not a security boundary to you? Isolating separate app responsibilities in separate UIDs? Filesystem permission bits? All those are "weaker" boundaries than containers. Do you really claim that these have ZERO security value? That's silly. Of course containers are a security boundary. They have advantages and disadvantages. Treat them as tools and not slogans.
- SoftTalker 3y ago> Containers are not a security boundary. The name is at least misleading if not wrong, then? What do they "contain"?
- Kluggy 3y agoThey contain the transitive closure of software and libraries to run a binary. Under the hood, they're a fancy wrapper around a pile of tar files. Tar files certainly contain other files and are also not a security boundary.
- arkadiyt 3y agoNot mentioned: use a kernel N-day, of which there are many. Patch your hosts folks
- insanitybit 3y agoFor many, many services, requiring an entire separate exploit against the kernel is a huge win. So many services can just be dropped into a container with roughly 0 effort. If you want a higher level of security it's going to take significantly more effort.
- deleted 3y ago[deleted]
- Mtinie 3y agoI found the content to be of interest and gave me new things to think about. But honestly, I was hoping before I clicked it that this was going to be about how to escape from the inside of a shipping container.
- phero_cnstrcts 3y agoMe too, that skill could come in handy one day.
- lurquer 3y agoLikewise. Damn. I guess I’ll stay stuck in here for a while longer.
- itishappy 3y agoApparently the answer to escaping from a shipping container is... You don't, they cannot be opened from inside once locked. Also they're airtight, so bang on the walls and hope help arrives before you suffocate. That's more nightmare inducing than I was hoping.
- knodi123 3y agocarry a pocket angle grinder.
- pmontra 3y agoAnd hope your container is not buried deep between hundreds of other containers. You'll need a lot of battery packs.
- Proven 3y ago[dead]
- 4ggr0 3y ago> Also they're airtight Really?! I know that these things are built to be very stable, but they never gave me the impression to be air-tight. Not bad. I'm sure the world of shipping containers is a huge rabbit-hole to read into.
- deleted 3y ago[deleted]
- AndyMcConachie 3y agoI clicked on this thinking I would learn how to escape a shipping container if I ever got trapped in one ;)
- shepherdjerred 3y agoI had the exact same thought! That would make a very interesting article. Related video :) https://www.youtube.com/watch?v=-trd_f6j3eI https://www.youtube.com/watch?v=-trd_f6j3eI
- deleted 3y ago[deleted]
- adriangrigore 3y agoWhy escape?
- heyoni 3y agoAll those processes living in our containers are going to want the red pill eventually. Haven’t you seen the matrix? But seriously though, it’s so you can write exploits or satisfy that curious itch when working with a cloud service.
- c0pium 3y agoYou are being downvoted right now, but this is a great point. If you have execution control in someone’s container, use the containers existing secrets to achieve your goals. I don’t need to escape your web apps container to steal all of the contents of the backend database.
- rompledorph 3y agoWhats the state of lightweight VMs? Can they replace containers yet? Not sharing the kernel with the host os (or other containers) is a huge security boundary.
- munawwar 3y agoAWS lambda and fly.io uses firecracker VMs internally. So I think it can replace containers to some extent.
- b112 3y agoTo me, a guy that's been doing this for decades, this is a weird thing to say. A bit of debootstrap, a few apt-get commands, and copying in config files, and you have a lightweight VM, minimal image. Something people have been doing for 20 years. There are also sorts of tricks, such as having two images, one for the app layer, one for the OS, which makes the deploy for app updates faster. I'm not even sure why people care about image size all that much. You copy it to your local cluster, then deploy from there.
- nicoburns 3y agoDo those lightweight VMs startup (cold start) in milliseconds? Because I think that's a key thing that people are looking for from containers and VMs that might be used in their stead.
- nijave 3y agoFirecracker, yes
- lifty 3y agoIt's managing them at a bigger scale that is the challenge. Besides that, the container ecosystem gives you APIs to do all this management with established tools, in an automatic way.
- ptx 3y agoThey would still be vulnerable to the sort of attack described in the article, though: If the host deliberately hands the guest a socket it can use to execute commands as root on the host, there's nothing that can be done to make it secure.
- tpetry 3y agoAny of the capabilities needed by the exploits is in linux only available to root users. And are now granted to a container. So these ”escapes“ are basically giving a container full access to the host and using that. None of them are enabled by default.
- t8sr 3y agoIn practice, though, containers are given these capabilities all the time, and in infosec people do refer to these as escapes. Generally, it’s not the stuff the container is intended to run that’s doing the escape, though. This is usually the second step, after getting local execution through a vulnerability in the application running in the container.
- ptx 3y agoWhy are these capabilities usually granted, in your experience?
- achierius 3y agoIn my experience profilers often need SYS_ADMIN or other capabilities. This is true even outside of a container. If I'm forced to work inside a container without the ability to add that cap, performance optimization work becomes very difficult.
- blackmesaind 3y agoUser error is the source of many vulnerabilities that shouldn't exist.
- t8sr 3y agoI posted another comment with some examples, but I think the general answer, IME, is that it's usually easier to do the wrong thing than to do the right thing. Security boundaries in Cloud-style Linux are typically hard to configure and not well documented. Engineers have deadlines and they're interested in building things, not debugging permissions. They'll do whatever is the easiest to get the system working. This, by the way, is why I think Google had it right by splitting most products into SWE and SRE groups, so you had a group of people who could focus full-time on the production environment. SWEs are rewarded for building and deploying stuff, so they're going to build and deploy stuff.
- fulafel 3y agoThis post seems confused and misleading. It only lists ways to escape containers with various non-default capabilities added, but doesn't address the other more realistic and seen in the wild ways (eg from uid=0 confusion due to disuse of user namespaces, kernel privilege escalation bugs, etc).
- t8sr 3y agoI don’t think it’s misleading - working in security, I can tell you that containers are misconfigured with too many privileges everywhere you look. SWEs like focusing on cool stuff, like kernel exploits, to the detriment of basic production security. From the point of view, this is an article I think many SWEs could benefit from reading, especially on teams that do their own deployments.
- mmis1000 3y agoFor example, it isn't that weird to give container access to ptrace cap. Sys_ptrace is a important capacity if you need to profile the process for whatever reason. Sys_admin is required to mount a tmpfs file system even it don't actually modify any file. There are cases that these are actually required. And you don't know what may also cause security issue and why it isn't given by default if you did not read about that.
- ptx 3y agoThe "escape" in the article also requires "--pid=host", i.e. permission to debug processes on the host. That seems like a very obviously bad idea.
- mmis1000 3y agoFor interactive debug, maybe? Container can be used a a dev environment to dev/test things that would otherwise affect the host irreversibly. For example: something like installer.
- fulafel 3y ago
- remram 3y ago1 and 7 need SYS_ADMIN, not available by default in any container runtime. 2 needs Docker socket, it's explicitly meant for running other workloads 3 needs shared PID namespace in addition to SYS_PTRACE, neither is granted by default by any container runtime 4 needs SYS_MODULE, again no one has that 5 and 6 need DAC_READ_SEARCH, no one grants that, no one uses that None of those seem like vulnerabilities or things that would be available without the admin taking explicit steps to specifically want to allow escaping. Being root in the container would not be enough to get any of those capabilities.
- cyrnel 3y ago> to specifically want to allow escaping Not really unfortunately. A novice sysadmin granting someone's request to add some permission called SYS_MODULE may not know that it could be equivalent to full root access. That's why posts like this are important for education.
- Karellen 3y ago> A novice sysadmin granting someone's request to [give them root in an obscure, sneaky manner] ...is a social engineering exploit, not a technical one.
- achierius 3y agoTo pose an alternative scenario: > A novice sysadmin granting someone's request to [be able to run some tool/software that says it needs this, but is in fact secretly malicious] or perhaps more common > A novice sysadmin granting someone's request to [be able to run some legitimate tool or software, then later a virus in the container taking advantage and breaking out] for a real world example, the Nvidia Nsight docs specifically direct you to add SYS_ADMIN; do you think your everyday ML engineer would know that doing so poses a security risk if they're doing this on Docker?
- clvx 3y agoI've seen people who use testcontainers and run their CI workloads in containers abusing the docker.sock mounting so they can spin up the tests. The anti-pattern of using docker.sock has been always a threat because when docker got popularity in CI/CD systems it was the easiest way to have a platform independent way to spin up isolated environments. In my perception, this was a very common pattern in Jenkins a few years ago.
- timost 3y agoUsing rootless podman limits the blast radius of a container escape. Also many of the cappabilities described in this article aren't compatible with a rootless user deployment scénario.
- _8j50 3y agoUseful: https://github.com/stealthcopter/deepce https://github.com/stealthcopter/deepce
- denton-scratch 3y agoAll these escapes seem to involve Docker. It looks to me as if at least some of them are strictly Docker-dependent, but I've never used Docker, so I'm no expert.
- danrl 3y agoI genuinely thought this link was about escaping a standardized steel shipping container, something I recently had to seriously consider. In that regard a disappointing click. I also wrote my own docker-like containerization code for educational purposes a while ago, so container has both these meanings for me. Yet, me brain was expecting a physical escape story. Brains are funny!
- jhiggins777 3y agoThought the same thing. I, however, have not written any containerization code.
- Sheeny96 3y agoI think there's some version of the dunning krueger effect going on in these comments - assuming that no one would include this number of security flaws unless intentionally. Perhaps it's that this forum tends to attract people more engaged in the CS space that wouldn't do this - but I've seen enough brute forcing in the wild to know that this ABSOLUTELY exists where a "just make it work" mentality is present.
- ipdashc 3y agoYou're totally right, but the part that annoys me is that articles like this one (and this sounds overly hostile, I don't intend that, but I'm not sure how else to phrase it) kind of pollute the topic of container security. I described it above, but I have this huge pet peeve where I hear "containers are insecure and trivial to break out of" and then when I go to look up examples of container breakouts, all I find is stuff like this; how to break through a wall that had a gaping, intentional hole left in it. It feels like "breaking out of vanilla containers" and "breaking out of misconfigured containers" are two different topics, two different threat models. And while the second absolutely matters in the real world, the really scary stuff is obviously the first (and usually involves 0-days, kernel exploits, etc?). But people seem to talk less about the first.