3 ms·

I think it's subtler than that. It is mostly safe to run a contained process as "root" because in theory the ways that root access can be exercised is highly sa

by geofft 8y ago

I think it's subtler than that. It is mostly safe to run a contained process as "root" because in theory the ways that root access can be exercised is highly sandbox by the use of various namespaces, as well as things like capability restrictions (you generally don't have CAP_SYS_ADMIN or a few others), limited syscall attack surface (you generally have a syscall allowlist via seccomp-bpf), etc. Yes, it's wrong to not understand that the runc process runs as root. But I think it's only very slightly less wrong to claim that a process inside the container has root access in the way that, say, ssh root@host-system has root access. It mostly does not, and this vulnerability is notable precisely because it's one of the rare ways to exercise that root privilege outside the container. We looked at this at $work and got into a serious rabbit hole about how exactly Linux capabilities work. I think if I started asking interviewees to explain permitted vs. effective capability sets and how file and process capabilities differ, I'd never hire anyone. (And I think to figure out yourself how to "correctly configure" a container, you need to have at least some understanding of that.)

wicket 8y ago
> I think it's subtler than that. It is mostly safe to run a process as "root" because in theory the ways that root access can be exercised is highly sandbox by the use of various namespaces, as well as things like capability restrictions (you generally don't have CAP_SYS_ADMIN or a few others), limited syscall attack surface (you generally have a syscall allowlist via seccomp-bpf), etc. Yes, it's wrong to not understand that the runc process runs as root. But I think it's only very slightly less wrong to claim that runc has root access in the way that, say, ssh root@host-system has root access. It mostly does not, and this vulnerability is notable in that it's one of the few ways to exercise that root privilege. > We looked at this at $work and got into a serious rabbit hole about how exactly Linux capabilities work. I think if I started asking interviewees to explain permitted vs. effective capability sets and how file and process capabilities differ, I'd never hire anyone. (And I think to figure out yourself how to "correctly configure" a container, you need to have at least some understanding of that.) I think you've hit the nail on the head. I only ask those interview questions because I believe it's important to find out just how much a candidate understands. I have to admit I've let some of these things go, otherwise I'd never hire anyone either. I think in the end what it comes down to is that Docker is an ambitious project that is somewhat flawed from a security perspective. There have been numerous namespace vulnerabilities to date and I expect there will be plenty more found in the future.
- nicoburns 8y ago
  I believe there was work on non-root container host processes going on at some point? Did that ever get to a usable state?
  - geofft 8y ago
    Rootless containers using user namespaces was merged into runc in March 2017: https://github.com/opencontainers/runc/pull/774 https://github.com/opencontainers/runc/pull/774 A week ago Docker gained support for running dockerd as non-root: https://github.com/moby/moby/pull/38050 https://github.com/moby/moby/pull/38050 And there is this project for running Kubernetes as non-root: https://github.com/rootless-containers/usernetes https://github.com/rootless-containers/usernetes
  - cyphar 8y ago
    Yes, it's in a usable state. Docker just merged a PR that allows you to run it rootless[1]. Props to Akihiro for bringing this one over the line, I didn't think it'd be possible three years ago. [1]: https://github.com/moby/moby/commit/ec87479b7e2bf6f1b5bcc657a377c6e6a847574f https://github.com/moby/moby/commit/ec87479b7e2bf6f1b5bcc657...
deleted 8y ago
[deleted]
cyphar 8y ago
> I think it's subtler than that. It is mostly safe to run a contained process as "root" because in theory the ways that root access can be exercised is highly sandbox by the use of various namespaces, as well as things like capability restrictions (you generally don't have CAP_SYS_ADMIN or a few others), limited syscall attack surface (you generally have a syscall allowlist via seccomp-bpf), etc. I disagree. It is definitely "safer" than running as root on the host -- but that shouldn't be the baseline. Honestly, I would argue that not using user namespaces (or non-root inside the container) is basically negligence at this point. Yes, it's annoying to do it with Docker, but other runtimes have solved this problem. LXC actually explicitly states that privileged (non-userns) containers are fundamentally unsafe and I agree. When you look at the wide array of ns_capable and other userns checks that protect against all sorts of attacks, you really start to not trust anything that doesn't use user namespaces. Kernel developers assume that container runtimes are using user namespaces if they are trying to secure something. Additionally, yes capabilities help. But you still have traditional Unix DAC issues (which is what is leveraged here). > and this vulnerability is notable precisely because it's one of the rare ways to exercise that root privilege outside the container. I would argue there's several very foundational security problems (which I'm trying to fix) that are made significantly worse by running a container process as root. Please don't do it.
- geofft 8y ago
  Can you use user namespaces with a released version of Docker or Kubernetes? I think the answer is no? (Or can you do it with Kubernetes + some other container runtime?) We run Kubernetes + Docker with a policy of no root inside containers (we map you to your normal UID inside the container), but most of the rabbit hole we got into was trying to figure out the implications of setuid binaries inside the container. It seems like on a normal system, setuid binaries inside a container do in fact get host root unless you tell Docker to drop those capabilites, and also fscaps aren't really usable in a container image, I think.
  - cyphar 8y ago
    User namespaces have been supported in Docker since 1.10. I don't think that it's necessarily "supported" in Kubernetes -- there was a KEP to add support last year but it's still a while away. As for setuid and fscaps, they both work in containers (container images can contain them but some filesystems don't support xattrs such as AUFS). And yes, without user namespaces, they escalate to host root. You can use no_new_privs which blocks things like setuid or fscap but it also can cause problems (though I think you can enable no_new_privs in Kubernetes).

by geofft 8y ago

wicket 8y ago

> I think it's subtler than that. It is mostly safe to run a process as "root" because in theory the ways that root access can be exercised is highly sandbox by the use of various namespaces, as well as things like capability restrictions (you generally don't have CAP_SYS_ADMIN or a few others), limited syscall attack surface (you generally have a syscall allowlist via seccomp-bpf), etc. Yes, it's wrong to not understand that the runc process runs as root. But I think it's only very slightly less wrong to claim that runc has root access in the way that, say, ssh root@host-system has root access. It mostly does not, and this vulnerability is notable in that it's one of the few ways to exercise that root privilege. > We looked at this at $work and got into a serious rabbit hole about how exactly Linux capabilities work. I think if I started asking interviewees to explain permitted vs. effective capability sets and how file and process capabilities differ, I'd never hire anyone. (And I think to figure out yourself how to "correctly configure" a container, you need to have at least some understanding of that.) I think you've hit the nail on the head. I only ask those interview questions because I believe it's important to find out just how much a candidate understands. I have to admit I've let some of these things go, otherwise I'd never hire anyone either. I think in the end what it comes down to is that Docker is an ambitious project that is somewhat flawed from a security perspective. There have been numerous namespace vulnerabilities to date and I expect there will be plenty more found in the future.

nicoburns 8y ago
I believe there was work on non-root container host processes going on at some point? Did that ever get to a usable state?
- geofft 8y ago
  Rootless containers using user namespaces was merged into runc in March 2017: https://github.com/opencontainers/runc/pull/774 https://github.com/opencontainers/runc/pull/774 A week ago Docker gained support for running dockerd as non-root: https://github.com/moby/moby/pull/38050 https://github.com/moby/moby/pull/38050 And there is this project for running Kubernetes as non-root: https://github.com/rootless-containers/usernetes https://github.com/rootless-containers/usernetes
- cyphar 8y ago
  Yes, it's in a usable state. Docker just merged a PR that allows you to run it rootless[1]. Props to Akihiro for bringing this one over the line, I didn't think it'd be possible three years ago. [1]: https://github.com/moby/moby/commit/ec87479b7e2bf6f1b5bcc657a377c6e6a847574f https://github.com/moby/moby/commit/ec87479b7e2bf6f1b5bcc657...

deleted 8y ago

[deleted]

cyphar 8y ago

> I think it's subtler than that. It is mostly safe to run a contained process as "root" because in theory the ways that root access can be exercised is highly sandbox by the use of various namespaces, as well as things like capability restrictions (you generally don't have CAP_SYS_ADMIN or a few others), limited syscall attack surface (you generally have a syscall allowlist via seccomp-bpf), etc. I disagree. It is definitely "safer" than running as root on the host -- but that shouldn't be the baseline. Honestly, I would argue that not using user namespaces (or non-root inside the container) is basically negligence at this point. Yes, it's annoying to do it with Docker, but other runtimes have solved this problem. LXC actually explicitly states that privileged (non-userns) containers are fundamentally unsafe and I agree. When you look at the wide array of ns_capable and other userns checks that protect against all sorts of attacks, you really start to not trust anything that doesn't use user namespaces. Kernel developers assume that container runtimes are using user namespaces if they are trying to secure something. Additionally, yes capabilities help. But you still have traditional Unix DAC issues (which is what is leveraged here). > and this vulnerability is notable precisely because it's one of the rare ways to exercise that root privilege outside the container. I would argue there's several very foundational security problems (which I'm trying to fix) that are made significantly worse by running a container process as root. Please don't do it.

geofft 8y ago
Can you use user namespaces with a released version of Docker or Kubernetes? I think the answer is no? (Or can you do it with Kubernetes + some other container runtime?) We run Kubernetes + Docker with a policy of no root inside containers (we map you to your normal UID inside the container), but most of the rabbit hole we got into was trying to figure out the implications of setuid binaries inside the container. It seems like on a normal system, setuid binaries inside a container do in fact get host root unless you tell Docker to drop those capabilites, and also fscaps aren't really usable in a container image, I think.
- cyphar 8y ago
  User namespaces have been supported in Docker since 1.10. I don't think that it's necessarily "supported" in Kubernetes -- there was a KEP to add support last year but it's still a while away. As for setuid and fscaps, they both work in containers (container images can contain them but some filesystems don't support xattrs such as AUFS). And yes, without user namespaces, they escalate to host root. You can use no_new_privs which blocks things like setuid or fscap but it also can cause problems (though I think you can enable no_new_privs in Kubernetes).