5 ms·

It's certainly possible to call nsenter in Go but even so, unlike other namespaces, mount namespaces affect the entire process rather than threads, and the thre

It's certainly possible to call nsenter in Go but even so, unlike other namespaces, mount namespaces affect the entire process rather than threads, and the thread must not have launched any threads yet at the time that nsenter is called. That said, docker does support `docker exec` which gives you a shell inside the container's namespaces and cgroups. I'd imagine they could do something similar and just not call exec once they've entered the container. This would be similar to calling `docker exec $containerid cat /path/to/file`

cyphar 7y ago
The problem is that doing this for every file operation becomes ludicrously expensive -- in Go you cannot use vfork and the only (safe) fork primitive is fork+exec. You could have a surrogate process in the container but then the code is significantly more complex (all of the operations need to be fully implemented in the surrogate process and you'll need to figure out a safe way of doing IPC without the container being able to attack it). You also definitely wouldn't want to be running container runtime code inside all of the namespaces in the container -- this is hard to do safely and should be avoided (there are at least two CVEs related to the joining phase of just runc found in the past few years).
- the8472 7y ago
  You don't necessarily need IPC. You only have to spawn a single-threaded child process that can call setns to switch back and forth between mount namespaces. You can also obtain file descriptors in batches to reduce the overhead. I wonder how an O_PATH handle from a different namespace behaves once you switch back. If *at lookups are performed under its original namespace you only have to obtain it once.
  - cyphar 7y ago
    O_PATH operations are performed under the original namespace, and you don't even need to join the mount namespace -- you could just do an O_PATH of /proc/$pid/root which pipes you directly to the root of the mount namespace. We came up with this idea before on LKML and I'm trying to remember what the issue with this solution was. It's definitely better than the current method by Docker (and actually, I might be able to get this to work within Docker more simply). It wouldn't work with rootless containers, but you could fix that somewhat trivially by switching to the userns of the container's pid1. Since /proc/self doesn't exist, obvious attacks through that such as /proc/self/root won't work. There is a somewhat esoteric problem, which is that you could trick the process into opening an O_PATH stashed by a bad container process -- but right now there is a pretty big flaw in how the kernel handles this problem anyway that I'm also trying to solve upstream.
    - cyphar 7y ago
      Sorry, I only just remembered what the issue with this solution was. Absolute symlink components in the path will "escape" the root. So it actually doesn't help overall -- you still need to verify after you've done the open. My kernel patchset for RESOLVE_IN_ROOT will scope absolute paths too -- so you definitely could combine it with /proc/$pid/root but that's not enough by itself.
      - the8472 7y ago
        Ok, so getting files from the root O_PATH without verification is not enough. But that still leaves joining the mount namespace to obtain individual fds, which can be done safely and in bulk.
- paulfurtado 7y ago
  So, `docker exec <container id> cat /path/to/file` is definitely something that works, is as safe as `docker exec`, and doesn't involve running any runtime code inside the container. The expense is that you have cat buffering the data and then the runtime reading the data from a pipe and re-buffering that data. Is that really ludicrously expensive? It doesn't seem any worse than SCP, which is just SSH executing the SCP binary as the command and communicating over the stdin/stdout channels. Doing it with cat is just an example, since cat may not be present inside the container. Instead of cat, and in the absence of fork-without-exec in go, you could execute a small C program injected similarly to `/dev/init` (or via memfd_create) that uses sendfd over a unix socket created by socketpair to pass back a file descriptor to the runtime so that the data is only buffered once.
  - cyphar 7y ago
    A privileged user in the container could ptrace(2) the process and start messing with its output. If you have an IPC protocol (like the sendfd you bring up later) then you've now opened windows for container runtime attacks. Yes, you could double (and triple) check the fd is not malicious but so many programs already don't do this properly -- depending on it being done properly is not an ideal solution. So, you don't want to join all the namespaces, only the mount namespace. But if you join the monut namespace then MS_MOVE might start messing with your resolution. So really, what you wnt to do is to just pivot_root(2) into the container's rootfs -- this is what "docker cp" already does today and it has a bug because it pivot_root(2)s into an attacker-controlled path. And all of these solutions will require a fork+exec with Go. > Is that really ludicrously expensive? If you're doing it once for every single filesystem syscall (every stat(2), readlink(2), open(2), getdents(2)) and so on then yes it gets very expensive. As I mentioned, the other solution is to run entire operations inside a pivot_root(2) but then your runtime code gets more complicated (Docker already does this and it's already hard enough to understand how things like chrootarchive actually work at the end of the day). Not to mention that (as this bug shows) even if you run inside a pivot_root(2) you might make a mistake that entirely negates the security benefits.
    - paulfurtado 7y ago

cyphar 7y ago

The problem is that doing this for every file operation becomes ludicrously expensive -- in Go you cannot use vfork and the only (safe) fork primitive is fork+exec. You could have a surrogate process in the container but then the code is significantly more complex (all of the operations need to be fully implemented in the surrogate process and you'll need to figure out a safe way of doing IPC without the container being able to attack it). You also definitely wouldn't want to be running container runtime code inside all of the namespaces in the container -- this is hard to do safely and should be avoided (there are at least two CVEs related to the joining phase of just runc found in the past few years).

the8472 7y ago
You don't necessarily need IPC. You only have to spawn a single-threaded child process that can call setns to switch back and forth between mount namespaces. You can also obtain file descriptors in batches to reduce the overhead. I wonder how an O_PATH handle from a different namespace behaves once you switch back. If *at lookups are performed under its original namespace you only have to obtain it once.
- cyphar 7y ago
  O_PATH operations are performed under the original namespace, and you don't even need to join the mount namespace -- you could just do an O_PATH of /proc/$pid/root which pipes you directly to the root of the mount namespace. We came up with this idea before on LKML and I'm trying to remember what the issue with this solution was. It's definitely better than the current method by Docker (and actually, I might be able to get this to work within Docker more simply). It wouldn't work with rootless containers, but you could fix that somewhat trivially by switching to the userns of the container's pid1. Since /proc/self doesn't exist, obvious attacks through that such as /proc/self/root won't work. There is a somewhat esoteric problem, which is that you could trick the process into opening an O_PATH stashed by a bad container process -- but right now there is a pretty big flaw in how the kernel handles this problem anyway that I'm also trying to solve upstream.
  - cyphar 7y ago
    Sorry, I only just remembered what the issue with this solution was. Absolute symlink components in the path will "escape" the root. So it actually doesn't help overall -- you still need to verify after you've done the open. My kernel patchset for RESOLVE_IN_ROOT will scope absolute paths too -- so you definitely could combine it with /proc/$pid/root but that's not enough by itself.
    - the8472 7y ago
      Ok, so getting files from the root O_PATH without verification is not enough. But that still leaves joining the mount namespace to obtain individual fds, which can be done safely and in bulk.
paulfurtado 7y ago
So, `docker exec <container id> cat /path/to/file` is definitely something that works, is as safe as `docker exec`, and doesn't involve running any runtime code inside the container. The expense is that you have cat buffering the data and then the runtime reading the data from a pipe and re-buffering that data. Is that really ludicrously expensive? It doesn't seem any worse than SCP, which is just SSH executing the SCP binary as the command and communicating over the stdin/stdout channels. Doing it with cat is just an example, since cat may not be present inside the container. Instead of cat, and in the absence of fork-without-exec in go, you could execute a small C program injected similarly to `/dev/init` (or via memfd_create) that uses sendfd over a unix socket created by socketpair to pass back a file descriptor to the runtime so that the data is only buffered once.
- cyphar 7y ago
  A privileged user in the container could ptrace(2) the process and start messing with its output. If you have an IPC protocol (like the sendfd you bring up later) then you've now opened windows for container runtime attacks. Yes, you could double (and triple) check the fd is not malicious but so many programs already don't do this properly -- depending on it being done properly is not an ideal solution. So, you don't want to join all the namespaces, only the mount namespace. But if you join the monut namespace then MS_MOVE might start messing with your resolution. So really, what you wnt to do is to just pivot_root(2) into the container's rootfs -- this is what "docker cp" already does today and it has a bug because it pivot_root(2)s into an attacker-controlled path. And all of these solutions will require a fork+exec with Go. > Is that really ludicrously expensive? If you're doing it once for every single filesystem syscall (every stat(2), readlink(2), open(2), getdents(2)) and so on then yes it gets very expensive. As I mentioned, the other solution is to run entire operations inside a pivot_root(2) but then your runtime code gets more complicated (Docker already does this and it's already hard enough to understand how things like chrootarchive actually work at the end of the day). Not to mention that (as this bug shows) even if you run inside a pivot_root(2) you might make a mistake that entirely negates the security benefits.
  - paulfurtado 7y ago