6 ms·

Or you can use the open + O_PATH + *at syscall family which can be used to get a handle on a resolved directory relative to which you can manipulate with no re-

Or you can use the open + O_PATH + *at syscall family which can be used to get a handle on a resolved directory relative to which you can manipulate with no re-traversal happening for different operations on that handle. This combo exists exactly to avoid these kinds of issues. Another way would be temporarily joining the container's mount namespace to obtain the source handle. But that can't really be done in go since goroutines don't play nicely with per-thread operations. Edit: After looking through the go standard library it seems that there is an impedance mismatch. Go just does not expose the necessary pieces to do this properly. A dedicated docker-cp tool in C or Rust could probably handle this better. I could be wrong though, maybe it's just not part of the stdlib.

kazinator 7y ago
But that starting point has to be initially checked somehow. Say we are user "bob" we open "/foo/bar/xyzzy" (for later use with openat), and, say, "bar" is writable to "mallory", then that isn't a path we can trust in the first place. O_PATH is a Linux Kernel 4.x thing; can't find it in POSIX. Won't compile on non-Linux POSIX operating systems or older kernels. It seems to be an optimization; if we omit it, we just get a "fatter" file descriptor that takes more cycles to set up.
- the8472 7y ago
  You can perform the verification after having obtained the dirfd. E.g. by doing a reverse lookup through procfs to see where it actually points. Or by walking the .., making sure you never leave the container root. It'll certainly be easier to get right with the newly proposed syscalls. But you can also get it right with the current ones. > O_PATH is a Linux Kernel 4.x thing; can't find it in POSIX. I don't think that is relevant to docker which relies on many linux-specific APIs anyway.
  - anaphor 7y ago
    I think another way might be to have a set of file descriptors you know correspond to safe paths before doing anything, as you say (they could be dirfds which let you traverse downwards). Then when the `docker cp` command is executed, it does not open any new files, it simply looks up the corresponding fd from a table mapping paths to fds. If the file no longer exists, or it doesn't have a fd matching that path, then error out. The construction could be based on where you're copying to/from on the host system. That way, even if the symlink somehow gets resolved to a bad path, it will refuse to read it because it does not exist from the point of view of the `docker cp` command. I.e. make the command use a "principle of least authority" where it cannot even see files outside of a certain set of paths, let alone be tricked into opening them by a symlink.
anaphor 7y ago
https://github.com/google/capsicum-linux https://github.com/google/capsicum-linux would basically render this entire class of bug impossible if used correctly. It would probably require a redesign of how docker interacts with the host filesystem though, and obviously relies on something that isn't yet in the mainline Linux.
cyphar 7y ago
Just using O_PATH won't completely save you. O_PATH can definitely help, especially if you actually check that the path is what you think through readlink(/proc/self/fd/$n), but doing it safely is quite a bit harder than just using *at(2). As mentioned in TFA, I am working on some kernel patches (which will probably result in an openat2(2)) which allow for restriction of path resolution so you can block symlink crossings entirely or force symlinks to resolve to the dirfd you passed.
- deleted 7y ago
  [deleted]
- the8472 7y ago
  Yeah, I have read about the precursors of those patches. But this has been simmering for a long time. The point is that one could do this correctly today with the right rituals. openat2 wouldn't save you if you were still doing plain realpath+open across a security boundary even though openat+procfs or setns are available. The new syscall implementation would still need a fallback impl. for older kernels after all.
  - cyphar 7y ago
    > The new syscall implementation would still need a fallback impl. for older kernels after all. Yup, this is why I'm planning on getting a sane API available in <https://github.com/cyphar/filepath-securejoin> https://github.com/cyphar/filepath-securejoin> which projects can use so that the correct thing is done with both old and new kernels. Right now it has a (slighly) improved version of the code Docker has, but I'm rewriting it. It should be noted that there are lots of examples of interfaces which are incredibly hard to make safe without openat2 -- such as O_CREAT.
paulfurtado 7y ago
It's certainly possible to call nsenter in Go but even so, unlike other namespaces, mount namespaces affect the entire process rather than threads, and the thread must not have launched any threads yet at the time that nsenter is called. That said, docker does support `docker exec` which gives you a shell inside the container's namespaces and cgroups. I'd imagine they could do something similar and just not call exec once they've entered the container. This would be similar to calling `docker exec $containerid cat /path/to/file`
- cyphar 7y ago
  The problem is that doing this for every file operation becomes ludicrously expensive -- in Go you cannot use vfork and the only (safe) fork primitive is fork+exec. You could have a surrogate process in the container but then the code is significantly more complex (all of the operations need to be fully implemented in the surrogate process and you'll need to figure out a safe way of doing IPC without the container being able to attack it). You also definitely wouldn't want to be running container runtime code inside all of the namespaces in the container -- this is hard to do safely and should be avoided (there are at least two CVEs related to the joining phase of just runc found in the past few years).
  - the8472 7y ago
    You don't necessarily need IPC. You only have to spawn a single-threaded child process that can call setns to switch back and forth between mount namespaces. You can also obtain file descriptors in batches to reduce the overhead. I wonder how an O_PATH handle from a different namespace behaves once you switch back. If *at lookups are performed under its original namespace you only have to obtain it once.
    - cyphar 7y ago
      O_PATH operations are performed under the original namespace, and you don't even need to join the mount namespace -- you could just do an O_PATH of /proc/$pid/root which pipes you directly to the root of the mount namespace. We came up with this idea before on LKML and I'm trying to remember what the issue with this solution was. It's definitely better than the current method by Docker (and actually, I might be able to get this to work within Docker more simply). It wouldn't work with rootless containers, but you could fix that somewhat trivially by switching to the userns of the container's pid1. Since /proc/self doesn't exist, obvious attacks through that such as /proc/self/root won't work. There is a somewhat esoteric problem, which is that you could trick the process into opening an O_PATH stashed by a bad container process -- but right now there is a pretty big flaw in how the kernel handles this problem anyway that I'm also trying to solve upstream.

kazinator 7y ago

But that starting point has to be initially checked somehow. Say we are user "bob" we open "/foo/bar/xyzzy" (for later use with openat), and, say, "bar" is writable to "mallory", then that isn't a path we can trust in the first place. O_PATH is a Linux Kernel 4.x thing; can't find it in POSIX. Won't compile on non-Linux POSIX operating systems or older kernels. It seems to be an optimization; if we omit it, we just get a "fatter" file descriptor that takes more cycles to set up.

the8472 7y ago
You can perform the verification after having obtained the dirfd. E.g. by doing a reverse lookup through procfs to see where it actually points. Or by walking the .., making sure you never leave the container root. It'll certainly be easier to get right with the newly proposed syscalls. But you can also get it right with the current ones. > O_PATH is a Linux Kernel 4.x thing; can't find it in POSIX. I don't think that is relevant to docker which relies on many linux-specific APIs anyway.
- anaphor 7y ago
  I think another way might be to have a set of file descriptors you know correspond to safe paths before doing anything, as you say (they could be dirfds which let you traverse downwards). Then when the `docker cp` command is executed, it does not open any new files, it simply looks up the corresponding fd from a table mapping paths to fds. If the file no longer exists, or it doesn't have a fd matching that path, then error out. The construction could be based on where you're copying to/from on the host system. That way, even if the symlink somehow gets resolved to a bad path, it will refuse to read it because it does not exist from the point of view of the `docker cp` command. I.e. make the command use a "principle of least authority" where it cannot even see files outside of a certain set of paths, let alone be tricked into opening them by a symlink.

anaphor 7y ago

https://github.com/google/capsicum-linux https://github.com/google/capsicum-linux would basically render this entire class of bug impossible if used correctly. It would probably require a redesign of how docker interacts with the host filesystem though, and obviously relies on something that isn't yet in the mainline Linux.

cyphar 7y ago

Just using O_PATH won't completely save you. O_PATH can definitely help, especially if you actually check that the path is what you think through readlink(/proc/self/fd/$n), but doing it safely is quite a bit harder than just using *at(2). As mentioned in TFA, I am working on some kernel patches (which will probably result in an openat2(2)) which allow for restriction of path resolution so you can block symlink crossings entirely or force symlinks to resolve to the dirfd you passed.

deleted 7y ago
[deleted]
the8472 7y ago
Yeah, I have read about the precursors of those patches. But this has been simmering for a long time. The point is that one could do this correctly today with the right rituals. openat2 wouldn't save you if you were still doing plain realpath+open across a security boundary even though openat+procfs or setns are available. The new syscall implementation would still need a fallback impl. for older kernels after all.
- cyphar 7y ago
  > The new syscall implementation would still need a fallback impl. for older kernels after all. Yup, this is why I'm planning on getting a sane API available in <https://github.com/cyphar/filepath-securejoin> https://github.com/cyphar/filepath-securejoin> which projects can use so that the correct thing is done with both old and new kernels. Right now it has a (slighly) improved version of the code Docker has, but I'm rewriting it. It should be noted that there are lots of examples of interfaces which are incredibly hard to make safe without openat2 -- such as O_CREAT.

paulfurtado 7y ago

It's certainly possible to call nsenter in Go but even so, unlike other namespaces, mount namespaces affect the entire process rather than threads, and the thread must not have launched any threads yet at the time that nsenter is called. That said, docker does support `docker exec` which gives you a shell inside the container's namespaces and cgroups. I'd imagine they could do something similar and just not call exec once they've entered the container. This would be similar to calling `docker exec $containerid cat /path/to/file`

cyphar 7y ago
The problem is that doing this for every file operation becomes ludicrously expensive -- in Go you cannot use vfork and the only (safe) fork primitive is fork+exec. You could have a surrogate process in the container but then the code is significantly more complex (all of the operations need to be fully implemented in the surrogate process and you'll need to figure out a safe way of doing IPC without the container being able to attack it). You also definitely wouldn't want to be running container runtime code inside all of the namespaces in the container -- this is hard to do safely and should be avoided (there are at least two CVEs related to the joining phase of just runc found in the past few years).
- the8472 7y ago
  You don't necessarily need IPC. You only have to spawn a single-threaded child process that can call setns to switch back and forth between mount namespaces. You can also obtain file descriptors in batches to reduce the overhead. I wonder how an O_PATH handle from a different namespace behaves once you switch back. If *at lookups are performed under its original namespace you only have to obtain it once.
  - cyphar 7y ago
    O_PATH operations are performed under the original namespace, and you don't even need to join the mount namespace -- you could just do an O_PATH of /proc/$pid/root which pipes you directly to the root of the mount namespace. We came up with this idea before on LKML and I'm trying to remember what the issue with this solution was. It's definitely better than the current method by Docker (and actually, I might be able to get this to work within Docker more simply). It wouldn't work with rootless containers, but you could fix that somewhat trivially by switching to the userns of the container's pid1. Since /proc/self doesn't exist, obvious attacks through that such as /proc/self/root won't work. There is a somewhat esoteric problem, which is that you could trick the process into opening an O_PATH stashed by a bad container process -- but right now there is a pretty big flaw in how the kernel handles this problem anyway that I'm also trying to solve upstream.