9 ms·
Why strace doesn't work in Docker
- enriquto 7y agoI love this writing style where the scientific method is laid bare so explicitly.
- blitmap 7y agoAgree! Wish I had the mental capacity to stay focused like this person :-)
- asdf21 7y ago>strace actually does work in newer versions of Docker Good to know, this should open up some troubleshooting
- tonetheman 7y agoA great writeup. And they end up looking at the source in the article. Good stuff.
- acqq 7y agoI also recommend: https://wizardzines.com/ https://wizardzines.com/ by the same author. It looks "popular" but it's written much more thoughtfully and competently than most of the content found on the web. Like this article, it's the result of real personal research, it is never just rewording of some manual or already existing material. In short, really worth reading, and also worth using inside of the companies.
- codazoda 7y agoAs a fan of zines, I've followed her for a while now. She does awesome work.
- NikolaeVarius 7y agoI find for Docker, the documentation is extremely sparse/omits details that have larger consequences. I've gotten used to just going to read the source to see what an option does.
- StreamBright 7y agoImagine a situation what something is broken and you need to figure out how to strace in production while a service is down. This is my main argument against Docker, it is black box computing and you are going to figure this out the hard way. Of course you can prepare, yet, I do not necessarily have the bandwidth.
- justincormack 7y agoAs the article points out, in any recent version of Docker and up to date kernel ptrace just works.
- mehrdadn 7y agoWow this was way more interesting than I expected. Cool of her to share!
- drvdevd 7y agoYeah this was a great write up and an interesting example of how the reality of the implementation of something can be quite mismatched to expectations. I wouldn’t necessarily call it surprising that the implementation is doing more than one would expect, but this is a good reminder of how to look around when things get weird.
- aritmo 7y agoWhy strace didn't work in Docker.
- croo 7y ago> This hypothesis doesn’t make much sense but I hadn’t realized that the root user in a Docker container is the same as the root user on the host, so I thought that was interesting Wait what? I didn't know that. This sounds terrible. Is the root in the container the same user who runs the container, or the same root who is root on the host machine?
- maccam94 7y agoDocker doesn't use user namespaces by default. They have been working on adding support for a while, but it makes permissions on volumes difficult, and things kept changing in kernel land for years. Root inside a non-userns container is the same as root on the Linux host, but it is constrained by security policies like seccomp and apparmor.
- tannhaeuser 7y ago> it makes permissions on volumes difficult Yeah much easier to run everything as root. Also, we don't need /etc/password and stuff for UID/GID resolution and credentials in the container anyway; we just use ad-hoc auth and crypto from a random third-party lib (that isn't vetted, is never updated, runs in the same address space as your app, and hasn't access to meaningful entropy since running in a container) and supply root credentials on the docker command line or the Dockerfile checked in to github, or both. That's what we've been saying for years: Docker doesn't solve anything, it merely hides problems from you (and helps your cloud provider's bottom line). Good luck, and yes, PHBs should be worried for civil/criminal gross negligence if the shit hits the fan. Your cloud provider is happy to take your checks, but will shrug-away and point out they're just providing the infrastructure; it's up to you to competently configure your ever-changing 12-factor k8s.
- jcranmer 7y ago> hasn't access to meaningful entropy since running in a container getrandom(2)/getentropy(3) should be using kernel randomness generation, which isn't affected by containers I thought.
- 7y ago
- lmilcin 7y agostrace/ptrace gives you huge amount of power. I had a discussion once with admins at my company that wanted to convert servers to have AD accounts in a naive way. The discussion could be shortened to something like this: Me: please, don't convert servers to AD because my password will be available to everybody as everybody has root on test environment. Admins: no, this is perfectly safe. Me: Guys, once you have root on machine you can do everything including accessing passwords as they are being typed. Admins: No, not possible. Perfectly safe. Me: Here, I ran the tool using ptrace from root account, here is a list of all accounts and passwords of all users using test env. This is actually read() call at some point, you just need to be able to filter it out from strace output. Admins: Wtf! You are going to get fired!
- tw04 7y agoI'm not sure what AD has to do with the fact as root, you have a huge amount of power on ANY box and abusing it will get you fired. strace/ptrace is just another tool in that toolbox that you can use maliciously. Your company either runs in a model where it trusts the employees or it doesn't. If it doesn't trust the employees they shouldn't be giving ANY of them root on a shared system.
- filleokus 7y agoIn my case, I'm using a different, easy password, for test servers that contain no important production data or are in a especially privileged network position. If they were connected to AD, I would be forced to present my "real" password. On production servers on the other hand, the amount of people with root is probably much smaller, and there might be bigger wins by being able to centrally enforce password policies or whatever the sysadmins want to do.
- lmilcin 7y agoThat was exactly the point. I had separate password on each box (which I did not have to use normally because I use it only to setup ssh key).
- 7y ago
- amelius 7y agoIf you think that fixed it, try stracing the container AND running strace inside the container at the same time.
- roblabla 7y agoThis is interesting, but stops just short of explaining why ptrace wasn't whitelisted, and what changed that allows it to be whitelisted on recent kernel versions. Here's what the linked commit to docker[0] says: > 4.8+ kernels have fixed the ptrace security issues so we can allow ptrace(2) on the default seccomp profile if we do the kernel version check. This commit itself links a commit in the linux kernel[1], which says: > x86/ptrace: run seccomp after ptrace > This moves seccomp after ptrace on x86 to that seccomp can catch changes made by ptrace. Emulation should skip the rest of processing too. This doesn't give us much more information, and I'm not familiar enough with this code to understand the changeset. Thankfully, by looking up the changeset name, we can easily find the email that proposed the changeset[2], which explains the issue more in-depth: > There has been a long-standing (and documented) issue with seccomp where ptrace can be used to change a syscall out from under seccomp. This is a problem for containers and other wider seccomp filtered environments where ptrace needs to remain available, as it allows for an escape of the seccomp filter. So the basic idea is to use ptrace to swap an allowed syscall with a forbidden syscall. Seccomp used to run before ptrace, so it would see the allowed syscall and allow the code to continue, but then ptrace would swap the syscall to something forbidden! The details are a bit thin, however, they don't explain how we can do that. After a bit more digging, I found this pdf[3] which contains a simple POC showing the issue, and explaining it. The TLDR is to use ptrace with `PTRACE_SYSCALLS` to execute until a syscall is hit. When a syscall is hit, seccomp would first check the syscall is allowed, and then pass execution to the tracing system of the kernel, that would stop the process for ptrace to inspect. From ptrace you can then modify the registers with `PTRACE_SET_REGS` to change the syscall being called (the syscall number is in register RAX on amd64) and resume execution. The kernel will then happily execute the modified, unchecked syscall! So what changed is that seccomp will be run after ptrace now, so there isn't a way to modify the registers before the syscall is run anymore. [0]: https://github.com/moby/moby/commit/1124543ca8071074a537a15db251af46a5189907 https://github.com/moby/moby/commit/1124543ca8071074a537a15d... [1]: https://github.com/torvalds/linux/commit/93e35efb8de45393cf61ed07f7b407629bf698ea https://github.com/torvalds/linux/commit/93e35efb8de45393cf6... [2]: https://www.mail-archive.com/linuxppc-dev@lists.ozlabs.org/msg104378.html https://www.mail-archive.com/linuxppc-dev@lists.ozlabs.org/m... [3]: http://asm.rajiska.fr/misc/container_security.pdf http://asm.rajiska.fr/misc/container_security.pdf page 9 for the explanation, page 16 for the POC code
- carapace 7y agoThat was fun but it pretty much reinforces my main reason for not using containers: it's reinventing the OS. I already have the OS, so... (Change my mind?)
- techntoke 7y agoNo need to change your mind. Containers improve orchestration of deploying scalable services. The Kubernetes ecosystem will continue to improve and one day your boss will tell you that you have to use containers to deploy your apps or find a different job.
- carapace 7y agoThat's so not what I meant and I think you know it. I don't mean to be rude but your response make you sound like a systemd enthusiast. Do you have a technical argument to put forward?
- ketzu 7y agoI mean, "systemd enthusiast" doesn't sound rude, unless you give it some tone, are part of a special group where it is, or prefix it with "i don't mean to be rude". For the difference of Docker and reinventing OSes, it depends on what you establish as reinventing OSes. Docker provides some form of separation/containerization, a function that is also implemented by some OS. But it's discussable of having the functionality in user or kernelspace or having a wrapper application with a consistent api across multiple operating systems isn't worth it.
- carapace 7y agoI had just read "Systemd, ten years later: a historical and technical retrospective" so it was on my mind. I know they didn't invent it but I associate that kind of attitude with that project. Shouldn't have mentioned it, really. https://news.ycombinator.com/item?id=23062072 https://news.ycombinator.com/item?id=23062072 https://blog.darknedgy.net/technology/2020/05/02/0/ https://blog.darknedgy.net/technology/2020/05/02/0/
- deleted 7y ago[deleted]
- jeffbee 7y agoTl;dr: because docker is doing a bunch of undocumented junk you didn’t ask for.
- jsnk 7y agoI had a memory leak problem running a node service inside a docker container. I tried to use gcore, but got the precise problem in the article. ptrace: Operation not permitted. You can't do that without a process to debug. The program is not being run. gcore: failed to create core.36 I found this SO answer and it worked out for me. https://stackoverflow.com/questions/42029834/gdb-in-docker-container-returns-ptrace-operation-not-permitted https://stackoverflow.com/questions/42029834/gdb-in-docker-c...
- richardwhiuk 7y agoPreviously: https://news.ycombinator.com/item?id=23021002 https://news.ycombinator.com/item?id=23021002