4 ms·
> In particular, it is not always possible for HQ to ensure that when a process that spawns tasks (called worker) quits unexpectedly (e.g. when it receives SIGK
by deathanatos 2y ago
> In particular, it is not always possible for HQ to ensure that when a process that spawns tasks (called worker) quits unexpectedly (e.g. when it receives SIGKILL), its spawned tasks will be cleaned up. Sadly, Linux does not seem to provide any way of implementing perfect structured process management in user space. In other words, when a parent process dies, it is possible for its (grand)children to continue executing.
Uh, it does? It's called pid-namesp—
> There is a solution for this called PID namespaces,
I think maybe you've got the wrong idea about what "in user space" means? — processes running as root are still "in user space". The opposite of "user space" is "in the kernel".
> but it requires elevated privileges
I think that's only technically true. I believe you can unshare the PID namespace if you first unshare the user namespace — which causes the thing doing the unsharing of the user namespace to become "root" within that new namespace, and from there is permitted to unshare the pid namespace. I think: https://unix.stackexchange.com/a/672462/6013 https://unix.stackexchange.com/a/672462/6013
I have no idea why that hoop has to be jumped through / I don't know what is being protected against by preventing unprivileged processes from making pid namespaces.
Whether or not that fits well with HQ's design … you'd have to be the judge of that.
There's also prctl(PR_SET_CHILD_SUBREAPER, ...)
- kobzol 2y agoYeah by user space I just meant without root, sorry. HQ runs on supercomputers where the environment is heavily locked up, even Docker doesn't work. I think that PID namespaces aren't really possible, but I haven't tried it yet. Subreaper doesn't help, because if the worker dies, the children aren't killed, even if they are the children of the worker, they will be just reparented to init.
- fc417fc802 2y ago> I think that PID namespaces aren't really possible Depends on the cluster. If they're using nix or guix then they presumably enabled user namespaces but a few years ago guix had an article about (generally shitty) workarounds for people running in environments where those were disabled. Edit: Maybe you should have two code paths. A fast namespaced one and the slower old one as a fallback.
- zamalek 2y agoFWIW you can unshare PID and user at the same time: https://github.com/porkg/porkg/blob/rs/crates/porkg-linux/src/sandbox.rs#L222 https://github.com/porkg/porkg/blob/rs/crates/porkg-linux/sr... If you don't care about being able to use different uids and gids then simply become root in the new namespace: https://github.com/porkg/porkg/blob/rs/crates/porkg-linux/src/proc.rs#L212 https://github.com/porkg/porkg/blob/rs/crates/porkg-linux/sr... . Root inside the namespace will then be equivalent to the original uid+gid outside. I am using clone, which has the very important caveat: more than one thread running is UB. That's why I use a zygote (a process forked from the root very early on - i.e. before starting the tokio runtime). You can probably avoid all of that by using exec+unshare. But, given you're running on old kernels and constrained environments this may be not possible at all. Maybe make it configurable?
- LegionMammal978 2y agoUbuntu [0] and some other distros have been trending towards disabling unprivileged user namespaces, unless you have specific AppArmor capabilities or other such mechanisms. So it's not something you can count on being available, unfortunately. (At least, not without jumping through many hoops to satisfy every distro's maintainers.) I've also had some ideas that have been stymied by a lack of unprivileged user namespaces. [0] https://ubuntu.com/blog/ubuntu-23-10-restricted-unprivileged-user-namespaces https://ubuntu.com/blog/ubuntu-23-10-restricted-unprivileged...