7 ms·

Spawn() of Satan (2018)

44 points by ingve 3y ago

jjgreen 3y ago
Interesting (though [2018]), dead-links to a "new improved" implementation, anyone know if that went anywhere?
- tavianator 3y ago
  The file got moved and renamed since then, it's here now: https://github.com/tavianator/bfs/blob/main/src/xspawn.h https://github.com/tavianator/bfs/blob/main/src/xspawn.h I'll fix the link, thanks
  - jjgreen 3y ago
    Ah, should have looked further down the page -- that's pretty slick (and clear) code, thanks!
the8472 3y ago
clone/exec via io_uring will hopefully end this madness. https://www.youtube.com/watch?v=jrA-y1HG31E https://www.youtube.com/watch?v=jrA-y1HG31E https://lpc.events/event/16/contributions/1213/attachments/1012/1945/io-uring-spawn.pdf https://lpc.events/event/16/contributions/1213/attachments/1... https://lwn.net/Articles/908268/ https://lwn.net/Articles/908268/
mpweiher 3y ago
I think Mach got this right, going the other direction from fork(). Instead of mashing even more functionality into one function, disaggregate more. You create a new task, which may or may not inherit your address space. This new task does not contain any threads. You can then give it address space as needed (for example if you didn't set it to inherit any). You can then create threads within the new task and start them as needed. And yes, having a separate task without threads can be useful.
seiferteric 3y ago
> But despite the apparent strangeness, fork() and exec() are actually pretty convenient once you want to do anything slightly more complicated than just launch a new process. What I don't understand is why there isn't something for the simple case where we do just want to run another process and not set anything else. There is system() which I believe is just a wrapper for fork/exec, and is tempting to use, but the issue is it just takes a string and passes it to sh which opens you up to shell expansion exploits if you have untrusted/improperly sanitized inputs. I wish there was a version of system() that took an array or args like exec instead and you could specify the command to run directly as well, instead of sh.
- dale_glass 3y ago
  That's exactly how it works in Perl. strace -s 80 -f perl -e 'system("ls", "/");' 2>&1 | grep exec ... [pid 1034640] execve("/usr/bin/ls", ["ls", "/"], 0x55b7f79873b0 /* 100 vars */) = 0
- jcranmer 3y ago
  What you're looking for is essentially posix_spawn, if you set the file_actions and attributes parameters to NULL. Then it becomes a function where you specify the executable, arguments, and environment for the new process, with a single error to process.
  - thayne 3y ago
    But posix_spawn is still implemented with fork/exec (or maybe vfork/exec) and as noted in the blog, has an unnecessarily complicated interface.
- adrian_b 3y ago
  As you have yourself said, system() is not a wrapper for fork and exec, but it executes a shell process and it gives to the shell a command to execute, which may be a binary executable or a script. Fork() and exec() are POSIX functions, while system() is a Standard C function that is available in C regardless of the operating system and regardless whether fork() and exec() exist.
dale_glass 3y ago
Unix is kind of crap with process management. That doesn't even begin to cover the issues. Eg, PIDs may be quickly reused, so having a PID is no guarantee that you're going to send a signal to the right process. Signals suck. Handles get passed to children, which can cause weird issues. Information about processes is racy -- if you want to know something about a process you have to look in /proc, and you could have that data disappear if the process exits, or get replaced with a different one mid-execution. On a higher level, things aren't much better. Bash is a constant annoyance if you want to actually accurately pass arguments to commands, and not have things break from an extra space. What surprises me the most is that until systemd there was very little effort made to improve the situation. You'd think that after decades and thousands of init scripts dealing with the same PID file nonsense and other things that go in init scripts often in buggy ways somebody would think that a fix is needed, but no, it took until 2010.
- gavinhoward 3y ago
  systemd was hardly the only one working on the problem. It was actually not even first. It just won. And it's still not ideal. Not even close.
  - dale_glass 3y ago
    It's not ideal, but it's the best solution I know of. systemd actually went for as complete of a redesign as it could. Rather than just patch one issue up it tried to improve almost everything at once. Which is a fresh approach compared to what we've had before -- every distro having their own flavor of SysV helper scripts, which only do part of the job and all slightly differently.
    - gavinhoward 3y ago
      Other projects tried to do that too. And before systemd. The unfortunate lesson of systemd is that playing politics will help you win over your competitors.
      - 3y ago
thayne 3y ago
There are two syscalls I really wish linux had (well maybe more, but these are the ones I've thought about the most, and are relevant here): 1. A spawn syscall that can replace fork/exec, that defaults to only inheriting the "std" file descriptors (0, 1, and 2), but allows passing others in an array. Has options for replacing the environment and the working directory, decreasing privileges (including capabilities and namespaces), etc. And has a way to return the pidfd for the child process. 2. Something like fork, but it only copies readonly program memory, and instead of returning in the child process, it calls a function, essentially the main function for the child process). This would be useful if you want to fork a child process using the same executable, but unlike fork can actually be thread safe.

44 points by ingve 3y ago

jjgreen 3y ago

Interesting (though [2018]), dead-links to a "new improved" implementation, anyone know if that went anywhere?

tavianator 3y ago
The file got moved and renamed since then, it's here now: https://github.com/tavianator/bfs/blob/main/src/xspawn.h https://github.com/tavianator/bfs/blob/main/src/xspawn.h I'll fix the link, thanks
- jjgreen 3y ago
  Ah, should have looked further down the page -- that's pretty slick (and clear) code, thanks!

the8472 3y ago

clone/exec via io_uring will hopefully end this madness. https://www.youtube.com/watch?v=jrA-y1HG31E https://www.youtube.com/watch?v=jrA-y1HG31E https://lpc.events/event/16/contributions/1213/attachments/1012/1945/io-uring-spawn.pdf https://lpc.events/event/16/contributions/1213/attachments/1... https://lwn.net/Articles/908268/ https://lwn.net/Articles/908268/

mpweiher 3y ago

I think Mach got this right, going the other direction from fork(). Instead of mashing even more functionality into one function, disaggregate more. You create a new task, which may or may not inherit your address space. This new task does not contain any threads. You can then give it address space as needed (for example if you didn't set it to inherit any). You can then create threads within the new task and start them as needed. And yes, having a separate task without threads can be useful.

seiferteric 3y ago

> But despite the apparent strangeness, fork() and exec() are actually pretty convenient once you want to do anything slightly more complicated than just launch a new process. What I don't understand is why there isn't something for the simple case where we do just want to run another process and not set anything else. There is system() which I believe is just a wrapper for fork/exec, and is tempting to use, but the issue is it just takes a string and passes it to sh which opens you up to shell expansion exploits if you have untrusted/improperly sanitized inputs. I wish there was a version of system() that took an array or args like exec instead and you could specify the command to run directly as well, instead of sh.

dale_glass 3y ago
That's exactly how it works in Perl. strace -s 80 -f perl -e 'system("ls", "/");' 2>&1 | grep exec ... [pid 1034640] execve("/usr/bin/ls", ["ls", "/"], 0x55b7f79873b0 /* 100 vars */) = 0
jcranmer 3y ago
What you're looking for is essentially posix_spawn, if you set the file_actions and attributes parameters to NULL. Then it becomes a function where you specify the executable, arguments, and environment for the new process, with a single error to process.
- thayne 3y ago
  But posix_spawn is still implemented with fork/exec (or maybe vfork/exec) and as noted in the blog, has an unnecessarily complicated interface.
adrian_b 3y ago
As you have yourself said, system() is not a wrapper for fork and exec, but it executes a shell process and it gives to the shell a command to execute, which may be a binary executable or a script. Fork() and exec() are POSIX functions, while system() is a Standard C function that is available in C regardless of the operating system and regardless whether fork() and exec() exist.

dale_glass 3y ago

Unix is kind of crap with process management. That doesn't even begin to cover the issues. Eg, PIDs may be quickly reused, so having a PID is no guarantee that you're going to send a signal to the right process. Signals suck. Handles get passed to children, which can cause weird issues. Information about processes is racy -- if you want to know something about a process you have to look in /proc, and you could have that data disappear if the process exits, or get replaced with a different one mid-execution. On a higher level, things aren't much better. Bash is a constant annoyance if you want to actually accurately pass arguments to commands, and not have things break from an extra space. What surprises me the most is that until systemd there was very little effort made to improve the situation. You'd think that after decades and thousands of init scripts dealing with the same PID file nonsense and other things that go in init scripts often in buggy ways somebody would think that a fix is needed, but no, it took until 2010.

gavinhoward 3y ago
systemd was hardly the only one working on the problem. It was actually not even first. It just won. And it's still not ideal. Not even close.
- dale_glass 3y ago
  It's not ideal, but it's the best solution I know of. systemd actually went for as complete of a redesign as it could. Rather than just patch one issue up it tried to improve almost everything at once. Which is a fresh approach compared to what we've had before -- every distro having their own flavor of SysV helper scripts, which only do part of the job and all slightly differently.
  - gavinhoward 3y ago
    Other projects tried to do that too. And before systemd. The unfortunate lesson of systemd is that playing politics will help you win over your competitors.
    - 3y ago

thayne 3y ago

There are two syscalls I really wish linux had (well maybe more, but these are the ones I've thought about the most, and are relevant here): 1. A spawn syscall that can replace fork/exec, that defaults to only inheriting the "std" file descriptors (0, 1, and 2), but allows passing others in an array. Has options for replacing the environment and the working directory, decreasing privileges (including capabilities and namespaces), etc. And has a way to return the pidfd for the child process. 2. Something like fork, but it only copies readonly program memory, and instead of returning in the child process, it calls a function, essentially the main function for the child process). This would be useful if you want to fork a child process using the same executable, but unlike fork can actually be thread safe.