4 ms·

If you want to be able to spawn processes that fask then `fork()` is NOT your friend. You want either `vfork()` (or `clone()` equivalent) or `posix_spawn()`.

If you want to be able to spawn processes that fask then `fork()` is NOT your friend. You want either `vfork()` (or `clone()` equivalent) or `posix_spawn()`. `fork()` is inherently very slow due to the need to either copy the VM of the parent, or arrange to copy pages on write, or copy the resident set of the parent (then copy any pages paged-in when those page-in events happen -- all three of these options are very expensive. Also, what I might recommend here is to create a `posix_spawn()`-like API that gives you asynchronous notification of the exec starting or failing, that way you don't block even for that. I'd use a `pipe()` that will be set to close on exec and which will therefore close when the exec starts, but if the exec fails I'd write the `errno` value into the pipe, that way EOF on the pipe implies the exec started while read on the pipe implies the exec failed and you can read the error number out of the pipe. https://gist.github.com/nicowilliams/a8a07b0fc75df05f684c23c18d7db234 https://news.ycombinator.com/item?id=30502392 https://github.com/famzah/popen-noshell

kobzol 2y ago
The stdlib already mostly does all of that :) Check out https://kobzol.github.io/rust/2024/01/28/process-spawning-performance-in-rust.html https://kobzol.github.io/rust/2024/01/28/process-spawning-pe....
fc417fc802 2y ago
Copying the page table isn't free but it isn't particularly expensive either. At least unless the parent is a real behemoth. Unless you enjoy footguns posix_spawn is probably a better idea than vfork. (Unless you actually need vfork of course.) The async pipe idea sounds interesting but I'm not clear how it would work. It seems like you'd have to use vfork to implement it but vfork is blocking until you call exec so doesn't that defeat the purpose?
- cryptonector 2y ago
  CoW is extremely expensive for threaded processes on multi-processor systems since you need TLB shootdowns. > The async pipe idea sounds interesting but I'm not clear how it would work. It seems like you'd have to use vfork to implement it but vfork is blocking until you call exec so doesn't that defeat the purpose? Yes, so have the `vfork()` happen in worker threads. > Unless you enjoy footguns posix_spawn is probably a better idea than vfork. (Unless you actually need vfork of course.) I have proof that `vfork()` can be used safely: the several `posix_spawn()` implementations that use it.
  - fc417fc802 2y ago
    Good point about multiprocessor systems. He did say this is being used on HPC clusters. You can also use C safely if you're careful. Doesn't mean it isn't full of footguns. > have the `vfork()` happen in worker threads Fair enough. Since this is an exercise in efficiency and latency, if you're creating a worker thread isn't an atomic write by the worker cheaper than creating a pipe?
    - cryptonector 2y ago
      > Good point about multiprocessor systems. He did say this is being used on HPC clusters. Plus essentially all systems now are multiprocessor systems. > You can also use C safely if you're careful. Doesn't mean it isn't full of footguns. You don't have to use much C here at all. > Since this is an exercise in efficiency and latency, if you're creating a worker thread isn't an atomic write by the worker cheaper than creating a pipe? The point is to signal that `execve(...)` did not fail but did start the new program -- but, how? You can use a condition variable or whatever to signal failure to exec, but you can't run code after a successful exec that would signal a condition variable or whatever because the program that was calling exec is now not running. A close-on-exec pipe serves to asynchronously signal success to the parent: it only closes either when the exec succeeds or after the exec fails and you close it after signaling the failure however you like. This works because the "code that runs after the exec succeeds" here is not code in the new program but code in the kernel after the kernel commits to not returning from `execve()`! What else has this property? Only file descriptors, or you can insist that the program you exec must signal some how that it started. But the latter is intrusive, while the former is not. If it's file descriptors then it has to be something that supports async I/O, and the simplest thing would be a pipe or a socketpair. Now if you're going to use a pipe for exec success reporting then you might as well also use it to signal failure because why have two mechanisms, one for reporting success and one for reporting failure? You still have to deal with the process' eventual exit, and reporting that asynchronously along with the exit status. But you get to report on all of three distinct events: - exec failure (e.g., ENOENT) - exec start (really *exec didn't fail*) - child process exit/death You could choose to not report exec start, I suppose. But you often want to know that exec didn't fail, and the only ways to know that are to either wait for the process to exit (which could be a long time!), assume that if enough time has passed then the exec did not fail (a lame heuristic), or just arrange to signal exec success. Since it is possible and easy to signal exec success via a close-on-exec pipe/socketpair then you might as well just do that.

kobzol 2y ago

The stdlib already mostly does all of that :) Check out https://kobzol.github.io/rust/2024/01/28/process-spawning-performance-in-rust.html https://kobzol.github.io/rust/2024/01/28/process-spawning-pe....

fc417fc802 2y ago

Copying the page table isn't free but it isn't particularly expensive either. At least unless the parent is a real behemoth. Unless you enjoy footguns posix_spawn is probably a better idea than vfork. (Unless you actually need vfork of course.) The async pipe idea sounds interesting but I'm not clear how it would work. It seems like you'd have to use vfork to implement it but vfork is blocking until you call exec so doesn't that defeat the purpose?

cryptonector 2y ago
CoW is extremely expensive for threaded processes on multi-processor systems since you need TLB shootdowns. > The async pipe idea sounds interesting but I'm not clear how it would work. It seems like you'd have to use vfork to implement it but vfork is blocking until you call exec so doesn't that defeat the purpose? Yes, so have the `vfork()` happen in worker threads. > Unless you enjoy footguns posix_spawn is probably a better idea than vfork. (Unless you actually need vfork of course.) I have proof that `vfork()` can be used safely: the several `posix_spawn()` implementations that use it.
- fc417fc802 2y ago
  Good point about multiprocessor systems. He did say this is being used on HPC clusters. You can also use C safely if you're careful. Doesn't mean it isn't full of footguns. > have the `vfork()` happen in worker threads Fair enough. Since this is an exercise in efficiency and latency, if you're creating a worker thread isn't an atomic write by the worker cheaper than creating a pipe?
  - cryptonector 2y ago
    > Good point about multiprocessor systems. He did say this is being used on HPC clusters. Plus essentially all systems now are multiprocessor systems. > You can also use C safely if you're careful. Doesn't mean it isn't full of footguns. You don't have to use much C here at all. > Since this is an exercise in efficiency and latency, if you're creating a worker thread isn't an atomic write by the worker cheaper than creating a pipe? The point is to signal that `execve(...)` did not fail but did start the new program -- but, how? You can use a condition variable or whatever to signal failure to exec, but you can't run code after a successful exec that would signal a condition variable or whatever because the program that was calling exec is now not running. A close-on-exec pipe serves to asynchronously signal success to the parent: it only closes either when the exec succeeds or after the exec fails and you close it after signaling the failure however you like. This works because the "code that runs after the exec succeeds" here is not code in the new program but code in the kernel after the kernel commits to not returning from `execve()`! What else has this property? Only file descriptors, or you can insist that the program you exec must signal some how that it started. But the latter is intrusive, while the former is not. If it's file descriptors then it has to be something that supports async I/O, and the simplest thing would be a pipe or a socketpair. Now if you're going to use a pipe for exec success reporting then you might as well also use it to signal failure because why have two mechanisms, one for reporting success and one for reporting failure? You still have to deal with the process' eventual exit, and reporting that asynchronously along with the exit status. But you get to report on all of three distinct events: - exec failure (e.g., ENOENT) - exec start (really *exec didn't fail*) - child process exit/death You could choose to not report exec start, I suppose. But you often want to know that exec didn't fail, and the only ways to know that are to either wait for the process to exit (which could be a long time!), assume that if enough time has passed then the exec did not fail (a lame heuristic), or just arrange to signal exec success. Since it is possible and easy to signal exec success via a close-on-exec pipe/socketpair then you might as well just do that.