4 ms·

CoW is extremely expensive for threaded processes on multi-processor systems since you need TLB shootdowns. > The async pipe idea sounds interesting but I'm no

CoW is extremely expensive for threaded processes on multi-processor systems since you need TLB shootdowns. > The async pipe idea sounds interesting but I'm not clear how it would work. It seems like you'd have to use vfork to implement it but vfork is blocking until you call exec so doesn't that defeat the purpose? Yes, so have the `vfork()` happen in worker threads. > Unless you enjoy footguns posix_spawn is probably a better idea than vfork. (Unless you actually need vfork of course.) I have proof that `vfork()` can be used safely: the several `posix_spawn()` implementations that use it.

fc417fc802 2y ago
Good point about multiprocessor systems. He did say this is being used on HPC clusters. You can also use C safely if you're careful. Doesn't mean it isn't full of footguns. > have the `vfork()` happen in worker threads Fair enough. Since this is an exercise in efficiency and latency, if you're creating a worker thread isn't an atomic write by the worker cheaper than creating a pipe?
- cryptonector 2y ago
  > Good point about multiprocessor systems. He did say this is being used on HPC clusters. Plus essentially all systems now are multiprocessor systems. > You can also use C safely if you're careful. Doesn't mean it isn't full of footguns. You don't have to use much C here at all. > Since this is an exercise in efficiency and latency, if you're creating a worker thread isn't an atomic write by the worker cheaper than creating a pipe? The point is to signal that `execve(...)` did not fail but did start the new program -- but, how? You can use a condition variable or whatever to signal failure to exec, but you can't run code after a successful exec that would signal a condition variable or whatever because the program that was calling exec is now not running. A close-on-exec pipe serves to asynchronously signal success to the parent: it only closes either when the exec succeeds or after the exec fails and you close it after signaling the failure however you like. This works because the "code that runs after the exec succeeds" here is not code in the new program but code in the kernel after the kernel commits to not returning from `execve()`! What else has this property? Only file descriptors, or you can insist that the program you exec must signal some how that it started. But the latter is intrusive, while the former is not. If it's file descriptors then it has to be something that supports async I/O, and the simplest thing would be a pipe or a socketpair. Now if you're going to use a pipe for exec success reporting then you might as well also use it to signal failure because why have two mechanisms, one for reporting success and one for reporting failure? You still have to deal with the process' eventual exit, and reporting that asynchronously along with the exit status. But you get to report on all of three distinct events: - exec failure (e.g., ENOENT) - exec start (really *exec didn't fail*) - child process exit/death You could choose to not report exec start, I suppose. But you often want to know that exec didn't fail, and the only ways to know that are to either wait for the process to exit (which could be a long time!), assume that if enough time has passed then the exec did not fail (a lame heuristic), or just arrange to signal exec success. Since it is possible and easy to signal exec success via a close-on-exec pipe/socketpair then you might as well just do that.
  - fc417fc802 2y ago
    Note my nomenclature here is parent (doesn't block), worker (calls vfork), and child (calls execve). If you used vfork then the worker doesn't unblock until execve has executed successfully (AFAIU). In that case if the exec succeeded then nothing will have been written whereas if it failed then the child will have written out the error code to the worker's stack before calling _exit. The worker can then use an atomic write to communicate the ultimate result back to the parent in an asynchronous manner.
    - cryptonector 2y ago
      Ah, yes, that's true. I'm so used to using pipes like this that I forgot. You're quite right. Though I'd still use something like a pipe rather than a condition variable, say, because on Unix and Linux systems you can't have an event loop handle events on file descriptors _and_ condition variables.
      - cryptonector 2y ago
        And those are my favorite signalling mechanisms: async I/O and condition variables.
      - fc417fc802 2y ago
        I thought the optimization goal was latency on the parent thread so I had in mind a bare atomic_store_explicit with memory_order_relaxed. However you wouldn't care about latency in the first place if you weren't scaling this up and obviously scaling up requires additional infrastructure that I'm ignoring here. That said the linked article is using spawn_blocking from tokio meaning you can just call vfork directly and return the result. No need to roll your own in this case. > you can't have an event loop handle events on file descriptors _and_ condition variables Since events crossing the kernel boundary aren't fast to begin with I prefer to use a secondary event loop for the sluggish stuff and feed the output to the lower latency one.

fc417fc802 2y ago

Good point about multiprocessor systems. He did say this is being used on HPC clusters. You can also use C safely if you're careful. Doesn't mean it isn't full of footguns. > have the `vfork()` happen in worker threads Fair enough. Since this is an exercise in efficiency and latency, if you're creating a worker thread isn't an atomic write by the worker cheaper than creating a pipe?

cryptonector 2y ago
> Good point about multiprocessor systems. He did say this is being used on HPC clusters. Plus essentially all systems now are multiprocessor systems. > You can also use C safely if you're careful. Doesn't mean it isn't full of footguns. You don't have to use much C here at all. > Since this is an exercise in efficiency and latency, if you're creating a worker thread isn't an atomic write by the worker cheaper than creating a pipe? The point is to signal that `execve(...)` did not fail but did start the new program -- but, how? You can use a condition variable or whatever to signal failure to exec, but you can't run code after a successful exec that would signal a condition variable or whatever because the program that was calling exec is now not running. A close-on-exec pipe serves to asynchronously signal success to the parent: it only closes either when the exec succeeds or after the exec fails and you close it after signaling the failure however you like. This works because the "code that runs after the exec succeeds" here is not code in the new program but code in the kernel after the kernel commits to not returning from `execve()`! What else has this property? Only file descriptors, or you can insist that the program you exec must signal some how that it started. But the latter is intrusive, while the former is not. If it's file descriptors then it has to be something that supports async I/O, and the simplest thing would be a pipe or a socketpair. Now if you're going to use a pipe for exec success reporting then you might as well also use it to signal failure because why have two mechanisms, one for reporting success and one for reporting failure? You still have to deal with the process' eventual exit, and reporting that asynchronously along with the exit status. But you get to report on all of three distinct events: - exec failure (e.g., ENOENT) - exec start (really *exec didn't fail*) - child process exit/death You could choose to not report exec start, I suppose. But you often want to know that exec didn't fail, and the only ways to know that are to either wait for the process to exit (which could be a long time!), assume that if enough time has passed then the exec did not fail (a lame heuristic), or just arrange to signal exec success. Since it is possible and easy to signal exec success via a close-on-exec pipe/socketpair then you might as well just do that.
- fc417fc802 2y ago
  Note my nomenclature here is parent (doesn't block), worker (calls vfork), and child (calls execve). If you used vfork then the worker doesn't unblock until execve has executed successfully (AFAIU). In that case if the exec succeeded then nothing will have been written whereas if it failed then the child will have written out the error code to the worker's stack before calling _exit. The worker can then use an atomic write to communicate the ultimate result back to the parent in an asynchronous manner.
  - cryptonector 2y ago
    Ah, yes, that's true. I'm so used to using pipes like this that I forgot. You're quite right. Though I'd still use something like a pipe rather than a condition variable, say, because on Unix and Linux systems you can't have an event loop handle events on file descriptors _and_ condition variables.
    - cryptonector 2y ago
      And those are my favorite signalling mechanisms: async I/O and condition variables.
    - fc417fc802 2y ago
      I thought the optimization goal was latency on the parent thread so I had in mind a bare atomic_store_explicit with memory_order_relaxed. However you wouldn't care about latency in the first place if you weren't scaling this up and obviously scaling up requires additional infrastructure that I'm ignoring here. That said the linked article is using spawn_blocking from tokio meaning you can just call vfork directly and return the result. No need to roll your own in this case. > you can't have an event loop handle events on file descriptors _and_ condition variables Since events crossing the kernel boundary aren't fast to begin with I prefer to use a secondary event loop for the sluggish stuff and feed the output to the lower latency one.