6 ms·

It's important to specify which OS is being used because spawning goes a lot faster on Linux which has vfork(). If 40 spawns per second is the Linux speed, then

by jart 2y ago

It's important to specify which OS is being used because spawning goes a lot faster on Linux which has vfork(). If 40 spawns per second is the Linux speed, then I don't even want to know what that looks like on Darwin and OpenBSD. With Cosmopolitan Libc, 20 spawns per second is about the speed I get when simulating fork() on Windows. What makes fork() slow for large programs like Node is that fork() needs to lock and hold every single mutex that exists in a process while it happens. See the IEEE POSIX.1 notes surrounding pthread_atfork(). So yeah, that means everything is de facto blocked until the spawn completes, regardless of the thread that's doing it. Using the separate process to spawn is a smart idea. Especially if you can talk to it via a process shared condition variable.

convolvatron 2y ago
vfork() was introduced in 3.0 BSD
amelius 2y ago
> What makes fork() slow for large programs like Node is that fork() needs to lock and hold every single mutex that exists in a process while it happens Are there any circumstances where you'd want to fork a multi-threaded process?
- toast0 2y ago
  forking is useful if you need to call other code when you can't or don't want to link that code into your own process. Pretty useful for things like ffmpeg where you wouldn't want that in your process space (and you may want to put more than just a single fork/exec between your code and theirs in that case), but also handy for "simple" os integrations where it's easier to run an existing utility than find a library or interface more directly yourself. Downsides in forking from a large multi-threaded process make it pretty common to add something in early init that forks a process (or several) that remain small and can be commanded to fork at runtime. On modern systems, it's a very small cost if you don't actually fork, and the small added cost of intermediating fork/exec pays off the first time you don't have to wait for hundreds or thousands or millions of fds to be closed in the child process during the process. Of course, if your parent process remains small with few opened filehandles, maybe it is a measurable negative, but it's still likely to be very small.
  - mook 2y ago
    Are there circumstances where one might want to fork a multi-threaded process over spawning a child process via posix_spawn?
    - toast0 2y ago
      posix_spawn is not a syscall, last I checked, on Linux or FreeBSD. It's a library function of libcs, implemented in terms of fork, vfork, or clone and execve and has the same underlying performance issues in a large process as doing it yourself. It would be a different thing if it was a syscall, because then you could presumably use it in a large, multi threaded process without any of the hazards that arise from using fork, vfork, and clone. Are you asking if there's a good reason to have a fork without exec in a multithreaded program? I would think probably not, it would be pretty confusing to have all the memory and filedescriptors in the child process but only one thread --- you'd still have the other thread's stacks, but no thread running on them; that's asking for massive confusion and I think you'd need to do some cleanup for locks and whatever else. There might well be some applications where it makes senss to fork, and then spawn threads in the child processes, although I can't think of one off the top of my head. I've definitely had experience with use cases where it made more sense to fork rather than spawn threads and vice versa, but I haven't seen a reason to do both.
      - mook 2y ago
        Yeah, I meant from a libc user's point of view, using posix_spawn() directly seems safer (presumably they've managed to ensure they don't do anything dumb; I don't trust myself to be as good). A random copy of its man page actually says it's implemented with clone() with CLONE_VFORK plus exec().
        toast0 2y ago
        IMHO, using posix_spawn is OK if you know what libc your application is going to be using. If you don't, you may have inconsistent results, and you'll end up having to sample the various libcs, and probably take the best posix_spawn so your application has consistent behavior. Or you'll have to build in detailed specifications of which libcs and which versions of those libcs you'll work with.
- o11c 2y ago
  There are basically 2 unrelated uses of fork(): * In single-threaded programs (potentially without calling exec). This was widely used to get parallelism within a single program prior to 1995 or so, and is still a valid approach (except on OS X, which forbids single-threaded programs). In this case `pthread_atfork` is a reasonable thing to use, and the child process is not limited to async-signal-safe functions. * In (potentially-multi-threaded) programs calling exec. In this case, `pthread_atfork` is not meaningful (there's a recent `_Fork` that avoids it entirely), and the child is limited to async-signal-safe functions - basically, you should do all the setup beforehand, then after `fork` just call various syscalls. `posix_spawn` and `vfork` may replace this, but both are bug-prone.
  - saagarjha 2y ago
    > except on OS X, which forbids single-threaded programs …uh, what?
    - o11c 2y ago
      In recent years, Apple's libc will aggressively create threads to perform background tasks. There is no way to prevent this. Thus, calling fork() on OS X means you can only use async-signal-safe functions, unlike other platforms where abstinence is a choice.
      - plorkyeran 2y ago
        There’s a lot of system libraries on macOS that’ll spawn threads, but I’ve definitely never seen libc do so.
      - olliej 2y ago
        No, you can fork, what you can’t do is “fork and then try to keep running after the fork”. That’s never really been a supported model on Darwin, and in recent years with more threading occurring it fails faster than it did in the past. On Darwin there is very little you can safely do in the child after fork(), essentially the only thing guaranteed to work at that point is exec() and a few other basic operations. Again, this has always been the case for fork under Darwin, the only difference is that you are more likely to fail sooner now than 20 years ago. This is also why all documentation says to use posix_spawn: fork +exec is necessarily slower, and can’t really do anything that posix spawn doesn’t.
- derefr 2y ago
  How else would you expect a green-threaded runtime with multiple scheduler threads to spawn a managed subprocess (think: Erlang port-program) that has access to the runtime's file descriptors, unix domain sockets, etc?
maxmcd 2y ago
Ah, I will add it to the post, an accidental omission. This was all on `Linux 6.8.0-35-generic`.
maxmcd 2y ago
Do you have a sense for why things were so much faster in Go and Rust if they are both using a single process? I have also been experimenting with napi-rs which uses https://nodejs.org/api/n-api.html https://nodejs.org/api/n-api.html and allows you to run Rust in threads alongside the Node process. Performance with that approach has been very good, so I have wondered about the within-process limitations.
saurik 2y ago
If you want to spawn a new process on Darwin you really want to reach for posix_spawn, not any variant of fork. Using this API should also do whatever is optimal on Linux, which still doesn't seem to be merely calling vfork, as you instead can use clone--with CLONE_VFORK, sure--to limit the work to CLONE_VM (per the man page for glibc's posix_spawn). (Of course, this API would also be the perfect fit for use on Windows, but I don't think MinGW or Cygwin implement it, for reasons which are beyond me, given how horrible simulating fork is in that OS; but, I believe gnulib provides it, hopefully using CreateProcess on Win32, for all of these platforms than are missing out... and, they really are "missing out", as I feel most developers--including me--end up incorrectly dealing with the issues surrounding multi-threaded fork.)
- o11c 2y ago
  > If you want to spawn a new process on Darwin you really want to reach for posix_spawn Assuming you can avoid the bugs and version-sensitivities. There are a lot of them on various platforms, and my gut says they're more prominent on Darwin since alternatives are unrecommended. The dup2/cloexec mess is the one that sticks in my mind but it's far from the only one.
  - manwe150 2y ago
    nodejs (via libuv) transparently switches from fork to pthread_spawn on Darwin when the options passed support it. The dup2/cloexec is a mess, but there are some Darwin-specific enhancements to work around it. There is work done to make it also do the same on Linux, but seems it hasn’t been merged yet. The reason glibc uses clone is that it can specify an extra flag to avoid some undefined behavior inherent in the way vfork was originally designed, so it can get all of the equivalent speed with none of the pointy edges
nullindividual 2y ago
To be fair for Windows, you shouldn’t emulate fork() but rather spawn new threads, which Windows can do in less than a millisecond. This should give you much quicker results, but comes with other complexities, especially for UN*X-ported apps. Given you emulated fork(), you probably know this.
- Spivak 2y ago
  But those aren't really the same? I mean yes they are both ways to acquire additional units of execution but if you're reaching for subprocesses it's likely specifically because it's something you can't do in threads*. * Or can do but would be horrifying, like trying to run a compiled binary by mmaping it into your address space.
  - nullindividual 2y ago
    > But those aren't really the same? It's the "Windows way". Windows process creation is slow. No, they're not identical, they're different approaches to the problem. > subprocesses it's likely specifically because it's something you can't do in threads I suppose the question would be, "what can't you do, if you did it the Windows way?"
    - olliej 2y ago
      I would agree, if you’re wanting to run your server software on windows, your software should be designed to work well with windows. That goes for any OS though - if you’re running a high volume server then you shouldn’t be throwing away perf by designing your software in a way that works badly on the host OS. That said multiple threads are not multiple processes: Using multiple threads means you get a single set of globals, and if any of them are contended you get a pile of sadness. Similarly, tearing down threads is not necessarily sufficient to release all resources a library may have consumed because again: globals. Related to globals: I’m not sure how heavily TLS is restricted on windows, but I recall various arbitrary limits being annoying on windows back when I did dev work, and I recall some annoying stuff with TLS — but that was back in the days of needing to support XP, so maybe it’s less of an issue these days? You also get security advantages by pushing things into multiple processes that you don’t get from multiple threads, though I don’t really think that that’s an issue in this particular context.
o11c 2y ago
> What makes fork() slow for large programs like Node is that fork() needs to lock and hold every single mutex that exists in a process while it happens It really doesn't; if it does that's a bug to fix. It only requires that the child process does not call any function that is not async-signal-safe if threads exist, and end with a call to exec or _exit. In C, this is usually pretty easy to arrange; at most the difficulty is if you want to support arbitrary file-descriptor mappings. However, it is impossible from languages like Javascript (or Python, etc.). pthread_atfork() is really only useful to call if you know you're using multiple processes and not using threads at all. _Fork() is supposed to be a thing now (to avoid any historical atfork handlers) but can't be relied on to exist. posix_spawn() has the unfortunate problem of being buggy and version-dependent when various options are used. It's probably fine if you don't need the options (for example, if you exec a shim first), or if you know how recent a platform you can use. vfork() on Linux is mostly an optimization (for scheduling or for when overcommit is disabled), but compiler optimizations can make it buggy (though given that /bin/sh etc. use it, compiler probably won't break it too much if you call it in the same way). For best results, it's advisable to either call clone (with a separate stack directly), or use architecture-specific assembly to call `vfork` in a way the compiler can't mess up. vfork() on many other platforms is basically just an alias for fork() (or maybe _Fork() ?), but I haven't investigated the details much. fork() is fine if your parent process isn't big enough that you're worried about memory accounting though, and you haven't added braindead atfork handlers.
- jart 2y ago
  You absolutely need to lock all the locks, because fork() kills all the other threads, so if any one of them was in a critical section while fork was called, then the new process will deadlock if it tries to acquire that lock afterwards. Yes, I know the POSIX standard has turned against fork() and considers vfork() beyond the pale. I don't know how they can forbid UNIX behavior that's essential to the workings of /bin/sh. No it's not realistic for the OS to define its most important function to be as useless as a signal handler, but here we are. That advice will be never be accepted. People on the best OSes like Linux and FreeBSD will always use vfork() regardless of what POSIX says. The de facto standard is that only vfork() must guarantee a swift execve() call. You learn to take the official standard with a grain of salt when you realize it's bankrolled by regulated enterprises who hire cheap engineers to work on life or death machinery and therefore can't be trusted with powerful footguns.
zbentley 2y ago
> Using the separate process to spawn is a smart idea. Especially if you can talk to it via a process shared condition variable. You can even use prctl(2) to set the parent as a subreaper for the spawner’s children so the process hierarchy continues to behave as expected and you don’t end up with orphans. Or just run such that the parent is PID 1 in a container anyway. With either approach, watch out if you’re using a thicker runtime/language whose child process accounting can get freaked out if an unexpectedly-present child indicates ready-for-reap. Using pidfds on Linux (optionally combined with fdpass via UNIX sockets to signal that the spawner did its thing) can help with this.
amluto 2y ago
> With Cosmopolitan Libc, 20 spawns per second is about the speed I get when simulating fork() on Windows. Egads, this reminds me of when I used to muck with Cygwin. fork() is quite a complex bit of kernel magic, and that’s on kernels designed for it. Simulating fork() on Windows well enough to be usable has always struck me as quite impressive. But it’s also ridiculous. Forking to spawn a process is an almost comically poor design, and this is one of maybe two things IMO that Windows got right! CreateProcess() may be ugly, its arguments may involve nasty data structures, the command line string may be a disaster that continues to cause problems to this day, but it’s a far superior design for spawning processes. I hope that no one who cares even slightly about process spawning performance on Windows tries to do it using fork. Not that Windows has ever been particularly fast at spawning processes…

by jart 2y ago

convolvatron 2y ago

vfork() was introduced in 3.0 BSD

amelius 2y ago

> What makes fork() slow for large programs like Node is that fork() needs to lock and hold every single mutex that exists in a process while it happens Are there any circumstances where you'd want to fork a multi-threaded process?

toast0 2y ago
forking is useful if you need to call other code when you can't or don't want to link that code into your own process. Pretty useful for things like ffmpeg where you wouldn't want that in your process space (and you may want to put more than just a single fork/exec between your code and theirs in that case), but also handy for "simple" os integrations where it's easier to run an existing utility than find a library or interface more directly yourself. Downsides in forking from a large multi-threaded process make it pretty common to add something in early init that forks a process (or several) that remain small and can be commanded to fork at runtime. On modern systems, it's a very small cost if you don't actually fork, and the small added cost of intermediating fork/exec pays off the first time you don't have to wait for hundreds or thousands or millions of fds to be closed in the child process during the process. Of course, if your parent process remains small with few opened filehandles, maybe it is a measurable negative, but it's still likely to be very small.
- mook 2y ago
  Are there circumstances where one might want to fork a multi-threaded process over spawning a child process via posix_spawn?
  - toast0 2y ago
    posix_spawn is not a syscall, last I checked, on Linux or FreeBSD. It's a library function of libcs, implemented in terms of fork, vfork, or clone and execve and has the same underlying performance issues in a large process as doing it yourself. It would be a different thing if it was a syscall, because then you could presumably use it in a large, multi threaded process without any of the hazards that arise from using fork, vfork, and clone. Are you asking if there's a good reason to have a fork without exec in a multithreaded program? I would think probably not, it would be pretty confusing to have all the memory and filedescriptors in the child process but only one thread --- you'd still have the other thread's stacks, but no thread running on them; that's asking for massive confusion and I think you'd need to do some cleanup for locks and whatever else. There might well be some applications where it makes senss to fork, and then spawn threads in the child processes, although I can't think of one off the top of my head. I've definitely had experience with use cases where it made more sense to fork rather than spawn threads and vice versa, but I haven't seen a reason to do both.
    - mook 2y ago
      Yeah, I meant from a libc user's point of view, using posix_spawn() directly seems safer (presumably they've managed to ensure they don't do anything dumb; I don't trust myself to be as good). A random copy of its man page actually says it's implemented with clone() with CLONE_VFORK plus exec().
      - toast0 2y ago
        IMHO, using posix_spawn is OK if you know what libc your application is going to be using. If you don't, you may have inconsistent results, and you'll end up having to sample the various libcs, and probably take the best posix_spawn so your application has consistent behavior. Or you'll have to build in detailed specifications of which libcs and which versions of those libcs you'll work with.
o11c 2y ago
There are basically 2 unrelated uses of fork(): * In single-threaded programs (potentially without calling exec). This was widely used to get parallelism within a single program prior to 1995 or so, and is still a valid approach (except on OS X, which forbids single-threaded programs). In this case `pthread_atfork` is a reasonable thing to use, and the child process is not limited to async-signal-safe functions. * In (potentially-multi-threaded) programs calling exec. In this case, `pthread_atfork` is not meaningful (there's a recent `_Fork` that avoids it entirely), and the child is limited to async-signal-safe functions - basically, you should do all the setup beforehand, then after `fork` just call various syscalls. `posix_spawn` and `vfork` may replace this, but both are bug-prone.
- saagarjha 2y ago
  > except on OS X, which forbids single-threaded programs …uh, what?
  - o11c 2y ago
    In recent years, Apple's libc will aggressively create threads to perform background tasks. There is no way to prevent this. Thus, calling fork() on OS X means you can only use async-signal-safe functions, unlike other platforms where abstinence is a choice.
    - plorkyeran 2y ago
      There’s a lot of system libraries on macOS that’ll spawn threads, but I’ve definitely never seen libc do so.
    - olliej 2y ago
      No, you can fork, what you can’t do is “fork and then try to keep running after the fork”. That’s never really been a supported model on Darwin, and in recent years with more threading occurring it fails faster than it did in the past. On Darwin there is very little you can safely do in the child after fork(), essentially the only thing guaranteed to work at that point is exec() and a few other basic operations. Again, this has always been the case for fork under Darwin, the only difference is that you are more likely to fail sooner now than 20 years ago. This is also why all documentation says to use posix_spawn: fork +exec is necessarily slower, and can’t really do anything that posix spawn doesn’t.
derefr 2y ago
How else would you expect a green-threaded runtime with multiple scheduler threads to spawn a managed subprocess (think: Erlang port-program) that has access to the runtime's file descriptors, unix domain sockets, etc?

maxmcd 2y ago

Ah, I will add it to the post, an accidental omission. This was all on `Linux 6.8.0-35-generic`.

maxmcd 2y ago

Do you have a sense for why things were so much faster in Go and Rust if they are both using a single process? I have also been experimenting with napi-rs which uses https://nodejs.org/api/n-api.html https://nodejs.org/api/n-api.html and allows you to run Rust in threads alongside the Node process. Performance with that approach has been very good, so I have wondered about the within-process limitations.

saurik 2y ago

If you want to spawn a new process on Darwin you really want to reach for posix_spawn, not any variant of fork. Using this API should also do whatever is optimal on Linux, which still doesn't seem to be merely calling vfork, as you instead can use clone--with CLONE_VFORK, sure--to limit the work to CLONE_VM (per the man page for glibc's posix_spawn). (Of course, this API would also be the perfect fit for use on Windows, but I don't think MinGW or Cygwin implement it, for reasons which are beyond me, given how horrible simulating fork is in that OS; but, I believe gnulib provides it, hopefully using CreateProcess on Win32, for all of these platforms than are missing out... and, they really are "missing out", as I feel most developers--including me--end up incorrectly dealing with the issues surrounding multi-threaded fork.)

o11c 2y ago
> If you want to spawn a new process on Darwin you really want to reach for posix_spawn Assuming you can avoid the bugs and version-sensitivities. There are a lot of them on various platforms, and my gut says they're more prominent on Darwin since alternatives are unrecommended. The dup2/cloexec mess is the one that sticks in my mind but it's far from the only one.
- manwe150 2y ago
  nodejs (via libuv) transparently switches from fork to pthread_spawn on Darwin when the options passed support it. The dup2/cloexec is a mess, but there are some Darwin-specific enhancements to work around it. There is work done to make it also do the same on Linux, but seems it hasn’t been merged yet. The reason glibc uses clone is that it can specify an extra flag to avoid some undefined behavior inherent in the way vfork was originally designed, so it can get all of the equivalent speed with none of the pointy edges

nullindividual 2y ago

To be fair for Windows, you shouldn’t emulate fork() but rather spawn new threads, which Windows can do in less than a millisecond. This should give you much quicker results, but comes with other complexities, especially for UN*X-ported apps. Given you emulated fork(), you probably know this.

Spivak 2y ago
But those aren't really the same? I mean yes they are both ways to acquire additional units of execution but if you're reaching for subprocesses it's likely specifically because it's something you can't do in threads*. * Or can do but would be horrifying, like trying to run a compiled binary by mmaping it into your address space.
- nullindividual 2y ago
  > But those aren't really the same? It's the "Windows way". Windows process creation is slow. No, they're not identical, they're different approaches to the problem. > subprocesses it's likely specifically because it's something you can't do in threads I suppose the question would be, "what can't you do, if you did it the Windows way?"
  - olliej 2y ago
    I would agree, if you’re wanting to run your server software on windows, your software should be designed to work well with windows. That goes for any OS though - if you’re running a high volume server then you shouldn’t be throwing away perf by designing your software in a way that works badly on the host OS. That said multiple threads are not multiple processes: Using multiple threads means you get a single set of globals, and if any of them are contended you get a pile of sadness. Similarly, tearing down threads is not necessarily sufficient to release all resources a library may have consumed because again: globals. Related to globals: I’m not sure how heavily TLS is restricted on windows, but I recall various arbitrary limits being annoying on windows back when I did dev work, and I recall some annoying stuff with TLS — but that was back in the days of needing to support XP, so maybe it’s less of an issue these days? You also get security advantages by pushing things into multiple processes that you don’t get from multiple threads, though I don’t really think that that’s an issue in this particular context.

o11c 2y ago

> What makes fork() slow for large programs like Node is that fork() needs to lock and hold every single mutex that exists in a process while it happens It really doesn't; if it does that's a bug to fix. It only requires that the child process does not call any function that is not async-signal-safe if threads exist, and end with a call to exec or _exit. In C, this is usually pretty easy to arrange; at most the difficulty is if you want to support arbitrary file-descriptor mappings. However, it is impossible from languages like Javascript (or Python, etc.). pthread_atfork() is really only useful to call if you know you're using multiple processes and not using threads at all. _Fork() is supposed to be a thing now (to avoid any historical atfork handlers) but can't be relied on to exist. posix_spawn() has the unfortunate problem of being buggy and version-dependent when various options are used. It's probably fine if you don't need the options (for example, if you exec a shim first), or if you know how recent a platform you can use. vfork() on Linux is mostly an optimization (for scheduling or for when overcommit is disabled), but compiler optimizations can make it buggy (though given that /bin/sh etc. use it, compiler probably won't break it too much if you call it in the same way). For best results, it's advisable to either call clone (with a separate stack directly), or use architecture-specific assembly to call `vfork` in a way the compiler can't mess up. vfork() on many other platforms is basically just an alias for fork() (or maybe _Fork() ?), but I haven't investigated the details much. fork() is fine if your parent process isn't big enough that you're worried about memory accounting though, and you haven't added braindead atfork handlers.

jart 2y ago
You absolutely need to lock all the locks, because fork() kills all the other threads, so if any one of them was in a critical section while fork was called, then the new process will deadlock if it tries to acquire that lock afterwards. Yes, I know the POSIX standard has turned against fork() and considers vfork() beyond the pale. I don't know how they can forbid UNIX behavior that's essential to the workings of /bin/sh. No it's not realistic for the OS to define its most important function to be as useless as a signal handler, but here we are. That advice will be never be accepted. People on the best OSes like Linux and FreeBSD will always use vfork() regardless of what POSIX says. The de facto standard is that only vfork() must guarantee a swift execve() call. You learn to take the official standard with a grain of salt when you realize it's bankrolled by regulated enterprises who hire cheap engineers to work on life or death machinery and therefore can't be trusted with powerful footguns.

zbentley 2y ago

> Using the separate process to spawn is a smart idea. Especially if you can talk to it via a process shared condition variable. You can even use prctl(2) to set the parent as a subreaper for the spawner’s children so the process hierarchy continues to behave as expected and you don’t end up with orphans. Or just run such that the parent is PID 1 in a container anyway. With either approach, watch out if you’re using a thicker runtime/language whose child process accounting can get freaked out if an unexpectedly-present child indicates ready-for-reap. Using pidfds on Linux (optionally combined with fdpass via UNIX sockets to signal that the spawner did its thing) can help with this.

amluto 2y ago

> With Cosmopolitan Libc, 20 spawns per second is about the speed I get when simulating fork() on Windows. Egads, this reminds me of when I used to muck with Cygwin. fork() is quite a complex bit of kernel magic, and that’s on kernels designed for it. Simulating fork() on Windows well enough to be usable has always struck me as quite impressive. But it’s also ridiculous. Forking to spawn a process is an almost comically poor design, and this is one of maybe two things IMO that Windows got right! CreateProcess() may be ugly, its arguments may involve nasty data structures, the command line string may be a disaster that continues to cause problems to this day, but it’s a far superior design for spawning processes. I hope that no one who cares even slightly about process spawning performance on Windows tries to do it using fork. Not that Windows has ever been particularly fast at spawning processes…