8 ms·

Why is spawning a new process in Node so slow?

95 points by maxmcd 2y ago

efilife 2y ago
Nice article, definitely wasn't expecting some of its results. Was good to see Node finally beat Deno near the end of the article. As a heads up, the author confuses its and it's. Makes the article look unprofessional. https://youryoure.com/?its https://youryoure.com/?its here's how to differentiate
- maxmcd 2y ago
  Thanks for noting it. It do indeed mix those two up all the time. Pushed a fix!
  - codr7 2y ago
    Don't we all :) It's easy to forget what a big deal being semi fluent in multiple languages is.
- seniorsassycat 2y ago
  [flagged]
jart 2y ago
It's important to specify which OS is being used because spawning goes a lot faster on Linux which has vfork(). If 40 spawns per second is the Linux speed, then I don't even want to know what that looks like on Darwin and OpenBSD. With Cosmopolitan Libc, 20 spawns per second is about the speed I get when simulating fork() on Windows. What makes fork() slow for large programs like Node is that fork() needs to lock and hold every single mutex that exists in a process while it happens. See the IEEE POSIX.1 notes surrounding pthread_atfork(). So yeah, that means everything is de facto blocked until the spawn completes, regardless of the thread that's doing it. Using the separate process to spawn is a smart idea. Especially if you can talk to it via a process shared condition variable.
- convolvatron 2y ago
  vfork() was introduced in 3.0 BSD
- amelius 2y ago
  > What makes fork() slow for large programs like Node is that fork() needs to lock and hold every single mutex that exists in a process while it happens Are there any circumstances where you'd want to fork a multi-threaded process?
  - toast0 2y ago
    forking is useful if you need to call other code when you can't or don't want to link that code into your own process. Pretty useful for things like ffmpeg where you wouldn't want that in your process space (and you may want to put more than just a single fork/exec between your code and theirs in that case), but also handy for "simple" os integrations where it's easier to run an existing utility than find a library or interface more directly yourself. Downsides in forking from a large multi-threaded process make it pretty common to add something in early init that forks a process (or several) that remain small and can be commanded to fork at runtime. On modern systems, it's a very small cost if you don't actually fork, and the small added cost of intermediating fork/exec pays off the first time you don't have to wait for hundreds or thousands or millions of fds to be closed in the child process during the process. Of course, if your parent process remains small with few opened filehandles, maybe it is a measurable negative, but it's still likely to be very small.
    - mook 2y ago
      Are there circumstances where one might want to fork a multi-threaded process over spawning a child process via posix_spawn?
      - toast0 2y ago
        posix_spawn is not a syscall, last I checked, on Linux or FreeBSD. It's a library function of libcs, implemented in terms of fork, vfork, or clone and execve and has the same underlying performance issues in a large process as doing it yourself. It would be a different thing if it was a syscall, because then you could presumably use it in a large, multi threaded process without any of the hazards that arise from using fork, vfork, and clone. Are you asking if there's a good reason to have a fork without exec in a multithreaded program? I would think probably not, it would be pretty confusing to have all the memory and filedescriptors in the child process but only one thread --- you'd still have the other thread's stacks, but no thread running on them; that's asking for massive confusion and I think you'd need to do some cleanup for locks and whatever else. There might well be some applications where it makes senss to fork, and then spawn threads in the child processes, although I can't think of one off the top of my head. I've definitely had experience with use cases where it made more sense to fork rather than spawn threads and vice versa, but I haven't seen a reason to do both.
thomasfromcdnjs 2y ago
Thanks for the super interesting read. Did you happen to look at how the load on the 8 cores looked at any given time?
- maxmcd 2y ago
  Both Go and Rust had CPU near 100% on all cores. Most of the Deno and Bun workloads were at around 80%, some of the slower Node benchmarks were around 40-60% and that increased over time, I think most of the fastest benchmarks were seeing very good CPU utilization across all runs. Sorry, this is all from memory of just having `btop` open for some of the benchmarks. Maybe in a future post I will dig into CPU utlization more.
Denvercoder9 2y ago
I'm curious if anyone has any insights into the answer to the titular question. The article, while certainly interesting, mostly discusses workarounds, but doesn't really dive into a root cause analysis.
- maxmcd 2y ago
  Yes, apologies for the less than satisfying conclusion. I am planning a part 2 of this post where I hope to actually answer the question. In the meantime the discussion on lobste.rs includes some lower-level speculation: https://lobste.rs/s/tr8ozm/why_is_spawning_new_process_node_so_slow https://lobste.rs/s/tr8ozm/why_is_spawning_new_process_node_...
yu3zhou4 2y ago
A really nice read, thanks for posting. I'm curious about why it's not as fast as Go and is it possible to speed up to Go's level?
throwitaway1123 2y ago
One addendum I would add to the section on node:cluster is that Deno hasn't actually implemented the cluster module yet (it's just a stub), so using it with Deno is pure overhead [1]. Also, there's ongoing work to add node:cluster to Bun [2]. [1] https://docs.deno.com/runtime/manual/node/compatibility/ https://docs.deno.com/runtime/manual/node/compatibility/ [2] https://github.com/oven-sh/bun/pull/11492 https://github.com/oven-sh/bun/pull/11492
ambicapter 2y ago
> Turns out, that if you use a Unix socket and the filename starts with a null byte, like \0foo the socket will not exist on the filesystem and it’ll be automatically removed when no longer used. Weird! Cool! Is that…intentional on Unix’s part? Seems kind of a weird thing to implement.
- spc476 2y ago
  I believe that's a Linux thing.
- o11c 2y ago
  It's very useful since "how do I atomically delete a stale socket file" is not actually an easy thing to do.
- deleted 2y ago
  [deleted]
- somat 2y ago
  files on unix are referenced counted. and unix sockets are just a type of file. every link adds to the count, a link can be a filesystem name but open file handles are also links. once there are no more links to a file the file can be deleted. so this is opening a unix socket with a no name(the initial null terminates the C style string) thus leaving the only link to it as the file discripter. One trick I used to do was to sneak another link to the file to keep it around before the process ended , The serious real use case for this was if you had accidentally deleted a file but some process was keeping it open somewhere, but I mainly used it back when youtube was flash based it would try and keep the video cache file hidden this way. and you could make a copy of the video by snaging the correct file descriptor.
  - jumploops 2y ago
    I have fond memories of modifying the memory of flash-based games when they first launched on Facebook. I don't remember the FB account credentials, but I received thousands of messages from people either telling me I was a terrible person, or trying to bribe me so they could beat their sibling... all for a simple helicopter game. Fun times.
cryptonector 2y ago
The problem is `fork()`. Use `vfork()` or `posix_spawn()`.
tamimio 2y ago
Nice article! I remember three years ago, I needed to spawn a few processes in Node, and I ended up using child process spawn to spawn an external binary. It was fast and instantaneous (no benchmarks). It was on a Linux OS, ARM-based.
bsaunder 2y ago
Have you looked into profiling with strace? You should be able to see the actual system calls each of your tests are making. This will probably give you some good insights into what accounts for the speed differences.

95 points by maxmcd 2y ago

Nice article, definitely wasn't expecting some of its results. Was good to see Node finally beat Deno near the end of the article. As a heads up, the author confuses its and it's. Makes the article look unprofessional. https://youryoure.com/?its https://youryoure.com/?its here's how to differentiate

maxmcd 2y ago
Thanks for noting it. It do indeed mix those two up all the time. Pushed a fix!
- codr7 2y ago
  Don't we all :) It's easy to forget what a big deal being semi fluent in multiple languages is.
seniorsassycat 2y ago
[flagged]

jart 2y ago

It's important to specify which OS is being used because spawning goes a lot faster on Linux which has vfork(). If 40 spawns per second is the Linux speed, then I don't even want to know what that looks like on Darwin and OpenBSD. With Cosmopolitan Libc, 20 spawns per second is about the speed I get when simulating fork() on Windows. What makes fork() slow for large programs like Node is that fork() needs to lock and hold every single mutex that exists in a process while it happens. See the IEEE POSIX.1 notes surrounding pthread_atfork(). So yeah, that means everything is de facto blocked until the spawn completes, regardless of the thread that's doing it. Using the separate process to spawn is a smart idea. Especially if you can talk to it via a process shared condition variable.

convolvatron 2y ago
vfork() was introduced in 3.0 BSD
amelius 2y ago
> What makes fork() slow for large programs like Node is that fork() needs to lock and hold every single mutex that exists in a process while it happens Are there any circumstances where you'd want to fork a multi-threaded process?
- toast0 2y ago
  forking is useful if you need to call other code when you can't or don't want to link that code into your own process. Pretty useful for things like ffmpeg where you wouldn't want that in your process space (and you may want to put more than just a single fork/exec between your code and theirs in that case), but also handy for "simple" os integrations where it's easier to run an existing utility than find a library or interface more directly yourself. Downsides in forking from a large multi-threaded process make it pretty common to add something in early init that forks a process (or several) that remain small and can be commanded to fork at runtime. On modern systems, it's a very small cost if you don't actually fork, and the small added cost of intermediating fork/exec pays off the first time you don't have to wait for hundreds or thousands or millions of fds to be closed in the child process during the process. Of course, if your parent process remains small with few opened filehandles, maybe it is a measurable negative, but it's still likely to be very small.
  - mook 2y ago
    Are there circumstances where one might want to fork a multi-threaded process over spawning a child process via posix_spawn?
    - toast0 2y ago
      posix_spawn is not a syscall, last I checked, on Linux or FreeBSD. It's a library function of libcs, implemented in terms of fork, vfork, or clone and execve and has the same underlying performance issues in a large process as doing it yourself. It would be a different thing if it was a syscall, because then you could presumably use it in a large, multi threaded process without any of the hazards that arise from using fork, vfork, and clone. Are you asking if there's a good reason to have a fork without exec in a multithreaded program? I would think probably not, it would be pretty confusing to have all the memory and filedescriptors in the child process but only one thread --- you'd still have the other thread's stacks, but no thread running on them; that's asking for massive confusion and I think you'd need to do some cleanup for locks and whatever else. There might well be some applications where it makes senss to fork, and then spawn threads in the child processes, although I can't think of one off the top of my head. I've definitely had experience with use cases where it made more sense to fork rather than spawn threads and vice versa, but I haven't seen a reason to do both.

thomasfromcdnjs 2y ago

Thanks for the super interesting read. Did you happen to look at how the load on the 8 cores looked at any given time?

maxmcd 2y ago
Both Go and Rust had CPU near 100% on all cores. Most of the Deno and Bun workloads were at around 80%, some of the slower Node benchmarks were around 40-60% and that increased over time, I think most of the fastest benchmarks were seeing very good CPU utilization across all runs. Sorry, this is all from memory of just having `btop` open for some of the benchmarks. Maybe in a future post I will dig into CPU utlization more.

Denvercoder9 2y ago

I'm curious if anyone has any insights into the answer to the titular question. The article, while certainly interesting, mostly discusses workarounds, but doesn't really dive into a root cause analysis.

maxmcd 2y ago
Yes, apologies for the less than satisfying conclusion. I am planning a part 2 of this post where I hope to actually answer the question. In the meantime the discussion on lobste.rs includes some lower-level speculation: https://lobste.rs/s/tr8ozm/why_is_spawning_new_process_node_so_slow https://lobste.rs/s/tr8ozm/why_is_spawning_new_process_node_...

yu3zhou4 2y ago

A really nice read, thanks for posting. I'm curious about why it's not as fast as Go and is it possible to speed up to Go's level?

throwitaway1123 2y ago

One addendum I would add to the section on node:cluster is that Deno hasn't actually implemented the cluster module yet (it's just a stub), so using it with Deno is pure overhead [1]. Also, there's ongoing work to add node:cluster to Bun [2]. [1] https://docs.deno.com/runtime/manual/node/compatibility/ https://docs.deno.com/runtime/manual/node/compatibility/ [2] https://github.com/oven-sh/bun/pull/11492 https://github.com/oven-sh/bun/pull/11492

ambicapter 2y ago

> Turns out, that if you use a Unix socket and the filename starts with a null byte, like \0foo the socket will not exist on the filesystem and it’ll be automatically removed when no longer used. Weird! Cool! Is that…intentional on Unix’s part? Seems kind of a weird thing to implement.

spc476 2y ago
I believe that's a Linux thing.
o11c 2y ago
It's very useful since "how do I atomically delete a stale socket file" is not actually an easy thing to do.
deleted 2y ago
[deleted]
somat 2y ago
files on unix are referenced counted. and unix sockets are just a type of file. every link adds to the count, a link can be a filesystem name but open file handles are also links. once there are no more links to a file the file can be deleted. so this is opening a unix socket with a no name(the initial null terminates the C style string) thus leaving the only link to it as the file discripter. One trick I used to do was to sneak another link to the file to keep it around before the process ended , The serious real use case for this was if you had accidentally deleted a file but some process was keeping it open somewhere, but I mainly used it back when youtube was flash based it would try and keep the video cache file hidden this way. and you could make a copy of the video by snaging the correct file descriptor.
- jumploops 2y ago
  I have fond memories of modifying the memory of flash-based games when they first launched on Facebook. I don't remember the FB account credentials, but I received thousands of messages from people either telling me I was a terrible person, or trying to bribe me so they could beat their sibling... all for a simple helicopter game. Fun times.

cryptonector 2y ago

The problem is `fork()`. Use `vfork()` or `posix_spawn()`.

tamimio 2y ago

Nice article! I remember three years ago, I needed to spawn a few processes in Node, and I ended up using child process spawn to spawn an external binary. It was fast and instantaneous (no benchmarks). It was on a Linux OS, ARM-based.

bsaunder 2y ago

Have you looked into profiling with strace? You should be able to see the actual system calls each of your tests are making. This will probably give you some good insights into what accounts for the speed differences.