5 ms·

Why someone needs 1000 created processes on one server? I worked with IO and CPU bound tasks and never needed more than ~50 processes, even on very big servers.

Why someone needs 1000 created processes on one server? I worked with IO and CPU bound tasks and never needed more than ~50 processes, even on very big servers. Is there any practical reason for this (besides pure interest)?

seba_dos1 7y ago
The article literally answers that. It's very common for unit tests and building infra, especially when ported from POSIX systems.
- MaulingMonkey 7y ago
  And as for why you might want to spawn processes in these cases - it helps isolate memory corruption, other unrecoverable process failures, or even just issues like memory fragmentation to a specific unit test / file build - so that issues building/testing X don't show up as intermitent failures that get misattributed to building/testing Y. As a workaround for perf issues like the article, you might choose to reuse the same process more instead, but it's a tradeoff.
- varelaz 7y ago
  Sorry, I thought problem is with thousands parallel processes execution. There problem is with some remainings from previous process runs.
  - seba_dos1 7y ago
    ...or with the fact that they're constantly being created and destroyed as some tests/build tasks finish and the new ones are spawned.
brucedawson 7y ago
When building Chrome from source code our build system creates about 70,000 processes. This is enough that, even without CFG issues, process-creation time is a factor in the minimum build time.
- BubRoss 7y ago
  Sounds like it might work much better to only have as many processes as there are logical cores.
  - toast0 7y ago
    Most build processes are going to run at least the compiler once per code file. Plus or minus a linker and preprocessor and whatever else. Nobody said all 70,000 quoted processes were run simultaneously. You could presumably write a compiler that didn't do the traditional start - read - process - output - quit cycle to avoid paying startup costs, but it is generally assumed that startup cost is small relative to processing, and that avoiding complexity is worth the cost; if your OS makes startup really expensive, maybe that's not a reasonable assumption.
  - brucedawson 7y ago
    We avoid having more processes that cores running at a time. But we run a separate compiler instance for each file we compile, 'cause why not? So, 70,000 processes over the length of the build.
nwallin 7y ago
Compilers generally instantiate one process per input file. (.cpp or .c files, but not .h) Because the compiler writer can generally be assured that the process is very short lived, compilers are typically written to never use free(). When the object file is created, the process exits, and the memory is freed by the OS. This is generally a significant speedup. Unit tests. When a unit test crashes, that crash needs to be logged, and it can't interrupt the other tests. The good way to do this is by creating a new process per unit test. Some web servers in certain configurations will start one process per connection. If you have 1000 simultaneous connections, you have 1000 processes. https://httpd.apache.org/docs/2.4/mod/mpm_winnt.html https://httpd.apache.org/docs/2.4/mod/mpm_winnt.html That's 3 examples. There are others.

seba_dos1 7y ago

The article literally answers that. It's very common for unit tests and building infra, especially when ported from POSIX systems.

MaulingMonkey 7y ago
And as for why you might want to spawn processes in these cases - it helps isolate memory corruption, other unrecoverable process failures, or even just issues like memory fragmentation to a specific unit test / file build - so that issues building/testing X don't show up as intermitent failures that get misattributed to building/testing Y. As a workaround for perf issues like the article, you might choose to reuse the same process more instead, but it's a tradeoff.
varelaz 7y ago
Sorry, I thought problem is with thousands parallel processes execution. There problem is with some remainings from previous process runs.
- seba_dos1 7y ago
  ...or with the fact that they're constantly being created and destroyed as some tests/build tasks finish and the new ones are spawned.

brucedawson 7y ago

When building Chrome from source code our build system creates about 70,000 processes. This is enough that, even without CFG issues, process-creation time is a factor in the minimum build time.

BubRoss 7y ago
Sounds like it might work much better to only have as many processes as there are logical cores.
- toast0 7y ago
  Most build processes are going to run at least the compiler once per code file. Plus or minus a linker and preprocessor and whatever else. Nobody said all 70,000 quoted processes were run simultaneously. You could presumably write a compiler that didn't do the traditional start - read - process - output - quit cycle to avoid paying startup costs, but it is generally assumed that startup cost is small relative to processing, and that avoiding complexity is worth the cost; if your OS makes startup really expensive, maybe that's not a reasonable assumption.
- brucedawson 7y ago
  We avoid having more processes that cores running at a time. But we run a separate compiler instance for each file we compile, 'cause why not? So, 70,000 processes over the length of the build.

nwallin 7y ago

Compilers generally instantiate one process per input file. (.cpp or .c files, but not .h) Because the compiler writer can generally be assured that the process is very short lived, compilers are typically written to never use free(). When the object file is created, the process exits, and the memory is freed by the OS. This is generally a significant speedup. Unit tests. When a unit test crashes, that crash needs to be logged, and it can't interrupt the other tests. The good way to do this is by creating a new process per unit test. Some web servers in certain configurations will start one process per connection. If you have 1000 simultaneous connections, you have 1000 processes. https://httpd.apache.org/docs/2.4/mod/mpm_winnt.html https://httpd.apache.org/docs/2.4/mod/mpm_winnt.html That's 3 examples. There are others.