3 ms·

> "The first fix was to avoid calling CommandLineToArgvW by manually parsing the command-line." > "The second fix was to delay load shell32.dll." If your buil

> "The first fix was to avoid calling CommandLineToArgvW by manually parsing the command-line." > "The second fix was to delay load shell32.dll." If your build pipeline is continuously spawning processes all-over, to the point "delay loading" makes a significant difference - it's time to start re-evaluating the entire pipeline and the practices employed.

wtallis 8y ago
Do you know of a build system that can handle a source tree as large as an entire web browser without spawning a lot of processes? It's hard to tell what, if anything, you are recommending here. Pass thousands of files to a single compiler invocation? Ignore the problems and stop trying to make process creation and clean-up faster?
- im3w1l 8y ago
  > Pass thousands of files to a single compiler invocation? Sure. Or pass it a file with all the filenames. Or have the compiler work as a server that takes compilation requests over a socket. It's not like passing thousands of filenames between two processes is a deep unsolved problem.
  - gpderetta 8y ago
    Or just spawn thousands of processes which has been done for the last 40 years without particular issues.
  - mannykannot 8y ago
    So, the solution to concurrency problems is to serialize everything?
    - deleted 8y ago
      [deleted]
    - fit2rule 8y ago
      "Concurrency is everything serialised, properly."
      - mannykannot 8y ago
        The posts we are replying to here seem to have a very narrow concept of what 'properly' entails in this case.
- xtrapolate 8y ago
  > "It's hard to tell what, if anything, you are recommending here." I don't think it's that hard to tell, you're simply ignoring what's been clearly written. So I'll quote it again and hope this time you'll actually read my comment: > "Start re-evaluating the entire pipeline and the practices employed". I've worked on several of projects far exceeding the size of "an entire web browser". You're taking the status quo for what it is. I'm asking questions. Are they employing caching of binaries/object files? Are they running a continuous build? Which parts of the build actually take the most time to go through? Can they benefit from building concurrently on multiple machines?
  - Dylan16807 8y ago
    > Are they employing caching of binaries/object files? Are they running a continuous build? Which parts of the build actually take the most time to go through? Can they benefit from building concurrently on multiple machines? So you're suggesting things they already do. And if you're using multiple machines, you probably want to max out each machine, so it matters a lot if there's code that falls over and dies when applied to a large number of cores.
kornish 8y ago
Did you read the entire article? The whole point was that "delay loading" a particular DLL prevents a static analysis in the compiler from inserting hooks to perform expensive operations.
- xtrapolate 8y ago
  > "Did you read the entire article? The whole point was that "delay loading" a particular DLL prevents a static analysis in the compiler from inserting hooks to perform expensive operations." I actually have read the entire article. Have you? Your explanation has absolutely nothing to do with the performance gains observed. Moreover, in the context of delay-loaded DLLs, your explanation actually makes no sense whatsoever. Delay loaded DLLs, a linker/loader optimization Microsoft has offered since the days of C++ 6.0 (1998), simply means most process invocations in OP's case won't actually end up loading said DLLs, reducing the amount of time spent in DLL_PROCESS_ATTACH/DLL_PROCESS_DETACH (and specifically during destruction, in the Kernel).
  - masklinn 8y ago
    > I actually have read the entire article. You provide no evidence of it, and ample evidence to the contrary. > Delay loaded DLLs, a linker/loader optimization Microsoft has offered since the days of C++ 6.0 (1998), simply means most process invocations in OP's case won't actually end up loading said DLLs, reducing the amount of time spent in DLL_PROCESS_ATTACH/DLL_PROCESS_DETACH. It also avoids loading gdi.dll, which avoids creating a bunch gdi objects, which avoids taking the "destroy gdi object" codepath on process termination… which is the bit that is both slow and globally serialised. TFA's final section even demonstrates the difference it makes: 30% increase in start time including 300% increase in lock contention time but 200% increase in shutdown time including 400% increase in lock contention. The process shutdown is almost entirely serialised due to (as TFA and its predecessor explain) a system-wide lock held during GDI cleanup.
fuzzy2 8y ago
The second paragraph mentions that this is about a test suite. It has to “spawn processes all-over” to do its job.
- nwellnhof 8y ago
  Technically, it doesn't has to. You can put your whole test suite into a single executable, making it run extremely fast for C/C++ projects where process startup is often much slower than running a single test unit. This approach is used in some OSS projects I've worked on but it also has its downsides.
  - fuzzy2 8y ago
    Some tests (e.g. unit tests) can run like this, yes. Other tests, including some benchmarks, are not meaningful when wrapped in a single process. Invocation speed and specifics matter.

wtallis 8y ago

Do you know of a build system that can handle a source tree as large as an entire web browser without spawning a lot of processes? It's hard to tell what, if anything, you are recommending here. Pass thousands of files to a single compiler invocation? Ignore the problems and stop trying to make process creation and clean-up faster?

im3w1l 8y ago
> Pass thousands of files to a single compiler invocation? Sure. Or pass it a file with all the filenames. Or have the compiler work as a server that takes compilation requests over a socket. It's not like passing thousands of filenames between two processes is a deep unsolved problem.
- gpderetta 8y ago
  Or just spawn thousands of processes which has been done for the last 40 years without particular issues.
- mannykannot 8y ago
  So, the solution to concurrency problems is to serialize everything?
  - deleted 8y ago
    [deleted]
  - fit2rule 8y ago
    "Concurrency is everything serialised, properly."
    - mannykannot 8y ago
      The posts we are replying to here seem to have a very narrow concept of what 'properly' entails in this case.
xtrapolate 8y ago
> "It's hard to tell what, if anything, you are recommending here." I don't think it's that hard to tell, you're simply ignoring what's been clearly written. So I'll quote it again and hope this time you'll actually read my comment: > "Start re-evaluating the entire pipeline and the practices employed". I've worked on several of projects far exceeding the size of "an entire web browser". You're taking the status quo for what it is. I'm asking questions. Are they employing caching of binaries/object files? Are they running a continuous build? Which parts of the build actually take the most time to go through? Can they benefit from building concurrently on multiple machines?
- Dylan16807 8y ago
  > Are they employing caching of binaries/object files? Are they running a continuous build? Which parts of the build actually take the most time to go through? Can they benefit from building concurrently on multiple machines? So you're suggesting things they already do. And if you're using multiple machines, you probably want to max out each machine, so it matters a lot if there's code that falls over and dies when applied to a large number of cores.

kornish 8y ago

Did you read the entire article? The whole point was that "delay loading" a particular DLL prevents a static analysis in the compiler from inserting hooks to perform expensive operations.

xtrapolate 8y ago
> "Did you read the entire article? The whole point was that "delay loading" a particular DLL prevents a static analysis in the compiler from inserting hooks to perform expensive operations." I actually have read the entire article. Have you? Your explanation has absolutely nothing to do with the performance gains observed. Moreover, in the context of delay-loaded DLLs, your explanation actually makes no sense whatsoever. Delay loaded DLLs, a linker/loader optimization Microsoft has offered since the days of C++ 6.0 (1998), simply means most process invocations in OP's case won't actually end up loading said DLLs, reducing the amount of time spent in DLL_PROCESS_ATTACH/DLL_PROCESS_DETACH (and specifically during destruction, in the Kernel).
- masklinn 8y ago
  > I actually have read the entire article. You provide no evidence of it, and ample evidence to the contrary. > Delay loaded DLLs, a linker/loader optimization Microsoft has offered since the days of C++ 6.0 (1998), simply means most process invocations in OP's case won't actually end up loading said DLLs, reducing the amount of time spent in DLL_PROCESS_ATTACH/DLL_PROCESS_DETACH. It also avoids loading gdi.dll, which avoids creating a bunch gdi objects, which avoids taking the "destroy gdi object" codepath on process termination… which is the bit that is both slow and globally serialised. TFA's final section even demonstrates the difference it makes: 30% increase in start time including 300% increase in lock contention time but 200% increase in shutdown time including 400% increase in lock contention. The process shutdown is almost entirely serialised due to (as TFA and its predecessor explain) a system-wide lock held during GDI cleanup.

fuzzy2 8y ago

The second paragraph mentions that this is about a test suite. It has to “spawn processes all-over” to do its job.

nwellnhof 8y ago
Technically, it doesn't has to. You can put your whole test suite into a single executable, making it run extremely fast for C/C++ projects where process startup is often much slower than running a single test unit. This approach is used in some OSS projects I've worked on but it also has its downsides.
- fuzzy2 8y ago
  Some tests (e.g. unit tests) can run like this, yes. Other tests, including some benchmarks, are not meaningful when wrapped in a single process. Invocation speed and specifics matter.