4 ms·

In my experience of building large C++ projects on machines with >= 64 threads (especially if using ccache or something), linking is often the bottleneck to inc

by berkut 4y ago

In my experience of building large C++ projects on machines with >= 64 threads (especially if using ccache or something), linking is often the bottleneck to incremental compilation as opposed to cpu-bound compilation. Some of our dev builds actually are split up into separate .so files which is a bit slower at execution time, but means linking is a lot faster, especially when you want full debug info...

jupp0r 4y ago
Chrome is doing the exact same thing for the same reason. Dev builds are dynamically linked while release builds are statically linked. Works quite well, it's a great technique.
- pjmlp 4y ago
  It is quite common in Windows, either produce lib or dlls for each set of main modules. In the old days it used to be quite common on UNIX world as well, no idea why "modern" Linux seems to have forgotten about this approach and always builds everything as a single project unit.
  - bombolo 4y ago
    Because google decided we all want one big binary that contains the go runtime and weights at least 100MB
    - mwcampbell 4y ago
      I have several Go-based binaries in my $HOME/bin, and not a single one is 100 MB.
      - vegerot 4y ago
        And all of them combined compile faster than either of these implementations!
      - bombolo 4y ago
        I'm honestly impressed, all my ones are at least 60MB
        vegerot 4y ago
        #itdepends on the kind of software you’re making
  - jupp0r 4y ago
    Because we don't want to release software for each Linux distribution and deal with perpetually outdated libraries.
fooker 4y ago
There are linkers which are significantly faster than the ones used by default. Gold and bfd are two examples. I use these for development and then test with the default linker just in case there’s some subtle issue hiding somewhere.
- berkut 4y ago
  Gold's pretty slow by today's standards! - it was fast in say 2009 compared to ld, but not really anything to write home about these days. lldb and mold (mentioned in article) are much faster.
  - fooker 4y ago
    Gold is marginally faster than ld and uses significantly less memory when building large-ish projects like LLVM.
    - jhasse 4y ago
      And lld is faster than gold. And mold is faster than lld.
  - jhasse 4y ago
    I think you meant lld instead of lldb (which is LLVM's debugger).
    - berkut 4y ago
      Yes, muscle memory :)
strager 4y ago
I checked how long linking takes for quick-lint-js [1] with Mold. 108 milliseconds. That's pretty chunky. I should try .so files like you suggest. [1] the full project, not the 17k SLOC subset discussed in the article
drothlis 4y ago
I found that Ninja tries hard to build things in the order they’re listed in the build file (dependencies permitting). Whereas GNU Make... well, it does start off trying to build depth-first, but if it needs to build a target's prerequisites, it pushes that target to the very end of the queue. So Make ends up building the leaf targets at the very end -- it leaves all the linking steps until all the object files have been compiled, instead of linking each executable as soon as its own dependencies are available. I reported that here: https://lists.gnu.org/archive/html/help-make/2016-11/msg00005.html https://lists.gnu.org/archive/html/help-make/2016-11/msg0000... ...and the maintainer said that the behaviour isn't intentional, but after a short look at the source code I decided fixing it was going to be beyond my abilities/motivation. (I suppose this is more relevant to full builds, not incremental ones.)
throw827474737 4y ago
I didn't get why he doesn't see barely any improvement using mold, while mold's advertising/benchmarks show 1-2 magnitudes of improvements over gold, and still at least 50% vs llvm. Reading through mold's nice docs I could only explain that by him not utilizing multiple cores (molds actual strength besides other things), but seems pretty unlikely. Who can explain?
- strager 4y ago
  For the 17k SLOC project, link times were already under 100 milliseconds without mold. (EDIT: Actually linking took 129 milliseconds with GNU ld.) There are other, bigger bottlenecks to deal with. As a project grows larger, link times become more noticeable, and mold starts to help.
  - throw827474737 4y ago
    Oh sorry you pointed that out below, thank you for doing again!
    - strager 4y ago
      I actually pointed it out in the article when I compared GNU ld with Mold. =] And I was actually wrong in my comment; run_linker took 129 ms, not under 100 ms. https://quick-lint-js.com/blog/cpp-vs-rust-build-times/#optimizing-rust-with-faster-linker https://quick-lint-js.com/blog/cpp-vs-rust-build-times/#opti...
fnord123 4y ago
On Linux, the default visibility is public (on Windows I think it's private). How much of an effect would the default visibility have when linking if most of the symbols in Rust will be kept private?

by berkut 4y ago

jupp0r 4y ago

Chrome is doing the exact same thing for the same reason. Dev builds are dynamically linked while release builds are statically linked. Works quite well, it's a great technique.

pjmlp 4y ago
It is quite common in Windows, either produce lib or dlls for each set of main modules. In the old days it used to be quite common on UNIX world as well, no idea why "modern" Linux seems to have forgotten about this approach and always builds everything as a single project unit.
- bombolo 4y ago
  Because google decided we all want one big binary that contains the go runtime and weights at least 100MB
  - mwcampbell 4y ago
    I have several Go-based binaries in my $HOME/bin, and not a single one is 100 MB.
    - vegerot 4y ago
      And all of them combined compile faster than either of these implementations!
    - bombolo 4y ago
      I'm honestly impressed, all my ones are at least 60MB
      - vegerot 4y ago
        #itdepends on the kind of software you’re making
- jupp0r 4y ago
  Because we don't want to release software for each Linux distribution and deal with perpetually outdated libraries.

fooker 4y ago

There are linkers which are significantly faster than the ones used by default. Gold and bfd are two examples. I use these for development and then test with the default linker just in case there’s some subtle issue hiding somewhere.

berkut 4y ago
Gold's pretty slow by today's standards! - it was fast in say 2009 compared to ld, but not really anything to write home about these days. lldb and mold (mentioned in article) are much faster.
- fooker 4y ago
  Gold is marginally faster than ld and uses significantly less memory when building large-ish projects like LLVM.
  - jhasse 4y ago
    And lld is faster than gold. And mold is faster than lld.
- jhasse 4y ago
  I think you meant lld instead of lldb (which is LLVM's debugger).
  - berkut 4y ago
    Yes, muscle memory :)

strager 4y ago

I checked how long linking takes for quick-lint-js [1] with Mold. 108 milliseconds. That's pretty chunky. I should try .so files like you suggest. [1] the full project, not the 17k SLOC subset discussed in the article

drothlis 4y ago

I found that Ninja tries hard to build things in the order they’re listed in the build file (dependencies permitting). Whereas GNU Make... well, it does start off trying to build depth-first, but if it needs to build a target's prerequisites, it pushes that target to the very end of the queue. So Make ends up building the leaf targets at the very end -- it leaves all the linking steps until all the object files have been compiled, instead of linking each executable as soon as its own dependencies are available. I reported that here: https://lists.gnu.org/archive/html/help-make/2016-11/msg00005.html https://lists.gnu.org/archive/html/help-make/2016-11/msg0000... ...and the maintainer said that the behaviour isn't intentional, but after a short look at the source code I decided fixing it was going to be beyond my abilities/motivation. (I suppose this is more relevant to full builds, not incremental ones.)

throw827474737 4y ago

I didn't get why he doesn't see barely any improvement using mold, while mold's advertising/benchmarks show 1-2 magnitudes of improvements over gold, and still at least 50% vs llvm. Reading through mold's nice docs I could only explain that by him not utilizing multiple cores (molds actual strength besides other things), but seems pretty unlikely. Who can explain?

strager 4y ago
For the 17k SLOC project, link times were already under 100 milliseconds without mold. (EDIT: Actually linking took 129 milliseconds with GNU ld.) There are other, bigger bottlenecks to deal with. As a project grows larger, link times become more noticeable, and mold starts to help.
- throw827474737 4y ago
  Oh sorry you pointed that out below, thank you for doing again!
  - strager 4y ago
    I actually pointed it out in the article when I compared GNU ld with Mold. =] And I was actually wrong in my comment; run_linker took 129 ms, not under 100 ms. https://quick-lint-js.com/blog/cpp-vs-rust-build-times/#optimizing-rust-with-faster-linker https://quick-lint-js.com/blog/cpp-vs-rust-build-times/#opti...

fnord123 4y ago

On Linux, the default visibility is public (on Windows I think it's private). How much of an effect would the default visibility have when linking if most of the symbols in Rust will be kept private?