5 ms·

From my experience, unity builds (single compilation unit with #include of all source files into a single main file) and use of orthodox C++ (avoidance of templ

From my experience, unity builds (single compilation unit with #include of all source files into a single main file) and use of orthodox C++ (avoidance of templates and fancy features) is giving good results.

ladberg 6y ago
That sounds like it breaks all incremental builds (correct me if I'm wrong) for relatively little benefit (skipping linking? less header parsing?). Are there scenarios where unity builds are faster than incremental builds? Or is the intended use-case only for from-scratch builds?
- keldaris 6y ago
  In my experience, you can just do full from-scratch builds in under 10 seconds for any sanely sized codebase (say, under 1M LoC conservatively). This obviously assumes the "orthodox C++" part, so YMMV wildly depending on how far you stray from that. I usually have compile times under 0.5s for all personal projects that don't use CUDA, but I basically write C with a bit of function/operator overloading, absolutely no STL or templates at all. Consequently, I have no use for incremental (or parallel, for that matter) builds whatsoever.
  - asdfasgasdgasdg 6y ago
    Only an extraordinarily ambitious personal project would experience a significant amount of build latency on modern compilers and CPUs, no matter what approach is used. I don't think you can reason from "it works for personal projects" to "it's fast." One thing I can tell you immediately about this unity build idea that is suboptimal for many codebases: modern compilers do not have much intra translation unit parallelism. If there's more in your project than one CPU core can compile quickly (i.e. most shipping products), it's going to be a serious bottleneck.
    - stephc_int13 6y ago
      I am not talking about hobby projects here, but codebases with more than 100K lines of code. I agree with your objection, scaling up to 2-8 compilation units to use several cores is something I also plan to test/benchmark. And to be honest, I don't completely understand why the classic way of compiling (makefiles + dozens separate files) is so slow but there is clearly something fishy as it is very often unbearable. Slow compilation time can be very detrimental for productivity as it creates a way for the mind to wander and lose focus.
    - jcelerier 6y ago
      > One thing I can tell you immediately about this unity build idea that is suboptimal for many codebases to give you a data point, in my case, https://github.com/OSSIA/score https://github.com/OSSIA/score (fairly mundane C++ project totaling ~360kloc in roughly 1300 .cpp files / 1800 .hpp files, split across ~15 libraries, using boost, Qt and a few other common libs), unity builds (one .cpp per library) divides the time by 5 compared to PCH.
      - asdfasgasdgasdg 6y ago
        360kloc certainly meets my definition of extraordinarily ambitious for a personal project. :) Well done you! "Unity" builds certainly can speed up project compilation when used judiciously. This is easily shown by imagining an absurd C++ codebase where each method or function is in its own module. This would increase the overhead of parsing .h files. Any two functions that could be combined into one while keeping the h files included the same would almost certainly reduce this overhead. But I doubt your project would compile faster if all of the cc files were combined into one.
        eska 6y ago
        > This is easily shown by imagining an absurd C++ codebase where each method or function is in its own module That's the musl codebase, in C. It does this in order to be able to remove unused code in a static build. It compiles ridiculously fast. So the problem does not seem to be the amount of headers.
- stephc_int13 6y ago
  Yes, absolutely, that's the trick with unity builds. It is supposed to be slower because you're asking the compiler to rebuild everything from scratch every time. But the simplicity of the single compilation unit leads to very sizeable gains. I plan to write an article on this very subject and post a few benchmarks. https://en.wikipedia.org/wiki/Unity_build https://en.wikipedia.org/wiki/Unity_build
- adrian17 6y ago
  There doesn't need to be exactly one unity translation unit. In Firefox, the build system groups source files by component, and then into "unity" batches containing 5-20 source files each. If a build spends a lot of time processing code in headers, and source files in the same component are likely to include very similar sets of headers, this kind of configuration gives decent benefits, while not making incremental compilation much more expensive (and possibly cheaper, if you edited a header affecting many files in a single component).
barumi 6y ago
It's reasonable to assume that compilation times can go down if you decide to forego compile-time features (i.e., templates). However, C++'s standard library is a productivity blessing. I'm not convinced of the advantage of replacing it with ad-hoc components that may or may not be buggy just because you want to shave off a few seconds of each build. More often than not, all you have to do to lower a double-digit percentage of your compilation time is to modularized your project, and follow basic principles like not have includes in your interface headers. This alone, along with PImpl, does far more for your compilation time than well-meaning but poorly thought-out tradeoff decisions like not using the STL.
- stephc_int13 6y ago
  Using the STL is a non-issue for me as it is almost never used in the video games industry, mostly for performance reasons. In my opinion the STL is still a big, bloated, unreadable piece of code with a few badly implemented good ideas (containers) You should read this article: https://zeux.io/2019/01/17/is-c-fast/ https://zeux.io/2019/01/17/is-c-fast/
  - will4274 6y ago
    I read the article. The author noticed that MSVC's STL has slower iterators in debug mode, because it does extra checks. Instead of reading the manual and turning off the extra checks and/or reading the manual and turning on some optimizations in debug, he re-wrote it in C, which doesn't have the extra checks to begin with, and was happy because it compiled faster. All the optimizations in release mode came from better algorithms or allocators. I can't say I'm particularly convinced. Yes, C++ debug builds can be slow. For most of us, that's ok. If it's not, make your "debug" build configuration a release build with the optimizer turned down (but not off) and incremental build enabled. Preprocessor definitions, symbol generation, optimization, and incremental build are all independent settings - if all you need from debug is symbols, incremental builds, and the inliner turned down (so stacks are retained) - then configure your compiler that way.
    - stephc_int13 6y ago
bregma 6y ago
"Orthodox C++" without templates, language features, or the standard library is also known as "C". I'd go one step further. Skip the compiler and just write in assembly. Your builds will be lightning fast especially if you use one single source file with no macros and static linking. You'll never ship, but you'll never ship really fast.
- stephc_int13 6y ago
  You'd be surprised by how many "C++" projects are in fact mostly C with a few C++ features in the videogames industry.
  - guenthert 6y ago
    > in the videogames industry. Or anywhere else. C++ is such a beast, you just can't expect your coworkers to memorize the whole standard, so it is quite common that teams agree (or the team-leader agrees for them) upon a subset. In the past that was due to limited availability of experienced C++ programmers and inconsistent support from compilers, these days it's due to the evolving nature of the language. Quick: in which standard was the variant class template introduced?
- keldaris 6y ago
  Except people do in fact ship large C projects all the time. The productivity gain associated with moving from assembly to C is undisputed and vastly exceeds any alleged productivity gain associated with moving from C to modern C++. Pretending otherwise is just facetious.

ladberg 6y ago

That sounds like it breaks all incremental builds (correct me if I'm wrong) for relatively little benefit (skipping linking? less header parsing?). Are there scenarios where unity builds are faster than incremental builds? Or is the intended use-case only for from-scratch builds?

keldaris 6y ago
In my experience, you can just do full from-scratch builds in under 10 seconds for any sanely sized codebase (say, under 1M LoC conservatively). This obviously assumes the "orthodox C++" part, so YMMV wildly depending on how far you stray from that. I usually have compile times under 0.5s for all personal projects that don't use CUDA, but I basically write C with a bit of function/operator overloading, absolutely no STL or templates at all. Consequently, I have no use for incremental (or parallel, for that matter) builds whatsoever.
- asdfasgasdgasdg 6y ago
  Only an extraordinarily ambitious personal project would experience a significant amount of build latency on modern compilers and CPUs, no matter what approach is used. I don't think you can reason from "it works for personal projects" to "it's fast." One thing I can tell you immediately about this unity build idea that is suboptimal for many codebases: modern compilers do not have much intra translation unit parallelism. If there's more in your project than one CPU core can compile quickly (i.e. most shipping products), it's going to be a serious bottleneck.
  - stephc_int13 6y ago
    I am not talking about hobby projects here, but codebases with more than 100K lines of code. I agree with your objection, scaling up to 2-8 compilation units to use several cores is something I also plan to test/benchmark. And to be honest, I don't completely understand why the classic way of compiling (makefiles + dozens separate files) is so slow but there is clearly something fishy as it is very often unbearable. Slow compilation time can be very detrimental for productivity as it creates a way for the mind to wander and lose focus.
  - jcelerier 6y ago
    > One thing I can tell you immediately about this unity build idea that is suboptimal for many codebases to give you a data point, in my case, https://github.com/OSSIA/score https://github.com/OSSIA/score (fairly mundane C++ project totaling ~360kloc in roughly 1300 .cpp files / 1800 .hpp files, split across ~15 libraries, using boost, Qt and a few other common libs), unity builds (one .cpp per library) divides the time by 5 compared to PCH.
    - asdfasgasdgasdg 6y ago
      360kloc certainly meets my definition of extraordinarily ambitious for a personal project. :) Well done you! "Unity" builds certainly can speed up project compilation when used judiciously. This is easily shown by imagining an absurd C++ codebase where each method or function is in its own module. This would increase the overhead of parsing .h files. Any two functions that could be combined into one while keeping the h files included the same would almost certainly reduce this overhead. But I doubt your project would compile faster if all of the cc files were combined into one.
      - eska 6y ago
        > This is easily shown by imagining an absurd C++ codebase where each method or function is in its own module That's the musl codebase, in C. It does this in order to be able to remove unused code in a static build. It compiles ridiculously fast. So the problem does not seem to be the amount of headers.
stephc_int13 6y ago
Yes, absolutely, that's the trick with unity builds. It is supposed to be slower because you're asking the compiler to rebuild everything from scratch every time. But the simplicity of the single compilation unit leads to very sizeable gains. I plan to write an article on this very subject and post a few benchmarks. https://en.wikipedia.org/wiki/Unity_build https://en.wikipedia.org/wiki/Unity_build
adrian17 6y ago
There doesn't need to be exactly one unity translation unit. In Firefox, the build system groups source files by component, and then into "unity" batches containing 5-20 source files each. If a build spends a lot of time processing code in headers, and source files in the same component are likely to include very similar sets of headers, this kind of configuration gives decent benefits, while not making incremental compilation much more expensive (and possibly cheaper, if you edited a header affecting many files in a single component).

barumi 6y ago

It's reasonable to assume that compilation times can go down if you decide to forego compile-time features (i.e., templates). However, C++'s standard library is a productivity blessing. I'm not convinced of the advantage of replacing it with ad-hoc components that may or may not be buggy just because you want to shave off a few seconds of each build. More often than not, all you have to do to lower a double-digit percentage of your compilation time is to modularized your project, and follow basic principles like not have includes in your interface headers. This alone, along with PImpl, does far more for your compilation time than well-meaning but poorly thought-out tradeoff decisions like not using the STL.

stephc_int13 6y ago
Using the STL is a non-issue for me as it is almost never used in the video games industry, mostly for performance reasons. In my opinion the STL is still a big, bloated, unreadable piece of code with a few badly implemented good ideas (containers) You should read this article: https://zeux.io/2019/01/17/is-c-fast/ https://zeux.io/2019/01/17/is-c-fast/
- will4274 6y ago
  I read the article. The author noticed that MSVC's STL has slower iterators in debug mode, because it does extra checks. Instead of reading the manual and turning off the extra checks and/or reading the manual and turning on some optimizations in debug, he re-wrote it in C, which doesn't have the extra checks to begin with, and was happy because it compiled faster. All the optimizations in release mode came from better algorithms or allocators. I can't say I'm particularly convinced. Yes, C++ debug builds can be slow. For most of us, that's ok. If it's not, make your "debug" build configuration a release build with the optimizer turned down (but not off) and incremental build enabled. Preprocessor definitions, symbol generation, optimization, and incremental build are all independent settings - if all you need from debug is symbols, incremental builds, and the inliner turned down (so stacks are retained) - then configure your compiler that way.
  - stephc_int13 6y ago

bregma 6y ago

"Orthodox C++" without templates, language features, or the standard library is also known as "C". I'd go one step further. Skip the compiler and just write in assembly. Your builds will be lightning fast especially if you use one single source file with no macros and static linking. You'll never ship, but you'll never ship really fast.

stephc_int13 6y ago
You'd be surprised by how many "C++" projects are in fact mostly C with a few C++ features in the videogames industry.
- guenthert 6y ago
  > in the videogames industry. Or anywhere else. C++ is such a beast, you just can't expect your coworkers to memorize the whole standard, so it is quite common that teams agree (or the team-leader agrees for them) upon a subset. In the past that was due to limited availability of experienced C++ programmers and inconsistent support from compilers, these days it's due to the evolving nature of the language. Quick: in which standard was the variant class template introduced?
keldaris 6y ago
Except people do in fact ship large C projects all the time. The productivity gain associated with moving from assembly to C is undisputed and vastly exceeds any alleged productivity gain associated with moving from C to modern C++. Pretending otherwise is just facetious.