24 ms·

The real solution to fast compilation is to have modules. These kind of one-off hacks are interesting but will not scale.

by npalli 3y ago

citrin_ru 3y ago
Modules is one of reasons why Pascal is fast to compile and I wish it was more popular than it was.
- jll29 3y ago
  Even more so in Modula-2 (as the name suggests), where you can compile module definitions separately from their implementations and already have the compiler perform syntax checks against APIs defined therein before they are even implemented.
foofie 3y ago
> The real solution to fast compilation is to have modules. I don't agree. The real solution for fast compilation times is to not have to recompile things. This means onboarding tools like ccache, organize your project around independent subprojects which eliminate/minimize compile-time dependencies, and leverage incremental builds. There's a C++ book somewhere that describes how the subprojects approach helps lower build times to a residual length which if my memory doesn't fail me was dubbed horizontal architecture. It consists of designing every single subprojects to be stand-alone and leave any integration to the linking stage. I've used it in the past on a legacy C++ project that took slightly over 1h to pull off an end-to-end build, and peeling out 4 or 5 subprojects following the horizontal architecture approach allowed me to cut down full end-to-end build times to slightly over 8 minutes and incremental builds down to less than a minute, without bothering with compiler caches. I'm sure that if I bothered onboarding ccache I could drive incremental build times to just a few seconds. If you know what you are doing, you can do a lot without begging for magic.
- krapht 3y ago
  Nah, C++ needs more than that. There's some ridiculous template heavy code out there where the majority of time is spent linking. You can't even do a debug build without optimizations on Windows with these programs because the coff file format can't handle it, even when compiling with "/bigobj".
  - foofie 3y ago
    > Nah, C++ needs more than that. There's some ridiculous template heavy code out there where the majority of time is spent linking. C++ does not "need" more than that. You personally might find it more convenient if you don't have to think through your software architecture, but the truth of the matter is that you only need to spend a few minutes looking at your project to speed everything up, which is exactly what everyone does the very moment they feel they have a problem. It makes no sense at all to demand a whole tech stack to change around you when you can't even spend a few minutes looking for an answer to the problem you have. > There's some ridiculous template heavy code out there where the majority of time is spent linking. That is not a problem. Templates are only special in a build because compilers spend time generating code. Again, you can work around that without any problem at all with the tools available to you for the past two or three decades. Properly modularizing your app with explicit template instantiation already drives down the build time of any naive project structure to a fraction of the time, and basic stuff like onboarding a compiler cache tool and moving template code out of interface headers is enough to get the linking stage to be the most expensive step of a build. > You can't even do a debug build without optimizations on Windows with these programs because the coff file format can't handle it, even when compiling with "/bigobj". I'm sorry but this is simply not true. I suggest you start to look at your projects to break it down to subprojects and see where the critical path of your build is. There is absolutely no project in the world whose build would not take less than a minute (or even a few seconds) with incremental builds, even if it's code that uses template metaprogramming extensively.
    - MauranKilom 3y ago
      I agree with the thrust of your argument that there are a lot of compile-time-encapsulation tools you can employ to reduce build times. However, I will still challenge you on your build time/object size claim. Take a codebase structured around lazy tasks (i.e. every function is a coroutine), factor in runtime parallelism/SIMD dispatch, multiply with loop abstractions (think std::ranges) and you get frighteningly close to linker limits in no time (which, of course, also implies minutes of compilation per source file). Each of those pieces is templates through and through, and there's no explicit template instantiation way out of any of them. Also, I have to (sadly) point out that explicit template instantiation declarations only take you so far. Yes, they reduce compiler time spent in codegen, but also cause the compiler to create all required transitive (!) declarations in every TU, which will quickly offset those gains when talking about template types that have lots of small member functions.
- cozzyd 3y ago
  Somehow my Fedora laptop seemed to automagically use ccache, which made me happy.
pjmlp 3y ago
Available since C++20, C++23 brings the whole standard library in a simple import std; and is already available in VC++, clang 17/cmake, with GCC 14 catching up.
- oldpersonintx 3y ago
  [dead]
vitaut 3y ago
I agree that modules is the right long-term solution and in fact {fmt} is modularized: https://vitaut.net/posts/2023/cxx20-modules-in-clang/ https://vitaut.net/posts/2023/cxx20-modules-in-clang/.

The real solution to fast compilation is to have modules. These kind of one-off hacks are interesting but will not scale.

by npalli 3y ago

The real solution to fast compilation is to have modules. These kind of one-off hacks are interesting but will not scale.

citrin_ru 3y ago

Modules is one of reasons why Pascal is fast to compile and I wish it was more popular than it was.

jll29 3y ago
Even more so in Modula-2 (as the name suggests), where you can compile module definitions separately from their implementations and already have the compiler perform syntax checks against APIs defined therein before they are even implemented.

foofie 3y ago

> The real solution to fast compilation is to have modules. I don't agree. The real solution for fast compilation times is to not have to recompile things. This means onboarding tools like ccache, organize your project around independent subprojects which eliminate/minimize compile-time dependencies, and leverage incremental builds. There's a C++ book somewhere that describes how the subprojects approach helps lower build times to a residual length which if my memory doesn't fail me was dubbed horizontal architecture. It consists of designing every single subprojects to be stand-alone and leave any integration to the linking stage. I've used it in the past on a legacy C++ project that took slightly over 1h to pull off an end-to-end build, and peeling out 4 or 5 subprojects following the horizontal architecture approach allowed me to cut down full end-to-end build times to slightly over 8 minutes and incremental builds down to less than a minute, without bothering with compiler caches. I'm sure that if I bothered onboarding ccache I could drive incremental build times to just a few seconds. If you know what you are doing, you can do a lot without begging for magic.

krapht 3y ago
Nah, C++ needs more than that. There's some ridiculous template heavy code out there where the majority of time is spent linking. You can't even do a debug build without optimizations on Windows with these programs because the coff file format can't handle it, even when compiling with "/bigobj".
- foofie 3y ago
  > Nah, C++ needs more than that. There's some ridiculous template heavy code out there where the majority of time is spent linking. C++ does not "need" more than that. You personally might find it more convenient if you don't have to think through your software architecture, but the truth of the matter is that you only need to spend a few minutes looking at your project to speed everything up, which is exactly what everyone does the very moment they feel they have a problem. It makes no sense at all to demand a whole tech stack to change around you when you can't even spend a few minutes looking for an answer to the problem you have. > There's some ridiculous template heavy code out there where the majority of time is spent linking. That is not a problem. Templates are only special in a build because compilers spend time generating code. Again, you can work around that without any problem at all with the tools available to you for the past two or three decades. Properly modularizing your app with explicit template instantiation already drives down the build time of any naive project structure to a fraction of the time, and basic stuff like onboarding a compiler cache tool and moving template code out of interface headers is enough to get the linking stage to be the most expensive step of a build. > You can't even do a debug build without optimizations on Windows with these programs because the coff file format can't handle it, even when compiling with "/bigobj". I'm sorry but this is simply not true. I suggest you start to look at your projects to break it down to subprojects and see where the critical path of your build is. There is absolutely no project in the world whose build would not take less than a minute (or even a few seconds) with incremental builds, even if it's code that uses template metaprogramming extensively.
  - MauranKilom 3y ago
    I agree with the thrust of your argument that there are a lot of compile-time-encapsulation tools you can employ to reduce build times. However, I will still challenge you on your build time/object size claim. Take a codebase structured around lazy tasks (i.e. every function is a coroutine), factor in runtime parallelism/SIMD dispatch, multiply with loop abstractions (think std::ranges) and you get frighteningly close to linker limits in no time (which, of course, also implies minutes of compilation per source file). Each of those pieces is templates through and through, and there's no explicit template instantiation way out of any of them. Also, I have to (sadly) point out that explicit template instantiation declarations only take you so far. Yes, they reduce compiler time spent in codegen, but also cause the compiler to create all required transitive (!) declarations in every TU, which will quickly offset those gains when talking about template types that have lots of small member functions.
cozzyd 3y ago
Somehow my Fedora laptop seemed to automagically use ccache, which made me happy.

pjmlp 3y ago

Available since C++20, C++23 brings the whole standard library in a simple import std; and is already available in VC++, clang 17/cmake, with GCC 14 catching up.

oldpersonintx 3y ago
[dead]

vitaut 3y ago

I agree that modules is the right long-term solution and in fact {fmt} is modularized: https://vitaut.net/posts/2023/cxx20-modules-in-clang/ https://vitaut.net/posts/2023/cxx20-modules-in-clang/.