6 ms·

I wrote about how I keep build times sane in SumatraPDF at https://blog.kowalczyk.info/article/96a4706ec8e44bc4b0bafda2d9ba502f/extreme-include-discipline-for-c

• sfpotter 2y ago
  How much of a speedup did you get switching over to Rob Pike style includes?
  • cpeterso 2y ago
    I had to look this up: Rob Pike’s rule is that header files should not include other header files; they should only document which other header files they depend on so programmers can directly include those other header files in their .c files. https://bytes.com/topic/c/answers/217074-rob-pikes-simple-include-rule https://bytes.com/topic/c/answers/217074-rob-pikes-simple-in...
    • fooker 2y ago
      This seems like the kind of thing that a compiler should be good at instead of making the user jump through hoops.
      • techbrovanguard 2y ago
        I'm glad Rob Pike didn't contribute to Go, that'd have been catastrophic to my productivity! Imagine having tools do work instead of users, it'd be like a world without lawyers. Anyhow, I've got to refactor my WIP code for the 10th time today since unused variables are compiler errors. Seeya!
    • ErikCorry 2y ago
      If the .h file is documenting which other header files it needs, what keeps that documentation up to date. This sounds like a very painful way to work, where .h files fail to compile (with very strange error messages because of the way C++ works) and then you have to hunt down the missing dependency.
      • flohofwoe 2y ago
        In my header libraries I have checks like this which produce a meaningful error message: #if !defined(SOKOL_GFX_INCLUDED) #error "Please include sokol_gfx.h before sokol_shape.h" #endif
    • iainmerrick 2y ago
      Wow, I hadn't heard of that one. It sounds very unpleasant. I have the exact opposite rule -- it should always be OK to include a header file on its own. As a corollary, headers must include or forward-declare everything they need.
• johannes1234321 2y ago
  > On another note: C++ compiler should by default keep statistics about the chain of #include's / parsing during compilation and dump it to a file at the end and also summarize how badly you're re-parsing the same .h files during build. Not exactly that, but do you know clang's -ftime-trace and tools like https://github.com/aras-p/ClangBuildAnalyzer https://github.com/aras-p/ClangBuildAnalyzer which help analyzing where time is actually spent? (In small repeated headers I don't see much of a problem, but they of course may contain not so small things ...)
  • touisteur 2y ago
    Any idea whether gcc is roadmapping including this?
    • gpderetta 2y ago
      https://gcc.gnu.org/bugzilla/show_bug.cgi?id=92396 https://gcc.gnu.org/bugzilla/show_bug.cgi?id=92396 In the meantime there is -ftime-report
• electroly 2y ago
  If you use a "unity" build (all .cpp files concatenated into one compilation unit) with the normal '#pragma once' or guard in the headers, then you can be certain that every header file is parsed only once, with no discipline needed. CMake can do this for you with an option. You lose the ability to do incremental builds of individual changed files, but it may be fast enough that you don't care. For my own personal projects, I just use ccache and precompiled headers. It's good enough for me. I don't want to have to apply "hard-core discipline" to my projects.
• Maxatar 2y ago
  >On another note: C++ compiler should by default keep statistics about the chain of #include's / parsing during compilation and dump it to a file at the end and also summarize how badly you're re-parsing the same .h files during build. Clang does offer something very close to this, and if you use it you'll find parsing the same/duplicate header files contributes on the order of microseconds to the overall compile time. Your article is from 1989, which is likely before compilers implemented duplicate header file elimination [1], but nowadays all C++ compilers optimize the common #ifndef/#define header guard as well as #pragma once so that they entirely ignore parsing the same file over and over again. [1] https://gcc.gnu.org/onlinedocs/cpp/Once-Only-Headers.html https://gcc.gnu.org/onlinedocs/cpp/Once-Only-Headers.html
  • WalterBright 2y ago
    True, but when you're compiling a.c and then b.c, the .h files get reparsed all over again.
    • deleted 2y ago
      [deleted]
    • kazinator 2y ago
      This is not only not true, but not possible. Many kind sof defintiions in .h files may not be repeated without error, like: struct foo { int bar; }; or typedef int xyzzy_t; This is why you have include guards: #ifndef FOO_H_3DF0_755A #define FOO_H_3DF0_755A struct foo { int bar; } #endif GCC optimized the handling of headers with include guards 30 years ago already. If you think most of your compile time is spent in preprocessing, benchmark a clean build with the optimization set to -O0 versus your full optimization like -O2 or whatever you are using. Both builds perform preprocessing; thus the preprocessing time is bounded by the total time spent in the -O0 build: if the actual semantic analysis of tokens and code generation at -O0 were to take next to no time at all, then we would have to attribute the time to tokenizing and preprocessing. But even then, all the additional time observed under -O2 is not tokenizing and preprocessing.
      • 2y ago
• jay-barronville 2y ago
  As much as I hate long compilation times, I also value code discoverability, readability, and consistency. Your (i.e., Rob Pike’s) strategy seems like a nightmare to me. I love Rob, but I can’t follow this rule.
• nextaccountic 2y ago
  Shouldn't precompiled headers mitigate that?
• eru 2y ago
  Oh, the joys of using a language that thinks (automated) copy-and-paste coding is the same as a module system.
• WalterBright 2y ago
  Having written, benchmarked, and maintained C and C++ compilers for decades, I know why the compiles are slow: 1. phases of translation 2. constant rescanning and reparsing of .h files 3. cannot parse without doing semantic analysis 4. the preprocessor has its own tokens - so you gotta tokenize the .h file, do the preprocessing, convert it back to text, then tokenize the text again with the C/C++ compiler. This is madness. (Although with the C compiler I did manage to merge the preprocessor lexer with the compiler lexer, this made it the speed champ.) This experience fed into D which: 1. uses modules instead of .h files. No matter how many times a module is imported, it is lexed/parsed/semanticed exactly once. 2. module semantics are independent of who/what imports them 3. no phases of translation 4. lexing and parsing is independent of semantic analysis
  • kazinator 2y ago
    Having done some casual benchmarking recently, I found that GCC is about 15 times slower when optimizing than when not. In both situations, the compiler is scanning the same header files, so that activity is bound up within the 1/15th of the optimized compilation time. It used to be a common wisdom that the character-level processing of code took the most time. Just like the old floating-point is slow; always use integer when possible. Also note that the ccache tool greatly speeds up C and C++ builds. Yet, the input to ccache is the preprocessed translation unit! When you're using ccache, none of the preprocessing is skipped. ccache hashes the preprocesed translation unit (plus compiler command line options and the path of the compiler executable and such) in an intelligent way and the checks its cache. If there is a hit, it pulls the .o out of its cache, otherwise it invokes the compiler on the preprocessed translation unit. If most of the time were spent in preprocessing, a much more modest speedup would be observed with ccache.
    • WalterBright 2y ago
      Generally when benchmarking compile speeds, the unoptimized build is used, as that is the edit-compile-debug loop. It's always been true that a good optimizer will dominate the build times. Back in the Bronze Age (1990s) I endeavored to speed up compilation in a manner that you describe ccache as doing. After the .h files were taken care of, the compiler would roll out to disk the state of the compiler. (It could also do this with individual .h files.) Then, instead of doing all the .h files again, it would just memory map in the precompiled .h file. And yes, it resulted in a dramatic improvement in compile times, as you describe. The downside was one had to be extremely careful about compiling the .h files the same way each time. One difference could affect the path through the .h files, and invalidate the precompiled version. It was quite a lot of careful work to make that work, and I expect ccache is also a complex piece of work. What I learned from that is it's easier to just fix the language so none of that is necessary. C/C++ can be so fixed, the proof is ImportC, a C compiler that can use imports instead of .h files, and can compile multiple .c files in one invocation and merge them into a single .o file.
• kazinator 2y ago
  In modern compilers, the scanning of tokens is a tiny fraction of the compile time. If the header files are properly protected with inclusion guards, then at worst the contents are tokenized by the preprocessor, and not seen by anything else. But for decades now, compilers have been smart enough to recognize include guards and not actually open the file. So that is to say, when the compiler has seen a file of this form once: #ifndef FOO #define FOO ... #endif and is asked to #include that same file again, it knows that this file is guarded by FOO. If FOO exists, it won't even open that file. There are reasons to avoid including .h files in .h files, but you're not going to get compile time gains out of it, other than perhaps through secondary effects. By secondary effects I mean this: when in a project you forbid .h files including other .h files, it hurts to add new dependencies into header files. When you change a header such that it needs some definition in another, you have to go into every .cpp file and add the #include. So, you find ways to reduce the dependencies to avoid doing that. When .h files are allowed to include others, the dependencies grow rampant. You want to compile a little banana, but it depends on the gorilla, which needs the whole jungle, just to be defined. Juggling the jungle definition takes a bit of time. C++ definitions are complicated, requiring lots of processing to develop. Not as much as optimizing the banana that is the actual subject being compiled to code, but not as little as skipping header files.
• highfrequency 2y ago
  > We try to mitigate it with #ifdef guards, #pragma once etc. but in my experience those band-aids don’t solve the problem. Why don't include guards solve the problem of duplicate parsing of .h files within one compilation unit? I believe modern compilers can entirely optimize away even the opening and scanning of the file. And even without that, modern NVMe disks are so fast I would imagine the file opens would be negligible. Curious to hear if anyone has data on whether duplicate header file parsing is still an actual performance issue even with modern compilers, modern SSDs, and #pragma once.
• greenavocado 2y ago
  Only a handful of commenters have mentioned "#pragma once", which is alarmingly few people considering how many C++ practitioners are out there. "#pragma once" is the obvious way to deal with this problem.
  • Arech 2y ago
    `#pragma once` isn't standardized. Even though all major modern compilers do support it (hopefully, but not guaranteed in the same way), if you care about portability to some not even very esoteric platform, it most likely uses ancient compilers (or based on ancient version of a major compiler) that might not support it. But besides that I too don't see a reason why not use a better `#pragma once` instead if include guards...
    • maccard 2y ago
      Pragma once is a defacto standard and is supported by every compiler and platform you’re ever going to work with. If you need to support the targets it doesn’t work with, then you’ll get a loud compile error. The old include guards are error prone, requiring a unique key per header. Getting that wrong can cause anything from a compile error, an obtuse linker error, or a runtime bug depending on what’s in the header. The chances of someone copy pasting a header and breaking the include guard are significantly higher than the chances of someone deciding that we need to support an esoteric embedded target overnight.
      • rcxdude 2y ago
        This. #pragma once is more consistently supported than many parts of the standard. Just use it. (And I've worked with some of those esoteric embedded targets. Haven't run into a compiler that didn't support it yet)
    • deleted 2y ago
      [deleted]
    • flohofwoe 2y ago
      FWIW I've never seen a compiler that doesn't support '#pragma once' (at least since the late 90s). I did come across some esoteric path-related problems with pragma once though which wouldn't occur with include guards. IIRC it was related to filesystem links.
• timvdalen 2y ago
  Thanks for building SumatraPDF! I just switched away from Windows, but I've been a happy user for a very long time, and it's hard to get that mix of user experience just right. Really appreciate your work.
• intelVISA 2y ago
  The entire article's premise is misguided as slow C++ build times are usually a result of inexperienced SWEs unfamiliar with the language's "fun" build rules.
  • Arech 2y ago
    In enterprise you are almost always under a pressure to deliver something asap, so some bad practices are just a natural consequence of that. They could have liked to do it differently, but then they would only be half way to delivering what they have already shipped. There are always tradeoffs.
  • maccard 2y ago
    This is the equivalent of saying C is safe to use, as long as you use it correctly, or your iPhone is fine you’re just holding it wrong. If build times are a wide spread problem with a language, that’s a language problem not a user problem.
    • exe34 2y ago
      Knives are really difficult to use, everybody keeps holding the nice shiny metal part, but somehow you're meant to just know to hold the wooden part.