4 ms·
There are so many tools to parallelize compilation, but comparatively little effort has gone into reducing the vast amounts of redundant work that compilers do.
by modeless 7y ago
There are so many tools to parallelize compilation, but comparatively little effort has gone into reducing the vast amounts of redundant work that compilers do. A staggeringly large percentage of compilation time is simply wasted duplicating work that was already done hundreds of times over.
Tools like ccache are really primitive compared to what is possible. Change one bit in a header file and you have to recompile your whole project, even if the output ends up being bit-identical.
Zapcc[1] is more advanced, but still far from ideal. What I really want is a compiler that can incrementally update the binary as I type code, and can start from a snapshot that I download from someone else so that nobody ever needs to do a full recompile. It would require a radically different compiler infrastructure, but I don't see any reason why it would be impossible.
[1] https://github.com/yrnkrn/zapcc https://github.com/yrnkrn/zapcc
- kachurovskiy 7y agoFirst time I spend a day on a partial-compile-only bug I switch if off and it's good old full recompilation from that point :)
- tempguy9999 7y agoThat happened to a colleague of mine with the Eiffel compiler (a long time ago). He was pissed off. Ditto full compilation thereafter. Had something similar in our office with scala & SBT, but SBT is a POS anyway.
- Animats 7y agoThat's a big problem with build systems based on timestamps, not checksums. Also, unless the build system can see all the inputs to the compilation, there's the potential for trouble.
- slacka 7y agoWhy not just do a full, clean build when you hit strange bug? If your workflow is anything like mine, partial-compile and tools like ccache have saved months of hours I would have wasted waiting for the compiler.
- NullPrefix 7y ago>when you hit strange bug Any bug is strange until you figure it out.
- fanf2 7y agoThere is an interview with Anders Hejlsborg https://youtu.be/wSdV1M7n4gQ https://youtu.be/wSdV1M7n4gQ where he talks about how the Roslyn and TypeScript compilers are designed to support IDEs, with a structure that allows the compiler’s model of the program to be incrementally updated as the code is modified. So there has been some fairly prominent work along the lines you want...
- anaphor 7y agoYou need something like http://adapton.org/ http://adapton.org/ for C/C++ compilers really
- the8472 7y agoAnother waste of resources is that global dead code elimination happens really late (LTO). So functions are compiled and then discarded much later instead of not compiling them in the first place.
- snazz 7y agoI have no idea how they do it, but Common Lisp compilers like SBCL and CCL compile each function incrementally, as you write them, ready to be played with in the REPL. This seems like a complex task in a language that doesn’t have the idea of an “image” that Smalltalk and Lisp do.
- imtringued 7y agoOne can also stop using boost. The last C++ project I worked on spent at least 30% of the compile time parsing boost/filesystem headers. I don't know what type of crackpot designed boost/filesystem but when you trace the includes (and the included includes) then you will see a reasonable amount for your own application. Maybe up to 50 headers that are part of the actual project and then comes a list of hundreds of boost header files. I then proceeded to simply copy the includes of a random .cpp file and put it into a new .cpp file with nothing but includes. I removed all of them one by one except boost/filesystem. Compile times didn't change and if they did then only by a few milliseconds. A single include for boost/filesystem took 2 seconds to compile. I didn't even instantiate any templates in that simplified .cpp file so in practice the overhead is even higher. Isn't that absolutely insane?
- big_chungus 7y agoMuch of boost's compilation time is a result of being so template-heavy. It's still got to parse and do much of the generic processing, even if you don't actually instantiate a template. Whether boost's style is the pinnacle of elegance or ugly as sin is still somewhat in contention, but no one wants to re-write it. I am some what put off as some one who started with C, but most of the C++ people reference it regularly as an excellent example of code style and structure.
- Const-me 7y agoMicrosoft's ATL and WTL are comparable to parts of standard library and boost. They build very fast, despite also based on templates. Often run much faster too, e.g. CAtlMap is typically faster than std::unordered_map by an order of magnitude, due to cache friendliness. The only downside, it's Windows only.
- segfaultbuserr 7y ago> One can also stop using boost. The last C++ project I worked on spent at least 30% of the compile time parsing boost/filesystem headers. I've seen one experimental IRC bot project that heavily uses template expansion, with the asynchronous model and other components in Boost, as a demonstration of advantages of modern C++. The whole compile requires 2 GiB+ memory, if multiprocess make is used, 4-8 GiB! The scale of the project is nowhere close to Chromium, building the Linux kernel doesn't need much RAM either, it's just a IRC bot! But the hardware requirements for using Boost and C++ template expansion is spectacular! The actual binary runs well though, and does not need such hardware. And the developer simply saw it as a small price to pay for using these language features.