6 ms·

I wish someone would pick up the zapcc project. Compilers do an insane amount of duplicated and redundant work and there's tremendous potential for speeding up

I wish someone would pick up the zapcc project. Compilers do an insane amount of duplicated and redundant work and there's tremendous potential for speeding up C++ builds if you are willing to rethink how the compiler works a little bit. https://github.com/yrnkrn/zapcc https://github.com/yrnkrn/zapcc

wheybags 6y ago
I work on the video game factorio, which is a c++ project. On a 9900k a rebuild takes about a minute, so it's pretty sizeable but not something ridiculous like oracle db or unreal engine. I tried using zapcc on it, and it was a complete failure. I don't have measurements to hand, but iirc it was actually slower than stock clang. I tested it on a threadripper 2950x with 64gb of ram, running Debian.
- modeless 6y ago
  I've had good results on some of my own projects. The current implementation certainly has limitations, but it's really early and the potential is huge. For example, because it can cache at a sub-compilation-unit granularity it can make rebuilds of a single file faster, which no distributed build tool or cache can do.
- MontagFTB 6y ago
  Thank you for Factorio!
- jiggawatts 6y ago
  I just installed the game again to play the v1.0 release. It looks and sounds very good! Congratulations on making it to the release milestone after many years of hard work. As a developer, I must say that I'm deeply impressed by the simulation complexity and incredible scalability. I keep telling people that Factorio is the most complex train simulator game that I've ever played. But it isn't a train simulator. It's a factory game. The trains are optional. That's crazy to me, that a subset of a game can be more complex than entire games that "specialise" in the topic.
  - secondcoming 6y ago
    How do I make curved tracks?
    - secondcoming 6y ago
      This was genuine question BTW.
      - jiggawatts 6y ago
        There is a tutorial in-game! After placing a track, click the end of the track and hold shift to use the "placement planner". Helps work out the curves and things.
    - ynik 6y ago
      The first bit of track you place is always straight; but if you extend the rail (by clicking on the green arrow of the first piece of rail) you can place curves. https://wiki.factorio.com/Rail_planner https://wiki.factorio.com/Rail_planner
- stephc_int13 6y ago
  Have you tried switching to a unity build? Or removing templates?
  - wheybags 6y ago
    We do use a unity build yeah. Without it would be way way longer, maybe 5 minutes+. As for removing templates, there are a few places where we specifically opted to use custom macros instead of std::variant, for compile time reasons, but mostly we do use templates quite a lot.
    - modeless 6y ago
      A unity build gives some of the same build time benefits as zapcc for full rebuilds, but requires more work and compromises in code structure, and also increases your single-file incremental build time. If you're already using a unity build then I'd expect zapcc to not speed up full builds much if at all. I'd be interested to see if zapcc could speed up your non-unity build to be close to your unity build, without requiring the same compromises. I'd also be interested to see if zapcc would speed up single-file incremental builds in your unity build. Ultimately I can't recommend zapcc for general use as it's a fork of clang that's not being regularly rebased anymore, so it's getting out of date. That's why I wish someone would pick it up again.
- jnxx 6y ago
  How does ccache compare to it? https://ccache.dev/ https://ccache.dev/
  - wheybags 6y ago
    That is what I normally use. ccache is great! With a unity build its usefulness is a bit diminished, but it's still great for switching back and forth between branches.
chubot 6y ago
Is this like llbuild? https://www.youtube.com/watch?v=b_T-eCToX1I https://www.youtube.com/watch?v=b_T-eCToX1I https://github.com/apple/swift-llbuild https://github.com/apple/swift-llbuild I remember watching the talk awhile ago, but haven't kept up with its progress. It's about using LLVM as a library to eliminate duplicate work, as far as I remember. I think it was for C++ as well as Swift.
usrnm 6y ago
> Compilers do an insane amount of duplicated and redundant work This is one of the problems that should be solved by modules in the near future.
- modeless 6y ago
  I hope so but last time I looked it seemed like the preliminary implementations didn't improve build times much in practice.
  - boris 6y ago
    Real-world usage of Clang modules (by Google) and preliminary/synthetic benchmarks of C++20 modules in GCC both show about 3x speedup. This might not be "much" for everyone, buy I will take it.
    - jnxx 6y ago
      > benchmarks of C++20 modules in GCC both show about 3x speedup compared to what? The worst possible case? Code in weakly coupled modules? Using PIMPL idiom? Unity builds? Using ccache in incremental builds? Especially ccache is a relevant comparison, because what to a developer is of far more interest than a full build is an incremental build with a few files changed. Using modules requires to scan the source code in order to determine dependencies, so it can be slower than an incremental build using other tools. And in addition to this, C++ compilation with include headers is an embarrassingly parallel problem which means speed advantages from using multi-core systems will probably very relevant in the, say, ten years time in which C++ modules will be somewhat standardized in practice. Not to say they can't help, but in any other large software project one would need to show that something is useful before adding a big change.
- jnxx 6y ago
  See also: Are modules fast? https://bfgroup.github.io/cpp_tooling_stats/modules/modules_perf_D1441R1.html https://bfgroup.github.io/cpp_tooling_stats/modules/modules_... One reason why this might not be so simple is that what is often measured as a potential for speed improvement is I/O, as in "lines of code a compiler needs to read to process a translation unit". However what happens is that C++ compilers are usually CPU-bound, not I/O bound. What happens also is that C++, very differently from idiomatic C, Pascal, Modula, and their successors, has no clear separation between declaration and use of an interface, as not functions, but classes form the main interface, and what defines a class is the signature of its member functions and its members, and to know the size of the class, the size of its members need to be known, which has the consequence that the C++ compiler needs to read recursively everything which the class is defined with. This is a big difference to modula etc. Also, of course the reading of data can be cached, but the question is how much the expansion of the read text can be cached - for example some will have different semantics if the DEBUG macro is set, or some WIN_xyz macro. As far as I understand, compilers supporting C++ modules will only cache the latest expansion. Also, the build system with modules is likely to become more complex, with additional dependencies. I do not know whether this is still actual in every aspect (it could be that what is allowed to go into a module header has been narrowed down a bit), but this post : https://vector-of-bool.github.io/2019/01/27/modules-doa.html https://vector-of-bool.github.io/2019/01/27/modules-doa.html explains at least some of the topics involved, and not all of them are trivial.

wheybags 6y ago

I work on the video game factorio, which is a c++ project. On a 9900k a rebuild takes about a minute, so it's pretty sizeable but not something ridiculous like oracle db or unreal engine. I tried using zapcc on it, and it was a complete failure. I don't have measurements to hand, but iirc it was actually slower than stock clang. I tested it on a threadripper 2950x with 64gb of ram, running Debian.

modeless 6y ago
I've had good results on some of my own projects. The current implementation certainly has limitations, but it's really early and the potential is huge. For example, because it can cache at a sub-compilation-unit granularity it can make rebuilds of a single file faster, which no distributed build tool or cache can do.
MontagFTB 6y ago
Thank you for Factorio!
jiggawatts 6y ago
I just installed the game again to play the v1.0 release. It looks and sounds very good! Congratulations on making it to the release milestone after many years of hard work. As a developer, I must say that I'm deeply impressed by the simulation complexity and incredible scalability. I keep telling people that Factorio is the most complex train simulator game that I've ever played. But it isn't a train simulator. It's a factory game. The trains are optional. That's crazy to me, that a subset of a game can be more complex than entire games that "specialise" in the topic.
- secondcoming 6y ago
  How do I make curved tracks?
  - secondcoming 6y ago
    This was genuine question BTW.
    - jiggawatts 6y ago
      There is a tutorial in-game! After placing a track, click the end of the track and hold shift to use the "placement planner". Helps work out the curves and things.
  - ynik 6y ago
    The first bit of track you place is always straight; but if you extend the rail (by clicking on the green arrow of the first piece of rail) you can place curves. https://wiki.factorio.com/Rail_planner https://wiki.factorio.com/Rail_planner
stephc_int13 6y ago
Have you tried switching to a unity build? Or removing templates?
- wheybags 6y ago
  We do use a unity build yeah. Without it would be way way longer, maybe 5 minutes+. As for removing templates, there are a few places where we specifically opted to use custom macros instead of std::variant, for compile time reasons, but mostly we do use templates quite a lot.
  - modeless 6y ago
    A unity build gives some of the same build time benefits as zapcc for full rebuilds, but requires more work and compromises in code structure, and also increases your single-file incremental build time. If you're already using a unity build then I'd expect zapcc to not speed up full builds much if at all. I'd be interested to see if zapcc could speed up your non-unity build to be close to your unity build, without requiring the same compromises. I'd also be interested to see if zapcc would speed up single-file incremental builds in your unity build. Ultimately I can't recommend zapcc for general use as it's a fork of clang that's not being regularly rebased anymore, so it's getting out of date. That's why I wish someone would pick it up again.
jnxx 6y ago
How does ccache compare to it? https://ccache.dev/ https://ccache.dev/
- wheybags 6y ago
  That is what I normally use. ccache is great! With a unity build its usefulness is a bit diminished, but it's still great for switching back and forth between branches.

chubot 6y ago

Is this like llbuild? https://www.youtube.com/watch?v=b_T-eCToX1I https://www.youtube.com/watch?v=b_T-eCToX1I https://github.com/apple/swift-llbuild https://github.com/apple/swift-llbuild I remember watching the talk awhile ago, but haven't kept up with its progress. It's about using LLVM as a library to eliminate duplicate work, as far as I remember. I think it was for C++ as well as Swift.

usrnm 6y ago

> Compilers do an insane amount of duplicated and redundant work This is one of the problems that should be solved by modules in the near future.

modeless 6y ago
I hope so but last time I looked it seemed like the preliminary implementations didn't improve build times much in practice.
- boris 6y ago
  Real-world usage of Clang modules (by Google) and preliminary/synthetic benchmarks of C++20 modules in GCC both show about 3x speedup. This might not be "much" for everyone, buy I will take it.
  - jnxx 6y ago
    > benchmarks of C++20 modules in GCC both show about 3x speedup compared to what? The worst possible case? Code in weakly coupled modules? Using PIMPL idiom? Unity builds? Using ccache in incremental builds? Especially ccache is a relevant comparison, because what to a developer is of far more interest than a full build is an incremental build with a few files changed. Using modules requires to scan the source code in order to determine dependencies, so it can be slower than an incremental build using other tools. And in addition to this, C++ compilation with include headers is an embarrassingly parallel problem which means speed advantages from using multi-core systems will probably very relevant in the, say, ten years time in which C++ modules will be somewhat standardized in practice. Not to say they can't help, but in any other large software project one would need to show that something is useful before adding a big change.
jnxx 6y ago
See also: Are modules fast? https://bfgroup.github.io/cpp_tooling_stats/modules/modules_perf_D1441R1.html https://bfgroup.github.io/cpp_tooling_stats/modules/modules_... One reason why this might not be so simple is that what is often measured as a potential for speed improvement is I/O, as in "lines of code a compiler needs to read to process a translation unit". However what happens is that C++ compilers are usually CPU-bound, not I/O bound. What happens also is that C++, very differently from idiomatic C, Pascal, Modula, and their successors, has no clear separation between declaration and use of an interface, as not functions, but classes form the main interface, and what defines a class is the signature of its member functions and its members, and to know the size of the class, the size of its members need to be known, which has the consequence that the C++ compiler needs to read recursively everything which the class is defined with. This is a big difference to modula etc. Also, of course the reading of data can be cached, but the question is how much the expansion of the read text can be cached - for example some will have different semantics if the DEBUG macro is set, or some WIN_xyz macro. As far as I understand, compilers supporting C++ modules will only cache the latest expansion. Also, the build system with modules is likely to become more complex, with additional dependencies. I do not know whether this is still actual in every aspect (it could be that what is allowed to go into a module header has been narrowed down a bit), but this post : https://vector-of-bool.github.io/2019/01/27/modules-doa.html https://vector-of-bool.github.io/2019/01/27/modules-doa.html explains at least some of the topics involved, and not all of them are trivial.