6 ms·

Clang vs GCC benchmark on UnrealEngine3

60 points by Aissen 13y ago

mschuster91 13y ago
Seriously, a 70MB binary? Is this all program code or does this include resources?
- Cthulhu_ 13y ago
  Well, judging by the title, it's the Unreal 3 engine; I'd say that would end up being a pretty big binary.
  - mschuster91 13y ago
    The problem is performance: a small binary might end up having lots of its code in L1 (16KByte) / L2 (512 KByte) cache (dec2013 values for i7, taken from https://www.scss.tcd.ie/Jeremy.Jones/CS3021/5%20caches.pdf https://www.scss.tcd.ie/Jeremy.Jones/CS3021/5%20caches.pdf)... but a 70mb binary should produce a lot of cache misses. I wonder what performance penalties arise from that.
    - w0utert 13y ago
      I assume the 70MB binary statically links in all kinds of assets that blow up the file size, static arrays of pre-calculated data etc. There's no way it compiles to 70MB of assembly. The code size would still be massively bigger than the CPU code cache, but the working set that actually matters for performance does not have to have any relation to the size of the total code itself, as long as time-critical sections are small and localized.
      - sharpneli 13y ago
        Considering the massive size increase due to debug symbols I'd guess that a sizable part of the 70MB indeed is assembly, at least 20MB and probably even more. C++ templates, while being absolutely lovely to write and use, tend to do that. Every single different type you instantiate the template into produces a separate compiled copy of the whole template. Write a sorting function and use it for 5 types -> you get the same sort in assembler 5 times with differences that relate to the types they use, alongside with 5 massive symbol names (C++ template symbols tend to be really large).
        exDM69 13y ago
        > Write a sorting function and use it for 5 types -> you get the same sort in assembler 5 times with differences that relate to the types they use, alongside with 5 massive symbol names (C++ template symbols tend to be really large). To be fair, this is also why the performance of C++ std::sort beats C qsort hands down. Calling the comparator function with a function pointer is expensive compared to doing an integer comparison. Additionally, the boundary between translation units (ie. object files) inhibits a lot of optimization that could otherwise take place (and does in the C++ case, when the function calls are inlined). There are also techniques to avoid C++ template bloat, such as type erasure. Although this makes the template code only a thin shim that provides type safety.
        sanxiyn 13y ago
        If functions really are same in assembly (e.g. type sizes are same) LLVM's mergefunc pass actually can merge them, leaving only a single copy. http://llvm.org/docs/Passes.html http://llvm.org/docs/Passes.html
- Negitivefrags 13y ago
  GCC seems to produce fairly fat binaries. Compared to what I'm used to seeing with MSVC at least. The code for our linux game server compiles to an 11MB stripped binary with 608 megs of debug data. The code for the windows game client contains the majority of that same code, but in addition also contains the entire renderer, UI, and a pile of other client side garbage and comes in at only 7MB with 86MB of debug data. The 70MB for UE which would presumably be a much larger code base seems reasonable to me in that context.
rockdoe 13y ago
So GCC takes longer to compile but produces a better optimized binary? That makes the comparison pretty useless as you can probably just reduce the GCC optimization level.
- matthewmacleod 13y ago
  It produces a smaller binary, but that doesn't really give us any information on whether it's a better one. AFAIK Clang and GCC are now roughly neck-and-neck in terms of performance though (at least as of December) - http://www.phoronix.com/scan.php?page=article&item=llvm34_gcc49_compilers http://www.phoronix.com/scan.php?page=article&item=llvm34_gc...
  - cottonseed 13y ago
    GCC produces a slightly smaller binary when debug information is stripped, but LLVM produces a massively smaller binary (half as big) when debug information is included. If you're building debug, why would you care how big the stripped binary is?
    - userbinator 13y ago
      This invites the question of whether GCC is outputting more detailed debugging information than LLVM (which is a good thing), or if it's just being inefficient at it.
      - cottonseed 13y ago
        The g++/gdb experience for C++ has always been pretty bad. Anyone know how lldb compares?
        dmm 13y ago
        In what ways is it bad? It works pretty damn good here.
davidgerard 13y ago
So how did the resulting binaries compare for performance? Compilation happens once, execution many times.
- shoo 13y ago
  except during iterative development, say, where maybe you execute once for test between each edit
  - thesz 13y ago
    You build Unreal Engine from scratch everytime after each edit? I think not. The difference will be between 1s and 2s, given overall difference in speed. I suspect that your time to think about problems will two orders of magnitude more. Mine is - minutes between builds.
    - celebril 13y ago
      How dare you bring sense into this debate and interrupt the LLVM/Clang circlejerk? GCC considered harmful and deprecated! Clang is our only saviour, and these cherry-picked benchmark results clearly proves my point!
tenfingers 13y ago
gcc at -O0 -g has always been decent enough for me, and I never saw any significant difference with clang (either in the past when gcc -O0 used to be /slighly/ slower, or now). Things change with -Ox, but then again, who really cares? Generating optimized binaries is not something I do routinely, except when tuning for performance. When I'm generating a binary for release, I couldn't care less if the project actually took 2x times to compile. In fact, to use feedback-directed optimization (which on some projects may make a marked difference, usually larger then the generated binary at O3 between gcc/clang), you actually have to build it twice, and _very_ few people actually bother. clang used to be much faster at building, but then again the binary was never equally on-par performance wise. As clang is getting better at optimizing, the compilation times are getting more and more similar. This is again, all hardly surprising for me. I _have_ worked on projects using C++ where build times where incredibly slow. They all shared the common attribute of #include <megaheader_with_all_class_definitions> and "using namespace library". For other projects, as long as rebuilding an object takes 2 seconds or 10, does it really make a difference? In those cases, linking actually takes disproportionately longer, to the point that compilation times are insignificant. It's a genuine question.
- cLeEOGPw 13y ago
  2x release build compile time might not be a huge deal to you personally, but over thousands of builds that are happening every day, a 2x decrease in time would mean a lot of saved processing power and electricity. Speed is always important in tools like compiler, so there is never a legitimate excuse for it to be slower than it could be.
  - tenfingers 13y ago
    Cannot deny that. And it's undoubtedly helpful also for continuous integration. Poor performance is never a good excuse. But my perception reading these articles is that the tone is different. There's seems to be a "huge problem with compilation speed in c++", that's also "a detrimental factor to using c++ itself" that I _never_ really experienced, except, like I said, in badly-organized projects. It always felt to me, by programming, that writing and debugging were _always_ the problem, never compilation itself. By reading these articles, it feels like I'm either too slow, or there is some magic that I would need to be doing to use the compiler 20x more than what I'm doing now.
bananas 13y ago
Either way the flat compile time is not important to me even on projects of this size. Most of it is probably going to be linking the .o files on dev builds which is considerably quicker than a full compile cycle. For CI builds ~ 10 mins is nothing (we're currently hitting 3 hours and managing fine with a rats nest of csc and cl compilers being invoked galore). What really brings the shit to the table with clang is simply this: Compiler error messages are beyond awesome as is checker. That is all. Productivity is way up for me. I do my own clang builds of stuff because it's a lot more helpful as a compiler.
Steltek 13y ago
If gcc is producing a final binary that's twice the size of clang's, wouldn't that point to IO being a big factor in compile time? What is this test really trying to determine? How the performance of the core of each compiler impacts productivity? Shouldn't the test parameters be adjusted to real world conditions, as seen by the test creator, then? Like most benchmarks, you could fabricate scenarios that favor one or the other all day long. Some closer to reality, some completely synthetic.
- masklinn 13y ago
  > If gcc is producing a final binary that's twice the size of clang's, wouldn't that point to IO being a big factor in compile time? 250MB of output adding 5mn of compile time would mean 800KiB/s. That's about what a 5400RPM HDD gets on random writes. Possible, but means GCC has an utterly terrible I/O architecture and flushes to disk after more or less every write call. > What is this test really trying to determine? Nothing? It's just trying to infirm or confirm the initial off-hand remark: https://twitter.com/icculus/status/448281959538888704 https://twitter.com/icculus/status/448281959538888704 > Like most benchmarks, you could fabricate scenarios that favor one or the other all day long. Some closer to reality, some completely synthetic. I'm not sure how you can get closer to reality than "I spend my days compiling Goat Simulator with Clang and GCC, let's compile Goat Simulator with Clang and GCC".
- azakai 13y ago
  > If gcc is producing a final binary that's twice the size of clang's, wouldn't that point to IO being a big factor in compile time? Very possibly IO, yes. And it is twice as big for a debug build, whereas a non-debug build is smaller in gcc - 69 MB vs 74 MB in clang. Perhaps more importantly, not all debug builds are equal, so the comparison is not apples-to-apples without saying whether the gcc debug builds contain more useful information. gcc also has various levels of debug info, perhaps a different one would provide less info for a smaller binary that would be more of a parallel to clang.
cpeterso 13y ago
Building Firefox for OS X using clang on my MacBook Pro takes about 8 minutes. Cross-compiling Firefox for Android using gcc 4.7 on the same MacBook Pro takes about 45 minutes. :(
- azakai 13y ago
  But that's not apples-to-apples. How about building for OS X using gcc?
  - cpeterso 13y ago
    Would you expect building for OS X using gcc to be much faster than building for Android? The KLOCs compiled for OS X and Android should be about the same, though the Android build is emitting ARM code and compiling some Java files (but that only takes 1–2 minutes).
    - azakai 13y ago
      Very possible the ARM and x86/x86-64 backends are very different in compilation speed, for example.
JupiterMoon 13y ago
According to this: http://gcc.gnu.org/ml/gcc/2014-04/msg00195.html http://gcc.gnu.org/ml/gcc/2014-04/msg00195.html these benchmarks are already out of date...

60 points by Aissen 13y ago

mschuster91 13y ago

Seriously, a 70MB binary? Is this all program code or does this include resources?

Cthulhu_ 13y ago
Well, judging by the title, it's the Unreal 3 engine; I'd say that would end up being a pretty big binary.
- mschuster91 13y ago
  The problem is performance: a small binary might end up having lots of its code in L1 (16KByte) / L2 (512 KByte) cache (dec2013 values for i7, taken from https://www.scss.tcd.ie/Jeremy.Jones/CS3021/5%20caches.pdf https://www.scss.tcd.ie/Jeremy.Jones/CS3021/5%20caches.pdf)... but a 70mb binary should produce a lot of cache misses. I wonder what performance penalties arise from that.
  - w0utert 13y ago
    I assume the 70MB binary statically links in all kinds of assets that blow up the file size, static arrays of pre-calculated data etc. There's no way it compiles to 70MB of assembly. The code size would still be massively bigger than the CPU code cache, but the working set that actually matters for performance does not have to have any relation to the size of the total code itself, as long as time-critical sections are small and localized.
    - sharpneli 13y ago
      Considering the massive size increase due to debug symbols I'd guess that a sizable part of the 70MB indeed is assembly, at least 20MB and probably even more. C++ templates, while being absolutely lovely to write and use, tend to do that. Every single different type you instantiate the template into produces a separate compiled copy of the whole template. Write a sorting function and use it for 5 types -> you get the same sort in assembler 5 times with differences that relate to the types they use, alongside with 5 massive symbol names (C++ template symbols tend to be really large).
      - exDM69 13y ago
        > Write a sorting function and use it for 5 types -> you get the same sort in assembler 5 times with differences that relate to the types they use, alongside with 5 massive symbol names (C++ template symbols tend to be really large). To be fair, this is also why the performance of C++ std::sort beats C qsort hands down. Calling the comparator function with a function pointer is expensive compared to doing an integer comparison. Additionally, the boundary between translation units (ie. object files) inhibits a lot of optimization that could otherwise take place (and does in the C++ case, when the function calls are inlined). There are also techniques to avoid C++ template bloat, such as type erasure. Although this makes the template code only a thin shim that provides type safety.
      - sanxiyn 13y ago
        If functions really are same in assembly (e.g. type sizes are same) LLVM's mergefunc pass actually can merge them, leaving only a single copy. http://llvm.org/docs/Passes.html http://llvm.org/docs/Passes.html
Negitivefrags 13y ago
GCC seems to produce fairly fat binaries. Compared to what I'm used to seeing with MSVC at least. The code for our linux game server compiles to an 11MB stripped binary with 608 megs of debug data. The code for the windows game client contains the majority of that same code, but in addition also contains the entire renderer, UI, and a pile of other client side garbage and comes in at only 7MB with 86MB of debug data. The 70MB for UE which would presumably be a much larger code base seems reasonable to me in that context.

rockdoe 13y ago

So GCC takes longer to compile but produces a better optimized binary? That makes the comparison pretty useless as you can probably just reduce the GCC optimization level.

matthewmacleod 13y ago
It produces a smaller binary, but that doesn't really give us any information on whether it's a better one. AFAIK Clang and GCC are now roughly neck-and-neck in terms of performance though (at least as of December) - http://www.phoronix.com/scan.php?page=article&item=llvm34_gcc49_compilers http://www.phoronix.com/scan.php?page=article&item=llvm34_gc...
- cottonseed 13y ago
  GCC produces a slightly smaller binary when debug information is stripped, but LLVM produces a massively smaller binary (half as big) when debug information is included. If you're building debug, why would you care how big the stripped binary is?
  - userbinator 13y ago
    This invites the question of whether GCC is outputting more detailed debugging information than LLVM (which is a good thing), or if it's just being inefficient at it.
    - cottonseed 13y ago
      The g++/gdb experience for C++ has always been pretty bad. Anyone know how lldb compares?
      - dmm 13y ago
        In what ways is it bad? It works pretty damn good here.

davidgerard 13y ago

So how did the resulting binaries compare for performance? Compilation happens once, execution many times.

shoo 13y ago
except during iterative development, say, where maybe you execute once for test between each edit
- thesz 13y ago
  You build Unreal Engine from scratch everytime after each edit? I think not. The difference will be between 1s and 2s, given overall difference in speed. I suspect that your time to think about problems will two orders of magnitude more. Mine is - minutes between builds.
  - celebril 13y ago
    How dare you bring sense into this debate and interrupt the LLVM/Clang circlejerk? GCC considered harmful and deprecated! Clang is our only saviour, and these cherry-picked benchmark results clearly proves my point!

tenfingers 13y ago

gcc at -O0 -g has always been decent enough for me, and I never saw any significant difference with clang (either in the past when gcc -O0 used to be /slighly/ slower, or now). Things change with -Ox, but then again, who really cares? Generating optimized binaries is not something I do routinely, except when tuning for performance. When I'm generating a binary for release, I couldn't care less if the project actually took 2x times to compile. In fact, to use feedback-directed optimization (which on some projects may make a marked difference, usually larger then the generated binary at O3 between gcc/clang), you actually have to build it twice, and _very_ few people actually bother. clang used to be much faster at building, but then again the binary was never equally on-par performance wise. As clang is getting better at optimizing, the compilation times are getting more and more similar. This is again, all hardly surprising for me. I _have_ worked on projects using C++ where build times where incredibly slow. They all shared the common attribute of #include <megaheader_with_all_class_definitions> and "using namespace library". For other projects, as long as rebuilding an object takes 2 seconds or 10, does it really make a difference? In those cases, linking actually takes disproportionately longer, to the point that compilation times are insignificant. It's a genuine question.

cLeEOGPw 13y ago
2x release build compile time might not be a huge deal to you personally, but over thousands of builds that are happening every day, a 2x decrease in time would mean a lot of saved processing power and electricity. Speed is always important in tools like compiler, so there is never a legitimate excuse for it to be slower than it could be.
- tenfingers 13y ago
  Cannot deny that. And it's undoubtedly helpful also for continuous integration. Poor performance is never a good excuse. But my perception reading these articles is that the tone is different. There's seems to be a "huge problem with compilation speed in c++", that's also "a detrimental factor to using c++ itself" that I _never_ really experienced, except, like I said, in badly-organized projects. It always felt to me, by programming, that writing and debugging were _always_ the problem, never compilation itself. By reading these articles, it feels like I'm either too slow, or there is some magic that I would need to be doing to use the compiler 20x more than what I'm doing now.

bananas 13y ago

Either way the flat compile time is not important to me even on projects of this size. Most of it is probably going to be linking the .o files on dev builds which is considerably quicker than a full compile cycle. For CI builds ~ 10 mins is nothing (we're currently hitting 3 hours and managing fine with a rats nest of csc and cl compilers being invoked galore). What really brings the shit to the table with clang is simply this: Compiler error messages are beyond awesome as is checker. That is all. Productivity is way up for me. I do my own clang builds of stuff because it's a lot more helpful as a compiler.

Steltek 13y ago

If gcc is producing a final binary that's twice the size of clang's, wouldn't that point to IO being a big factor in compile time? What is this test really trying to determine? How the performance of the core of each compiler impacts productivity? Shouldn't the test parameters be adjusted to real world conditions, as seen by the test creator, then? Like most benchmarks, you could fabricate scenarios that favor one or the other all day long. Some closer to reality, some completely synthetic.

masklinn 13y ago
> If gcc is producing a final binary that's twice the size of clang's, wouldn't that point to IO being a big factor in compile time? 250MB of output adding 5mn of compile time would mean 800KiB/s. That's about what a 5400RPM HDD gets on random writes. Possible, but means GCC has an utterly terrible I/O architecture and flushes to disk after more or less every write call. > What is this test really trying to determine? Nothing? It's just trying to infirm or confirm the initial off-hand remark: https://twitter.com/icculus/status/448281959538888704 https://twitter.com/icculus/status/448281959538888704 > Like most benchmarks, you could fabricate scenarios that favor one or the other all day long. Some closer to reality, some completely synthetic. I'm not sure how you can get closer to reality than "I spend my days compiling Goat Simulator with Clang and GCC, let's compile Goat Simulator with Clang and GCC".
azakai 13y ago
> If gcc is producing a final binary that's twice the size of clang's, wouldn't that point to IO being a big factor in compile time? Very possibly IO, yes. And it is twice as big for a debug build, whereas a non-debug build is smaller in gcc - 69 MB vs 74 MB in clang. Perhaps more importantly, not all debug builds are equal, so the comparison is not apples-to-apples without saying whether the gcc debug builds contain more useful information. gcc also has various levels of debug info, perhaps a different one would provide less info for a smaller binary that would be more of a parallel to clang.

cpeterso 13y ago

Building Firefox for OS X using clang on my MacBook Pro takes about 8 minutes. Cross-compiling Firefox for Android using gcc 4.7 on the same MacBook Pro takes about 45 minutes. :(

azakai 13y ago
But that's not apples-to-apples. How about building for OS X using gcc?
- cpeterso 13y ago
  Would you expect building for OS X using gcc to be much faster than building for Android? The KLOCs compiled for OS X and Android should be about the same, though the Android build is emitting ARM code and compiling some Java files (but that only takes 1–2 minutes).
  - azakai 13y ago
    Very possible the ARM and x86/x86-64 backends are very different in compilation speed, for example.

JupiterMoon 13y ago

According to this: http://gcc.gnu.org/ml/gcc/2014-04/msg00195.html http://gcc.gnu.org/ml/gcc/2014-04/msg00195.html these benchmarks are already out of date...