5 ms·

GCC x86 Performance Hints (2012)

65 points by ZnZirconium 6y ago

glouwbug 6y ago
-flto is great if your project is less than 15,0000 lines. Otherwise, -Ofast in conjunction with -flto creates a single threaded link time step that's longer than half a minute, and it only goes up from there
- vii 6y ago
  The gold linker and now lld can do LTO pretty fast across huge codebases: this breaks down the techniques used https://archive.fosdem.org/2019/schedule/event/llvm_lld/ https://archive.fosdem.org/2019/schedule/event/llvm_lld/ Curious that Intel seems to recommend dynamic linking to get architecture specific libc function implementations. Dynamic linking has considerable overhead.
  - btown 6y ago
    I’m curious why this is necessary nowadays. Why not just ship a multi architecture static libc that chooses implementation at runtime? Branch prediction should make that practically zero cost, and with cache sizes at all levels I doubt binary size would have an impact. Are there other reasons?
    - nicoburns 6y ago
      > Are there other reasons? I think some systems don't have a stable kernel interface. The system-provided libc is that interface. On Linux I believe that this is possible (and fairly common using the MUSL libc), but that glibc doesn't support static linking and many people want to use glibc for performance and compatibility reasons.
      - roblabla 6y ago
        > I think some systems don't have a stable kernel interface. The system-provided libc is that interface. In fact, a lot of systems don't have a stable kernel interface, Linux is the exceptions here. In Microsoft Windows, one must link against Kernel32.dll to communicate with the kernel. On OpenBSD, the libc provides the stable syscall interface (and in fact the kernel will refuse syscalls from outside the libc as a security measure, see [0]). MacOS and Solaris are two other OSes where, if I understand correctly, syscall ABI is not guaranteed and must go through a common library. Go used to embed syscalls, and ran into a lot of problems because of this. [0]: https://marc.info/?l=openbsd-tech&m=157488907117170&w=2 https://marc.info/?l=openbsd-tech&m=157488907117170&w=2
        voldacar 6y ago
        I get that openbsd is obsessed with security, but this still makes me a little sad. C is very old, and it should be possible to create programs and programming languages that have no idea what C is or what libc is. Forcing all programs to communicate with the OS via libc seems wrong
        roblabla 6y ago
        On one hand, from an idealistic point of view, I agree, and am a very, very big believer of having actual, carefully designed ABIs for kernel (and inter-process) communications. On the other hand, the structures required to do syscalls via libc/kernel32 are generally simple enough that it's not a huge deal, and the difference between doing a raw syscall and doing a function call is unlikely to actually matter. Fun fact/pet peeve: on android, if you want to communicate with most services, you need to go through Binder. While the low-level Binder ABI is stable, the services written on top of it aren't, and often change in backwards-incompatible ways. This includes core services like SurfaceFlinger (necessary to draw on the screen). This means that it's generally impossible to create purely native software on android - you always have to call into Java to talk to those services.
        moonchild 6y ago
        > In Microsoft Windows, one must link against Kernel32.dll Not kernel32, but ntdll. FreeBSD is another OS that provides a stable kernel ABI.
    - xxpor 6y ago
      The L1 icache on most Arm chips is still fairly tiny. For example on the Snapdragon 865, it's only 64k. An i7-8700k (already 3 years old) on the other hand has 192k.
      - kjs3 6y ago
        That's pretty meaningless. The effect of cache is very processor and ISA specific and has a log tail of value (as in, depending on the processor, X amount of cache may get you a 50% speedup, but 2 times X might only give you 60% so is the cost/benefit trade-off worth it). It's plausible that the i7 needs 3 times the cache to get the same benefit and the 64k in the 865. N.B. I don't know for sure that that is the case.
  - gnufx 6y ago
    Dynamic linking has considerable advantages in the facilities it provides, which seem to outweigh any efficiency concerns, e.g. in HPC.
- google234123 6y ago
  Google has FB do LTO on their large code bases.
  - sp332 6y ago
    FB?
    - woadwarrior01 6y ago
      OP probably meant Google and FB. LTO was used extensively at FB when I worked there, a couple of years ago.
- jhallenworld 6y ago
  -flto is great at the other end of the spectrum- tiny systems. With "-Os -lto", much unused code from newlib is deleted, allowing the rest to fit in a tiny Cortex-M0 system. In the old days, each function of the C library was in its own object file in the library, but these days this appears to not be the case. But -lto gives the same effect.
  - bigdict 6y ago
    I use it too for embedded. Functions get inlined, fewer registers get saved unnecessarily in functions. I also use “-fdata-sections -ffunction-sections” with “-Wl,--gc-sections” to prune unused sections. There is some evidence this can actually increase your code size sometimes: https://stackoverflow.com/a/29951897 https://stackoverflow.com/a/29951897.
- abra529 6y ago
  > 15,0000 lines Did you mean 150,000 or 15,000?
  - glouwbug 6y ago
    15,000, because I'm too poor to buy a new laptop
ilaksh 6y ago
It feels like they are actually trying to convince me to finally switch to Clang or something. I mean, I have actually still never used Clang, but even before this list, I was aware that there were quite a few gcc defaults that were ridiculous. But every gcc article is about how there are even more ways that gcc doesn't actually work sensibly unless you know the trick. I mean, why doesn't it say somewhere in the help next to the optimization stuff that you might want to consider specifying the architecture?
- lathiat 6y ago
  To be fair, it is from 2012
- vii 6y ago
  clang also doesn't assume -march=native - just like gcc it makes binaries that can run on machines of the same architecture rather than specialising to the current processor's features. You can see the flags it's using by comparing clang -E -v - </dev/null 2>&1 and clang -march=native -E -v - </dev/null 2>&1 The second output will probably show a lot of important features enabled - for example +sse4.2 -target-feature +avx2 -target-feature +fxsr -target-feature -wbnoinvd -target-feature +sse -target-feature +lzcnt -target-feature +pclmul -target-feature -prefetchwt1 -target-feature +f16c -target-feature +ssse3 -target-feature -sgx -target-feature -shstk -target-feature +cmov
- canarypilot 6y ago
  Clang works much the same way by default. All compilers have to pick a baseline, and assuming it is arch_of(your_core) is a recipe for disaster if you are compiling on your high-end machine for release to a broad user base. For many users of GCC the intended user base will be “the same as my current distribution” and so GCC can be configured using —-with-{cpu, arch, schedule, tune} depending on the architecture to set a default in line with user/distro expectations.
  - rndgermandude 6y ago
    Exactly. What you do not want to happen is that your program crashes with fancy "illegal instruction" errors on older CPUs that do not have the newest fancy features yet, but are still in use by a large chunk of your (paying) users. Performance critical programs usually deal with that by either providing multiple builds targeting different CPU features/CPUs, or let the user compile themselves from source with the right flags, or have CPU runtime detection and provide alternative versions of a few important performance critical functions for different CPU feature sets (e.g. browsers, ffmpeg, glibc, various VMs/runtimes like Java Hotspot or dotnet, do that) while the majority of code is still compiled for a lowest common subset of CPU features. Of course, languages that run on (usually) JIT-ed VMs/runtimes have a bit of an advantage here, as the actual machine code is generated from source code or byte code only at runtime, at which point it is clear what kind of CPU is underneath the program. They can - but not always do - implement optimized JITting depending on the CPU features. (of course, every language/VM/runtime comes with its own set of pros and cons and there is no silver bullet). To make matters even more complicated: compiling code to use the newest CPU features or newest optimization techniques will not mean it will actually run faster. E.g. AVX512 may actually slow down your code (when multi-threaded) on many CPUs[1]. Or heavily "optimized" code may become larger in machine code, to the point where your "unoptimized" code may run faster because it fits in the CPU cache(s) properly while the "optimized" version does not. "-Os" optimized code may run faster than "-Ofast" optimized code for this matter. Or it may not. Depending on the actual code. I remember compiling ffmpeg and libx264 myself a bunch of years ago, with the "best" flags for my system, starting with "-march=" and "-Ofast" of course, thinking I am a tough skillful super geek now. Imagine my surprise when I tested the performance against a default ffmpeg build and my optimized build was 2-5% slower. [1] https://blog.cloudflare.com/on-the-dangers-of-intels-frequency-scaling/ https://blog.cloudflare.com/on-the-dangers-of-intels-frequen...
known 6y ago
In my .bashrc I use export CFLAGS='-O2 -march=native -mfpmath=sse -fopenmp -fomit-frame-pointer -pipe -fno-unwind-tables -fno-asynchronous-unwind-tables'
- kanox 6y ago
  Why would you put something like that in bashrc?!
Supersaiyan_IV 6y ago
This article is way too outdated. Also, who the heck uses -flto on GCC 4.7.1? LTO is experimental in GCC 4.7.1 and will make any application nigh-impossible to debug. Here's something more relevant to modern times: http://hubicka.blogspot.com/2019/05/gcc-9-link-time-and-inter-procedural.html http://hubicka.blogspot.com/2019/05/gcc-9-link-time-and-inte...
- gnufx 6y ago
  Turning on LTO in Fedora hasn't been entirely painless despite SUSE's pioneering.
vsskanth 6y ago
I wonder if there is an equivalent for MSVC and Clang Compilers. Would be nice if there was a good guide somewhere on flags to use (and their trade-offs) for fast floating point performance for numerical intensive programs.

65 points by ZnZirconium 6y ago

-flto is great if your project is less than 15,0000 lines. Otherwise, -Ofast in conjunction with -flto creates a single threaded link time step that's longer than half a minute, and it only goes up from there

vii 6y ago
The gold linker and now lld can do LTO pretty fast across huge codebases: this breaks down the techniques used https://archive.fosdem.org/2019/schedule/event/llvm_lld/ https://archive.fosdem.org/2019/schedule/event/llvm_lld/ Curious that Intel seems to recommend dynamic linking to get architecture specific libc function implementations. Dynamic linking has considerable overhead.
- btown 6y ago
  I’m curious why this is necessary nowadays. Why not just ship a multi architecture static libc that chooses implementation at runtime? Branch prediction should make that practically zero cost, and with cache sizes at all levels I doubt binary size would have an impact. Are there other reasons?
  - nicoburns 6y ago
    > Are there other reasons? I think some systems don't have a stable kernel interface. The system-provided libc is that interface. On Linux I believe that this is possible (and fairly common using the MUSL libc), but that glibc doesn't support static linking and many people want to use glibc for performance and compatibility reasons.
    - roblabla 6y ago
      > I think some systems don't have a stable kernel interface. The system-provided libc is that interface. In fact, a lot of systems don't have a stable kernel interface, Linux is the exceptions here. In Microsoft Windows, one must link against Kernel32.dll to communicate with the kernel. On OpenBSD, the libc provides the stable syscall interface (and in fact the kernel will refuse syscalls from outside the libc as a security measure, see [0]). MacOS and Solaris are two other OSes where, if I understand correctly, syscall ABI is not guaranteed and must go through a common library. Go used to embed syscalls, and ran into a lot of problems because of this. [0]: https://marc.info/?l=openbsd-tech&m=157488907117170&w=2 https://marc.info/?l=openbsd-tech&m=157488907117170&w=2
      - voldacar 6y ago
        I get that openbsd is obsessed with security, but this still makes me a little sad. C is very old, and it should be possible to create programs and programming languages that have no idea what C is or what libc is. Forcing all programs to communicate with the OS via libc seems wrong
        roblabla 6y ago
        On one hand, from an idealistic point of view, I agree, and am a very, very big believer of having actual, carefully designed ABIs for kernel (and inter-process) communications. On the other hand, the structures required to do syscalls via libc/kernel32 are generally simple enough that it's not a huge deal, and the difference between doing a raw syscall and doing a function call is unlikely to actually matter. Fun fact/pet peeve: on android, if you want to communicate with most services, you need to go through Binder. While the low-level Binder ABI is stable, the services written on top of it aren't, and often change in backwards-incompatible ways. This includes core services like SurfaceFlinger (necessary to draw on the screen). This means that it's generally impossible to create purely native software on android - you always have to call into Java to talk to those services.
      - moonchild 6y ago
        > In Microsoft Windows, one must link against Kernel32.dll Not kernel32, but ntdll. FreeBSD is another OS that provides a stable kernel ABI.
  - xxpor 6y ago
    The L1 icache on most Arm chips is still fairly tiny. For example on the Snapdragon 865, it's only 64k. An i7-8700k (already 3 years old) on the other hand has 192k.
    - kjs3 6y ago
      That's pretty meaningless. The effect of cache is very processor and ISA specific and has a log tail of value (as in, depending on the processor, X amount of cache may get you a 50% speedup, but 2 times X might only give you 60% so is the cost/benefit trade-off worth it). It's plausible that the i7 needs 3 times the cache to get the same benefit and the 64k in the 865. N.B. I don't know for sure that that is the case.
- gnufx 6y ago
  Dynamic linking has considerable advantages in the facilities it provides, which seem to outweigh any efficiency concerns, e.g. in HPC.
google234123 6y ago
Google has FB do LTO on their large code bases.
- sp332 6y ago
  FB?
  - woadwarrior01 6y ago
    OP probably meant Google and FB. LTO was used extensively at FB when I worked there, a couple of years ago.
jhallenworld 6y ago
-flto is great at the other end of the spectrum- tiny systems. With "-Os -lto", much unused code from newlib is deleted, allowing the rest to fit in a tiny Cortex-M0 system. In the old days, each function of the C library was in its own object file in the library, but these days this appears to not be the case. But -lto gives the same effect.
- bigdict 6y ago
  I use it too for embedded. Functions get inlined, fewer registers get saved unnecessarily in functions. I also use “-fdata-sections -ffunction-sections” with “-Wl,--gc-sections” to prune unused sections. There is some evidence this can actually increase your code size sometimes: https://stackoverflow.com/a/29951897 https://stackoverflow.com/a/29951897.
abra529 6y ago
> 15,0000 lines Did you mean 150,000 or 15,000?
- glouwbug 6y ago
  15,000, because I'm too poor to buy a new laptop

ilaksh 6y ago

It feels like they are actually trying to convince me to finally switch to Clang or something. I mean, I have actually still never used Clang, but even before this list, I was aware that there were quite a few gcc defaults that were ridiculous. But every gcc article is about how there are even more ways that gcc doesn't actually work sensibly unless you know the trick. I mean, why doesn't it say somewhere in the help next to the optimization stuff that you might want to consider specifying the architecture?

lathiat 6y ago
To be fair, it is from 2012
vii 6y ago
clang also doesn't assume -march=native - just like gcc it makes binaries that can run on machines of the same architecture rather than specialising to the current processor's features. You can see the flags it's using by comparing clang -E -v - </dev/null 2>&1 and clang -march=native -E -v - </dev/null 2>&1 The second output will probably show a lot of important features enabled - for example +sse4.2 -target-feature +avx2 -target-feature +fxsr -target-feature -wbnoinvd -target-feature +sse -target-feature +lzcnt -target-feature +pclmul -target-feature -prefetchwt1 -target-feature +f16c -target-feature +ssse3 -target-feature -sgx -target-feature -shstk -target-feature +cmov
canarypilot 6y ago
Clang works much the same way by default. All compilers have to pick a baseline, and assuming it is arch_of(your_core) is a recipe for disaster if you are compiling on your high-end machine for release to a broad user base. For many users of GCC the intended user base will be “the same as my current distribution” and so GCC can be configured using —-with-{cpu, arch, schedule, tune} depending on the architecture to set a default in line with user/distro expectations.
- rndgermandude 6y ago
  Exactly. What you do not want to happen is that your program crashes with fancy "illegal instruction" errors on older CPUs that do not have the newest fancy features yet, but are still in use by a large chunk of your (paying) users. Performance critical programs usually deal with that by either providing multiple builds targeting different CPU features/CPUs, or let the user compile themselves from source with the right flags, or have CPU runtime detection and provide alternative versions of a few important performance critical functions for different CPU feature sets (e.g. browsers, ffmpeg, glibc, various VMs/runtimes like Java Hotspot or dotnet, do that) while the majority of code is still compiled for a lowest common subset of CPU features. Of course, languages that run on (usually) JIT-ed VMs/runtimes have a bit of an advantage here, as the actual machine code is generated from source code or byte code only at runtime, at which point it is clear what kind of CPU is underneath the program. They can - but not always do - implement optimized JITting depending on the CPU features. (of course, every language/VM/runtime comes with its own set of pros and cons and there is no silver bullet). To make matters even more complicated: compiling code to use the newest CPU features or newest optimization techniques will not mean it will actually run faster. E.g. AVX512 may actually slow down your code (when multi-threaded) on many CPUs[1]. Or heavily "optimized" code may become larger in machine code, to the point where your "unoptimized" code may run faster because it fits in the CPU cache(s) properly while the "optimized" version does not. "-Os" optimized code may run faster than "-Ofast" optimized code for this matter. Or it may not. Depending on the actual code. I remember compiling ffmpeg and libx264 myself a bunch of years ago, with the "best" flags for my system, starting with "-march=" and "-Ofast" of course, thinking I am a tough skillful super geek now. Imagine my surprise when I tested the performance against a default ffmpeg build and my optimized build was 2-5% slower. [1] https://blog.cloudflare.com/on-the-dangers-of-intels-frequency-scaling/ https://blog.cloudflare.com/on-the-dangers-of-intels-frequen...

known 6y ago

In my .bashrc I use export CFLAGS='-O2 -march=native -mfpmath=sse -fopenmp -fomit-frame-pointer -pipe -fno-unwind-tables -fno-asynchronous-unwind-tables'

kanox 6y ago
Why would you put something like that in bashrc?!

Supersaiyan_IV 6y ago

This article is way too outdated. Also, who the heck uses -flto on GCC 4.7.1? LTO is experimental in GCC 4.7.1 and will make any application nigh-impossible to debug. Here's something more relevant to modern times: http://hubicka.blogspot.com/2019/05/gcc-9-link-time-and-inter-procedural.html http://hubicka.blogspot.com/2019/05/gcc-9-link-time-and-inte...

gnufx 6y ago
Turning on LTO in Fedora hasn't been entirely painless despite SUSE's pioneering.

vsskanth 6y ago

I wonder if there is an equivalent for MSVC and Clang Compilers. Would be nice if there was a good guide somewhere on flags to use (and their trade-offs) for fast floating point performance for numerical intensive programs.