8 ms·

Firefox 64 built with GCC and Clang

188 points by Twirrim 8y ago

hsivonen 8y ago
It'll be interesting to see the numbers again after LLVM thinLTO starts applying across C++ and Rust resulting in cross-language inlining.
- unixhero 8y ago
  I don't know what you are saying here. Can you elaborate?
  - hsivonen 8y ago
    A goal that is being worked towards is making LLVM thinLTO not just consider clang output but to consider clang-generated LLVM IR together with rustc-generated LLVM IR. This is expected to lead to inlining between C++ and Rust making the FFI layer of C-linkage function calls melt away between C++ and Rust.
    - pedrocr 8y ago
      This seems to be the bug that's tracking this in rust: https://github.com/rust-lang/rust/issues/49879 https://github.com/rust-lang/rust/issues/49879
duhast 8y ago
It's hard to count the number of major improvements that landed in GCC since the inception of clang. Competition in this landscape is benefiting everyone.
- Avamander 8y ago
  I like the rising popularity of LTO the most.
nkurz 8y ago
I realize it's not the focus of his test, but as someone who thinks often about how to take advantage of advanced vectorization techniques on modern processors, I was surprised by statements like this: Moreover GCC -O2 defaults are (in my opinion unfortunately) still not enabling vectorization and unrolling which may have noticeable effects on benchmarks. This led to enabling AVX and since the global constructor now gets some code auto-vectorized the binary crashed on invalid instruction during the build (my testing machine has no AVX). No AVX? He wants to better take advantage of vectorization, but he's doing the testing on a processor that is 3 generations behind in vectorization support. AVX (128-bit) come out in 2011, and has been followed by AVX2 (256-bit) and (still limited release) AVX-512. Clock speeds have been fairly flat, and most of the improvements to recent processors have been microarchitectural. A lot of the optimization done by compilers ends up being architecture specific. Seeing which brand-new compiler best targets old hardware seems like it might produce misleading results. I realize that not everyone has (or can have) the most recent hardware, but this seems like a case where it would be strongly in AMD and Intel's interest to make sure that people like Jan have better access to the improvements made in the last few years.
- hsivonen 8y ago
  Firefox still treats SSE2 as its SIMD assumed-present baseline. Does GCC introduce multiversioned AVX functions on its own initiative?
  - nkurz 8y ago
    Does GCC introduce multiversioned AVX functions on its own initiative? Yes, this seems like one reasonable approach. The current approach of compiling to a "least common denominator" and then updating this denominator every decade or so seems insufficient. Instead of interpreting the absence of a "-march=" flag to mean optimize for nothing, maybe it can mean that multiple optimizations are automatically compiled and the appropriate one is selected at runtime. Alternatively, maybe we need to move away from the idea of compiling a single binary that runs on all platforms, and encourage greater use of platform specific compilations.
mhh__ 8y ago
The 48% difference in code size is surprising. But after all who cares in this world of electron etc.
- MaxBarraclough 8y ago
  If it affects cache behaviour, we should all care.
  - mhh__ 8y ago
    I guess that LLVM is fairly cache aware so I think the performance probably isn't affected too greatly (I hope).
    - hubicka 8y ago
      I would be interested to know what cache aware code layout optimizations are available in LLVM. I personally know of none. GCC is bit simplistic in this sense (it does reorder functions based on profile feedback and execution time) and I plan to change that for next stage 1 (i.e. GCC 10)
      - DannyBee 8y ago
        Hey Jan, Long time ;) LLVM will do the same kind of reordering. (Both are interestingly well behind what commercial compilers do, and this is one of the very few areas where that is true. My suspicion is that it does not matter as much in practice as we want it to. Most forms of layout optimization are also very hard to perform on the C++ code you want to optimize due to inability to prove safety)
        hubicka 8y ago
        Hehe, nice to see you :) Yep, I have code layout pass in my tree for a while, but because I was never really able to measure off-noise improvements it is not in the tree, yet. I hope to make more sense of it with help of CPU counters which improved over the time.
21 8y ago
He's testing on a 7 year old 8-core server CPU. As irrelevant as possible for your average laptop Firefox user.
- hubicka 8y ago
  You can try the binary on your CPU. This particular workload does not make much difference between modern CPUs. I just tried the Sunspider benchmark on my skylake and it has similar outcomes as reported, but there is more noise since it is notebook What I got is: GCC 8 build: 333 +- 3.3% Tumbleweed distro firefox: 352 +- 3.4% Firefox 63 (GCC) official binary: 346 +- 5.6% Firefox 64 (llvm) official binary: 342 +- 5.1% but I do not completely trust the numbers as re-running the benchmark leads to different outcome each time
- josteink 8y ago
  I suspect you seriously overestimate the amount of developers on bleeding edge hardware. My desktop is a first gen i7. My Laptop a 4th gen. My work machine a 5 year old Xeon. And you know what? They all work great and I see no reason to upgrade. I guess you will be shocked to hear I’m doing fine with 8GBs of ram too :)
ajross 8y ago
I'm not loving the Firefox motion toward clang. For years we've been told that clang is great because we finally have a competitor for gcc and that multiple interoperable compilers can only improve the ecosystem (which is undeniably true). Now we have a big project deciding to move from a reasonably portable gcc build to a clang-specific LTO framework that required significant engineering effort to achieve and which apparently isn't easily portable to the equivalent gcc effort, requiring a gcc maintainer to jump in on their behalf to show equivalence. How is this not moving backwards?
- kibwen 8y ago
  The advantage of having multiple C compilers has always been broader platform support and competition to improve compiler speed, error messages, and codegen quality. Conspicuously, one complete non-advantage of having multiple C compilers is portability of a codebase between compilers; the C specification contains too much implementation-defined behavior (including undefined behavior) and is too anemic (requiring compilers to come up with nonstandard extensions to support things like assembly and SIMD) for compiler-portability to be anything other than a nightmare for large projects. C projects even have a hard time upgrading to new versions of the same compiler, which helps to explain why so many shops are still using positively ancient versions of GCC.
  - gumby 8y ago
    I had the good fortune to work on a code base that started as a blank emacs buffer and was compiled with both g++ and clang++ from the get go (and built on multiple platforms, with maximal warnings enabled). The two compilers surfaced different bugs in our codebase which, in addition to finding actual (semantic) bugs has been quite valuable for portability and maintainability. (Also, as others have noted, the existence of clang has really been good for gcc).
- nn3 8y ago
  And even worse, it's generating worse and ridiculously bigger code. So they spend a lot of effort for worse result. And apparently they don't catch serious problems in their build system on their own (profile feedback pass timing out and truncating data) Overall not a great result at all for Mozilla. Where is the regression testing for performance? They are not data driven?
navjack27 8y ago
I think I'm going to mess with this later myself. On Manjaro I usually compile Chromium with -O3 and -march=native with no mtune or any of that but I never benchmarked it against anything. I'll do the same with Firefox. This is on coffee lake BTW.
navjack27 8y ago
I wouldn't run testing on a notebook. I mean you can if what you are possibly testing is boosting characteristics and other variables... But best bet for low variable consistent testing is a machine where you have set a static core clock speed and have disabled c-states and other power save things. Remember, you are testing differences in compile optimization. You don't want your system being a variable.
deleted 8y ago
[deleted]
Twirrim 8y ago
One of the things that disturbed me from the article was how the Mozilla build chain just merrily ignores that profiling had failed, and moves on building stuff using that profile. That seems like quite dangerous behaviour. Surely that should be a failing step for a build, or at the very least a large warning should go out at the end "This build may be optimised based on complete nonsense because profiling failed"
carapace 8y ago
This is all way way WAY too complicated. It's like a fractal Rube Goldberg machine made of Rube Goldberg machines. All this to render web pages. I think we must have made a wrong turn somewhere.
- zbentley 8y ago
  > All this to render web pages. I think we must have made a wrong turn somewhere. We've taken plenty of wrong turns, but none, I think, accounted for more than a rounding error in time or code needed to render web pages. Writing a browser is hard. Hell, even writing a toy browser-like mockup isn't easy. I built an extremely bad renderer for an extremely simple class-provided XML-ish grammar in school. It only supported a handful of styling keywords (all inline/attribute-based), only one of which was positioning-related ("wrap to next fixed-height global line of display after this element"). It was really hard. Like, really hard. Even looking back on the code with the benefit of experience, it still would not be a breeze. It supported a single fixed window size and a guaranteed-correct input file. Removing either of those constraints would have exploded the code size to the point I doubt I could have done it alone then, and if I could now it would take me an incredibly long time. Adding the full HTML spec would probably bring its SLoC counts into the 100ks, if not millions. Supporting re-renders and after-the-fact DOM updates would blow it far beyond that, making them fast might require me to go back to school, but who knows; maybe it's easier than my hunch. I suppose I could shave time by moving some of those hundreds of thousands of lines into the libraries which evolved during the many years since browsers became popular, but it would still be a gargantuan undertaking. And all of that is before the immense amount of person-hours which would be needed for: - Supporting cascading styling of any kind, with or without embedding another language. - Adding networking, even if interoperability/an agreed-upon communication pattern or protocol already existed. - Displaying assets other than styled text and SVG-esque drawings (images, videos, etc). - Securing the request/response protocol, even if leveraging existing tools like OpenSSL to the max. - Adding another turing-complete and secure programming language for communicating with random local/networked resources and producing more requests or DOM updates. It's hard. TL;DR There are plenty of needlessly-complex tools and technologies out there. But I don't think web browsers are some of them. Even if you're anti-JS and anti-CSS, there is still an absolute shitload of complex, careful, hard-to-get-right interactions going on under the hood.

188 points by Twirrim 8y ago

It'll be interesting to see the numbers again after LLVM thinLTO starts applying across C++ and Rust resulting in cross-language inlining.

unixhero 8y ago
I don't know what you are saying here. Can you elaborate?
- hsivonen 8y ago
  A goal that is being worked towards is making LLVM thinLTO not just consider clang output but to consider clang-generated LLVM IR together with rustc-generated LLVM IR. This is expected to lead to inlining between C++ and Rust making the FFI layer of C-linkage function calls melt away between C++ and Rust.
  - pedrocr 8y ago
    This seems to be the bug that's tracking this in rust: https://github.com/rust-lang/rust/issues/49879 https://github.com/rust-lang/rust/issues/49879

duhast 8y ago

It's hard to count the number of major improvements that landed in GCC since the inception of clang. Competition in this landscape is benefiting everyone.

Avamander 8y ago
I like the rising popularity of LTO the most.

nkurz 8y ago

I realize it's not the focus of his test, but as someone who thinks often about how to take advantage of advanced vectorization techniques on modern processors, I was surprised by statements like this: Moreover GCC -O2 defaults are (in my opinion unfortunately) still not enabling vectorization and unrolling which may have noticeable effects on benchmarks. This led to enabling AVX and since the global constructor now gets some code auto-vectorized the binary crashed on invalid instruction during the build (my testing machine has no AVX). No AVX? He wants to better take advantage of vectorization, but he's doing the testing on a processor that is 3 generations behind in vectorization support. AVX (128-bit) come out in 2011, and has been followed by AVX2 (256-bit) and (still limited release) AVX-512. Clock speeds have been fairly flat, and most of the improvements to recent processors have been microarchitectural. A lot of the optimization done by compilers ends up being architecture specific. Seeing which brand-new compiler best targets old hardware seems like it might produce misleading results. I realize that not everyone has (or can have) the most recent hardware, but this seems like a case where it would be strongly in AMD and Intel's interest to make sure that people like Jan have better access to the improvements made in the last few years.

hsivonen 8y ago
Firefox still treats SSE2 as its SIMD assumed-present baseline. Does GCC introduce multiversioned AVX functions on its own initiative?
- nkurz 8y ago
  Does GCC introduce multiversioned AVX functions on its own initiative? Yes, this seems like one reasonable approach. The current approach of compiling to a "least common denominator" and then updating this denominator every decade or so seems insufficient. Instead of interpreting the absence of a "-march=" flag to mean optimize for nothing, maybe it can mean that multiple optimizations are automatically compiled and the appropriate one is selected at runtime. Alternatively, maybe we need to move away from the idea of compiling a single binary that runs on all platforms, and encourage greater use of platform specific compilations.

mhh__ 8y ago

The 48% difference in code size is surprising. But after all who cares in this world of electron etc.

MaxBarraclough 8y ago
If it affects cache behaviour, we should all care.
- mhh__ 8y ago
  I guess that LLVM is fairly cache aware so I think the performance probably isn't affected too greatly (I hope).
  - hubicka 8y ago
    I would be interested to know what cache aware code layout optimizations are available in LLVM. I personally know of none. GCC is bit simplistic in this sense (it does reorder functions based on profile feedback and execution time) and I plan to change that for next stage 1 (i.e. GCC 10)
    - DannyBee 8y ago
      Hey Jan, Long time ;) LLVM will do the same kind of reordering. (Both are interestingly well behind what commercial compilers do, and this is one of the very few areas where that is true. My suspicion is that it does not matter as much in practice as we want it to. Most forms of layout optimization are also very hard to perform on the C++ code you want to optimize due to inability to prove safety)
      - hubicka 8y ago
        Hehe, nice to see you :) Yep, I have code layout pass in my tree for a while, but because I was never really able to measure off-noise improvements it is not in the tree, yet. I hope to make more sense of it with help of CPU counters which improved over the time.

21 8y ago

He's testing on a 7 year old 8-core server CPU. As irrelevant as possible for your average laptop Firefox user.

hubicka 8y ago
You can try the binary on your CPU. This particular workload does not make much difference between modern CPUs. I just tried the Sunspider benchmark on my skylake and it has similar outcomes as reported, but there is more noise since it is notebook What I got is: GCC 8 build: 333 +- 3.3% Tumbleweed distro firefox: 352 +- 3.4% Firefox 63 (GCC) official binary: 346 +- 5.6% Firefox 64 (llvm) official binary: 342 +- 5.1% but I do not completely trust the numbers as re-running the benchmark leads to different outcome each time
josteink 8y ago
I suspect you seriously overestimate the amount of developers on bleeding edge hardware. My desktop is a first gen i7. My Laptop a 4th gen. My work machine a 5 year old Xeon. And you know what? They all work great and I see no reason to upgrade. I guess you will be shocked to hear I’m doing fine with 8GBs of ram too :)

ajross 8y ago

I'm not loving the Firefox motion toward clang. For years we've been told that clang is great because we finally have a competitor for gcc and that multiple interoperable compilers can only improve the ecosystem (which is undeniably true). Now we have a big project deciding to move from a reasonably portable gcc build to a clang-specific LTO framework that required significant engineering effort to achieve and which apparently isn't easily portable to the equivalent gcc effort, requiring a gcc maintainer to jump in on their behalf to show equivalence. How is this not moving backwards?

kibwen 8y ago
The advantage of having multiple C compilers has always been broader platform support and competition to improve compiler speed, error messages, and codegen quality. Conspicuously, one complete non-advantage of having multiple C compilers is portability of a codebase between compilers; the C specification contains too much implementation-defined behavior (including undefined behavior) and is too anemic (requiring compilers to come up with nonstandard extensions to support things like assembly and SIMD) for compiler-portability to be anything other than a nightmare for large projects. C projects even have a hard time upgrading to new versions of the same compiler, which helps to explain why so many shops are still using positively ancient versions of GCC.
- gumby 8y ago
  I had the good fortune to work on a code base that started as a blank emacs buffer and was compiled with both g++ and clang++ from the get go (and built on multiple platforms, with maximal warnings enabled). The two compilers surfaced different bugs in our codebase which, in addition to finding actual (semantic) bugs has been quite valuable for portability and maintainability. (Also, as others have noted, the existence of clang has really been good for gcc).
nn3 8y ago
And even worse, it's generating worse and ridiculously bigger code. So they spend a lot of effort for worse result. And apparently they don't catch serious problems in their build system on their own (profile feedback pass timing out and truncating data) Overall not a great result at all for Mozilla. Where is the regression testing for performance? They are not data driven?

navjack27 8y ago

I think I'm going to mess with this later myself. On Manjaro I usually compile Chromium with -O3 and -march=native with no mtune or any of that but I never benchmarked it against anything. I'll do the same with Firefox. This is on coffee lake BTW.

navjack27 8y ago

I wouldn't run testing on a notebook. I mean you can if what you are possibly testing is boosting characteristics and other variables... But best bet for low variable consistent testing is a machine where you have set a static core clock speed and have disabled c-states and other power save things. Remember, you are testing differences in compile optimization. You don't want your system being a variable.

deleted 8y ago

[deleted]

Twirrim 8y ago

One of the things that disturbed me from the article was how the Mozilla build chain just merrily ignores that profiling had failed, and moves on building stuff using that profile. That seems like quite dangerous behaviour. Surely that should be a failing step for a build, or at the very least a large warning should go out at the end "This build may be optimised based on complete nonsense because profiling failed"

carapace 8y ago

This is all way way WAY too complicated. It's like a fractal Rube Goldberg machine made of Rube Goldberg machines. All this to render web pages. I think we must have made a wrong turn somewhere.

zbentley 8y ago
> All this to render web pages. I think we must have made a wrong turn somewhere. We've taken plenty of wrong turns, but none, I think, accounted for more than a rounding error in time or code needed to render web pages. Writing a browser is hard. Hell, even writing a toy browser-like mockup isn't easy. I built an extremely bad renderer for an extremely simple class-provided XML-ish grammar in school. It only supported a handful of styling keywords (all inline/attribute-based), only one of which was positioning-related ("wrap to next fixed-height global line of display after this element"). It was really hard. Like, really hard. Even looking back on the code with the benefit of experience, it still would not be a breeze. It supported a single fixed window size and a guaranteed-correct input file. Removing either of those constraints would have exploded the code size to the point I doubt I could have done it alone then, and if I could now it would take me an incredibly long time. Adding the full HTML spec would probably bring its SLoC counts into the 100ks, if not millions. Supporting re-renders and after-the-fact DOM updates would blow it far beyond that, making them fast might require me to go back to school, but who knows; maybe it's easier than my hunch. I suppose I could shave time by moving some of those hundreds of thousands of lines into the libraries which evolved during the many years since browsers became popular, but it would still be a gargantuan undertaking. And all of that is before the immense amount of person-hours which would be needed for: - Supporting cascading styling of any kind, with or without embedding another language. - Adding networking, even if interoperability/an agreed-upon communication pattern or protocol already existed. - Displaying assets other than styled text and SVG-esque drawings (images, videos, etc). - Securing the request/response protocol, even if leveraging existing tools like OpenSSL to the max. - Adding another turing-complete and secure programming language for communicating with random local/networked resources and producing more requests or DOM updates. It's hard. TL;DR There are plenty of needlessly-complex tools and technologies out there. But I don't think web browsers are some of them. Even if you're anti-JS and anti-CSS, there is still an absolute shitload of complex, careful, hard-to-get-right interactions going on under the hood.