3 ms·

GCC is a useful benchmark not because it is unoptimized, but because it represents a real-world work profile that is otherwise poorly represented in SPEC. The

GCC is a useful benchmark not because it is unoptimized, but because it represents a real-world work profile that is otherwise poorly represented in SPEC. The working set sizes are large relative to the cache size, and its composed of a bunch of pointer-chasing branch-heavy logic as it winds through the various trees and intermediate languages. That profile is quite a bit closer to the profile of common business and server applications than nearly all of the other members in SPEC.

DannyBee 10y ago
"GCC is a useful benchmark not because it is unoptimized, but because it represents a real-world work profile that is otherwise poorly represented in SPEC." Sure, in that sense, but you also don't want benchmarks that are trivial to game by performing optimizations to them. :)
- brandmeyer 10y ago
  For SPEC, you can't edit the source code of the unit under test. Only the compiler and host hardware count. Some components of SPEC have been "broken" by compilers in the past, by implementing some optimization or other that suddenly made that particular test code much faster. But you don't ever get to re-write the benchmark itself, you can't replace a 2^N path with an N^3 path or anything like that. So these kinds of breakages have almost always been restricted to benchmarks that are reducible to one or two relatively tight inner arithmetic-heavy loop(s). If someone managed to make an optimization that could suddenly make GCC 2x faster without humans editing GCC's source, that would be a major CS breakthrough :)
  - DannyBee 10y ago
    I'm aware of the limitations on SPEC editing. However, "If someone managed to make an optimization that could suddenly make GCC 2x faster without humans editing GCC's source, that would be a major CS breakthrough :) " No, it really wouldn't. For example, if you optimize the line-ending and tokenization finding in GCC's parser to use SIMD, you would become significantly faster. Not 2x, but significantly. These are entirely doable by compilers. " But you don't ever get to re-write the benchmark itself, you can't replace a 2^N path with an N^3 path or anything like that." Except, you can, by hoisting calls out of loops, etc. There are also dynamic memoization optimziations that can be applied, etc. "If someone managed to make an optimization " This is a common fallacy that it's ever really one optimization. It's usually combinations of optimizations, each buying you N%. I will state outright, having worked on pretty much all of the slow parts of gcc (according to spec profiles), and having a team of 80+ people working on compilers, 2x on gcc through compiler optimization is more than within the realm of possibility.

DannyBee 10y ago

"GCC is a useful benchmark not because it is unoptimized, but because it represents a real-world work profile that is otherwise poorly represented in SPEC." Sure, in that sense, but you also don't want benchmarks that are trivial to game by performing optimizations to them. :)

brandmeyer 10y ago
For SPEC, you can't edit the source code of the unit under test. Only the compiler and host hardware count. Some components of SPEC have been "broken" by compilers in the past, by implementing some optimization or other that suddenly made that particular test code much faster. But you don't ever get to re-write the benchmark itself, you can't replace a 2^N path with an N^3 path or anything like that. So these kinds of breakages have almost always been restricted to benchmarks that are reducible to one or two relatively tight inner arithmetic-heavy loop(s). If someone managed to make an optimization that could suddenly make GCC 2x faster without humans editing GCC's source, that would be a major CS breakthrough :)
- DannyBee 10y ago
  I'm aware of the limitations on SPEC editing. However, "If someone managed to make an optimization that could suddenly make GCC 2x faster without humans editing GCC's source, that would be a major CS breakthrough :) " No, it really wouldn't. For example, if you optimize the line-ending and tokenization finding in GCC's parser to use SIMD, you would become significantly faster. Not 2x, but significantly. These are entirely doable by compilers. " But you don't ever get to re-write the benchmark itself, you can't replace a 2^N path with an N^3 path or anything like that." Except, you can, by hoisting calls out of loops, etc. There are also dynamic memoization optimziations that can be applied, etc. "If someone managed to make an optimization " This is a common fallacy that it's ever really one optimization. It's usually combinations of optimizations, each buying you N%. I will state outright, having worked on pretty much all of the slow parts of gcc (according to spec profiles), and having a team of 80+ people working on compilers, 2x on gcc through compiler optimization is more than within the realm of possibility.