3 ms·

And CPU/GPU companies have been caught multiple times in the past optimizing for these particular benchmarks. To the point where if you changed the name of the

And CPU/GPU companies have been caught multiple times in the past optimizing for these particular benchmarks. To the point where if you changed the name of the executable, the performance would drop significantly.

User23 6y ago
Ah yes the classic Quack.exe[1]. [1] https://techreport.com/review/3089/how-atis-drivers-optimize-quake-iii/ https://techreport.com/review/3089/how-atis-drivers-optimize...
- danielbarla 6y ago
  Has the situation changed since then? I mean, there's a reason why e.g. Nvidia drivers know and care about what games you have installed. I can also imagine quite a bit of rationale behind this, from a positive aspect - paying some knowledgeable people to extensively tweak settings by hand for popular games, and then having these settings become part of the driver seems somewhat reasonable. The above example on the other hand, probably was done for nefarious purposes. But how would we know from the outside, when it seems to have become standard practice, and for good reason?
systemvoltage 6y ago
Do you have sources? I have never heard of any CPU manufacturer optimizing Geekbench scores. Consumer grade CPUs are driven by a variety of things - marketing, brand loyalty, gaming benchmarks, reviewers such as Anantech and Extremetech, and yes, Geekbench scores. Datacenter processors are a whole another beast and driven by proprietary benchmarks by vendors. Geekbench runs so many different tests, atleast a couple of dozens of tests. So, if CPU manufacturers are optimizing to those tests, its hard to believe that that's not a good thing and would apply to pretty much what an average user does in the first place.
- lini 6y ago
  There are multiple instances where phone manufacturers optimize the CPU performance when a specific app is loaded - e.g. Oneplus and Geekbench in the OnePlus 5 [1]. [1] https://www.xda-developers.com/oneplus-5-benchmark-cheating-reviews/ https://www.xda-developers.com/oneplus-5-benchmark-cheating-...
- unsigner 6y ago
  There’s the saying that Intel used to employ one compiler engineer per line of SPECint/SPECfp source code; and I heard it from an Intel engineer! (This is not something that should be taken literally, but rather seriously; a lot of money was hanging on these particular benchmarks at some point - when enterprises were buying serious enterprise hardware - so it made sense to invest in compiler tech to make sure you’re not wasting all the hard work done by hardware designers on stupid code generation.)
_huayra_ 6y ago
Emery Berger [0] at UMass Amherst does great work in this, and it made me very wary of a lot of performance benchmarks. A big source no one considers is alignment, wherein things like the executable name as you mentioned or the username or anything else in the shell environment (which iirc is copied into the first few pages of a process). I think in one of the cases, one of Emery's students with a very long username ran the same application significantly slower because the username pushed the pages of code over a boundary. [0] https://emeryberger.github.io/ https://emeryberger.github.io/
- acidbaseextract 6y ago
  I would highly recommend Emery Berger's talks [1] for anyone interested in benchmarking software. The alignment and code layout in memory that huayra mentions has shockingly large performance implications. [1] https://www.youtube.com/watch?v=r-TLSBdHe1A https://www.youtube.com/watch?v=r-TLSBdHe1A

User23 6y ago

Ah yes the classic Quack.exe[1]. [1] https://techreport.com/review/3089/how-atis-drivers-optimize-quake-iii/ https://techreport.com/review/3089/how-atis-drivers-optimize...

danielbarla 6y ago
Has the situation changed since then? I mean, there's a reason why e.g. Nvidia drivers know and care about what games you have installed. I can also imagine quite a bit of rationale behind this, from a positive aspect - paying some knowledgeable people to extensively tweak settings by hand for popular games, and then having these settings become part of the driver seems somewhat reasonable. The above example on the other hand, probably was done for nefarious purposes. But how would we know from the outside, when it seems to have become standard practice, and for good reason?

systemvoltage 6y ago

Do you have sources? I have never heard of any CPU manufacturer optimizing Geekbench scores. Consumer grade CPUs are driven by a variety of things - marketing, brand loyalty, gaming benchmarks, reviewers such as Anantech and Extremetech, and yes, Geekbench scores. Datacenter processors are a whole another beast and driven by proprietary benchmarks by vendors. Geekbench runs so many different tests, atleast a couple of dozens of tests. So, if CPU manufacturers are optimizing to those tests, its hard to believe that that's not a good thing and would apply to pretty much what an average user does in the first place.

lini 6y ago
There are multiple instances where phone manufacturers optimize the CPU performance when a specific app is loaded - e.g. Oneplus and Geekbench in the OnePlus 5 [1]. [1] https://www.xda-developers.com/oneplus-5-benchmark-cheating-reviews/ https://www.xda-developers.com/oneplus-5-benchmark-cheating-...
unsigner 6y ago
There’s the saying that Intel used to employ one compiler engineer per line of SPECint/SPECfp source code; and I heard it from an Intel engineer! (This is not something that should be taken literally, but rather seriously; a lot of money was hanging on these particular benchmarks at some point - when enterprises were buying serious enterprise hardware - so it made sense to invest in compiler tech to make sure you’re not wasting all the hard work done by hardware designers on stupid code generation.)

_huayra_ 6y ago

Emery Berger [0] at UMass Amherst does great work in this, and it made me very wary of a lot of performance benchmarks. A big source no one considers is alignment, wherein things like the executable name as you mentioned or the username or anything else in the shell environment (which iirc is copied into the first few pages of a process). I think in one of the cases, one of Emery's students with a very long username ran the same application significantly slower because the username pushed the pages of code over a boundary. [0] https://emeryberger.github.io/ https://emeryberger.github.io/

acidbaseextract 6y ago
I would highly recommend Emery Berger's talks [1] for anyone interested in benchmarking software. The alignment and code layout in memory that huayra mentions has shockingly large performance implications. [1] https://www.youtube.com/watch?v=r-TLSBdHe1A https://www.youtube.com/watch?v=r-TLSBdHe1A