3 ms·

All benchmarks are misleading.

by stephencanon 7y ago

VC999 7y ago
Could you expand on that please
- jstewartmobile 7y ago
  The search space has many, many dimensions--CPU, MMU, OS implementation details, how well the compiler translated source to machine code for that particular CPU, how well did the benchmark implementation cater to the target architecture, etc, etc. Immense opportunities to fool others, and even yourself.
- me_me_me 7y ago
  The fastest animal on earth is the fastest not because it has huge muscles, but because it falls. Is it fair compare top speed of falcon to cheetah? Yeah but... x y z. Every benchmark can be set up in a way that will give edge to one part or another. Especially if there is a vested interest.
  - deaddodo 7y ago
    And even in your example, the Cheetah can hit 65mph for a whopping 330 feet before risking total exhaustion. Meanwhile Pronghorns, Impala and Antelope can reach 55mph-60mph and maintain it for about half a mile, and vary their speed to increase distance. So which would you consider faster? The one that can give you a few seconds of effort really quickly; or the one that gives you 90% of that effort but for hours?
- stephencanon 7y ago
  All benchmarks are chosen to show precisely what the person presenting the benchmark wants them to, no more, no less. If the benchmark didn't tell a story that they want to show you, they wouldn't be showing it to you (or they'd be showing you a different benchmark). This doesn't make them wrong, but benchmarks are never the full story.
sin7 7y ago
The map is not the territory. All models are wrong, some are useful. Blah, blah, blah.
- bigmattystyles 7y ago
  https://www.youtube.com/watch?v=yZT-wVnFn60 https://www.youtube.com/watch?v=yZT-wVnFn60
Fnoord 7y ago
..but some are more [obviously] misleading than other.
Tepix 7y ago
Not if you designed a benchmark for your special requirements.
- emmelaich 7y ago
  It still is misleading, because your special requirements will not perfectly match your benchmark.
  - zerkten 7y ago
    That's true, but like many human endeavors we can reach a point that is good enough, or the process of implementing the benchmark leads to useful discoveries. It is not going to be perfect for all situations, but that doesn't mean it's not a valuable tool to use in some situations.
  - gameswithgo 7y ago
    they might, if your benchmark IS your special requirements.
  - stephencanon 7y ago
    Or your requirements will change in ways that you didn't expect, invalidating your benchmark, or ...

All benchmarks are misleading.

VC999 7y ago

Could you expand on that please

jstewartmobile 7y ago
The search space has many, many dimensions--CPU, MMU, OS implementation details, how well the compiler translated source to machine code for that particular CPU, how well did the benchmark implementation cater to the target architecture, etc, etc. Immense opportunities to fool others, and even yourself.
me_me_me 7y ago
The fastest animal on earth is the fastest not because it has huge muscles, but because it falls. Is it fair compare top speed of falcon to cheetah? Yeah but... x y z. Every benchmark can be set up in a way that will give edge to one part or another. Especially if there is a vested interest.
- deaddodo 7y ago
  And even in your example, the Cheetah can hit 65mph for a whopping 330 feet before risking total exhaustion. Meanwhile Pronghorns, Impala and Antelope can reach 55mph-60mph and maintain it for about half a mile, and vary their speed to increase distance. So which would you consider faster? The one that can give you a few seconds of effort really quickly; or the one that gives you 90% of that effort but for hours?
stephencanon 7y ago
All benchmarks are chosen to show precisely what the person presenting the benchmark wants them to, no more, no less. If the benchmark didn't tell a story that they want to show you, they wouldn't be showing it to you (or they'd be showing you a different benchmark). This doesn't make them wrong, but benchmarks are never the full story.

sin7 7y ago

The map is not the territory. All models are wrong, some are useful. Blah, blah, blah.

bigmattystyles 7y ago
https://www.youtube.com/watch?v=yZT-wVnFn60 https://www.youtube.com/watch?v=yZT-wVnFn60

Fnoord 7y ago

..but some are more [obviously] misleading than other.

Tepix 7y ago

Not if you designed a benchmark for your special requirements.

emmelaich 7y ago
It still is misleading, because your special requirements will not perfectly match your benchmark.
- zerkten 7y ago
  That's true, but like many human endeavors we can reach a point that is good enough, or the process of implementing the benchmark leads to useful discoveries. It is not going to be perfect for all situations, but that doesn't mean it's not a valuable tool to use in some situations.
- gameswithgo 7y ago
  they might, if your benchmark IS your special requirements.
- stephencanon 7y ago
  Or your requirements will change in ways that you didn't expect, invalidating your benchmark, or ...