3 ms·

Software benchmarks are super subjective. Michael Larabel at Phoronix and Isaac Gouy of the benchmark game have done a lot in this area. But everyone says you n

Software benchmarks are super subjective. Michael Larabel at Phoronix and Isaac Gouy of the benchmark game have done a lot in this area. But everyone says you need to take it with a gain if salt (which is often true). There's also TPC-C benchmark suites where people benchmark their own software and claim results. Not really independent journalists there.

Rochus 5y ago
> Software benchmarks are super subjective. No, they are not, but they are just a measurement tool, not a source of absolute truth. When I studied engineering at ETH we learned "Who measures measures rubbish!" ("Wer misst misst Mist!" in German). Every measurement has errors and being aware of these errors and coping with it is part of the engineering profession. The problem with programming language benchmarks is often that the goal is to win by all means; to compare as fairly and objectively as possible instead, there must be a set of suitable rules adhered to by all benchmark implementations. Such a set of rules is e.g. given for the Are-we-fast-yet suite (https://github.com/smarr/are-we-fast-yet https://github.com/smarr/are-we-fast-yet).
- fnord123 5y ago
  It's subjective because it can't be used as a source of truth. Of course "I measured X and my results were Y using methodology Z" can be a statement of fact but X and Z are where the subjectivity lie. For example, benchmark game allows for warmups and so does awfy. This favors jits because it allows them to warm up when they would otherwise be slower. This might give the mistaken impression that java is a great choice for command line tools due to the performance characteristics. In contrast, most benchmarks I've seen don't use profiler guided optimizations for C or C++. Hence the subjectivity. And the claim of only wanting idiomatic code in awfy. This is, of course, subjective as well.
  - kaba0 5y ago
    Last I checked benchmark games didn’t care about warmups.
    - fnord123 5y ago
      You could be correct. The point doesn't rely on what a specific benchmark collection in particular does but that there is an open discussion on what is appropriate in the context of what people using the results find important.
  - deleted 5y ago
    [deleted]

Rochus 5y ago

> Software benchmarks are super subjective. No, they are not, but they are just a measurement tool, not a source of absolute truth. When I studied engineering at ETH we learned "Who measures measures rubbish!" ("Wer misst misst Mist!" in German). Every measurement has errors and being aware of these errors and coping with it is part of the engineering profession. The problem with programming language benchmarks is often that the goal is to win by all means; to compare as fairly and objectively as possible instead, there must be a set of suitable rules adhered to by all benchmark implementations. Such a set of rules is e.g. given for the Are-we-fast-yet suite (https://github.com/smarr/are-we-fast-yet https://github.com/smarr/are-we-fast-yet).

fnord123 5y ago
It's subjective because it can't be used as a source of truth. Of course "I measured X and my results were Y using methodology Z" can be a statement of fact but X and Z are where the subjectivity lie. For example, benchmark game allows for warmups and so does awfy. This favors jits because it allows them to warm up when they would otherwise be slower. This might give the mistaken impression that java is a great choice for command line tools due to the performance characteristics. In contrast, most benchmarks I've seen don't use profiler guided optimizations for C or C++. Hence the subjectivity. And the claim of only wanting idiomatic code in awfy. This is, of course, subjective as well.
- kaba0 5y ago
  Last I checked benchmark games didn’t care about warmups.
  - fnord123 5y ago
    You could be correct. The point doesn't rely on what a specific benchmark collection in particular does but that there is an open discussion on what is appropriate in the context of what people using the results find important.
- deleted 5y ago
  [deleted]