3 ms·
I agree the need to produce a single overall number is nuts. We need a better setup for comparing things, somehow... That said, if you _do_ need to produce on
by bzbarsky 13y ago
I agree the need to produce a single overall number is nuts. We need a better setup for comparing things, somehow...
That said, if you _do_ need to produce one in a situation like this, then geometric mean is the least bad way to do it because it does not depend on the normalization of the individual tests at all: if you have 'k' tests and change the workload for one of them by a factor of N, then the average will change by a factor of N^{1/k} no matter what the actual numbers are. This is why it's used so commonly in benchmark that want to produce a single number.
Of course this very property is what makes geometric-mean-based benchmarks so gameable: a 20% speedup of an already-fast operation (say eliminating a single instruction) is worth as much for purposes of the benchmark as a 20% speedup in something that's really slow. The result is that browsers keep micro-optimizing already-fast things to win the benchmark game....