5 ms·
Does using the geometric mean do something to the size of the error? If the geometric mean of the original implementation varies 1% under normal conditions then
by chrisseaton 11y ago
Does using the geometric mean do something to the size of the error? If the geometric mean of the original implementation varies 1% under normal conditions then this is still not a significant result is it? I have benchmarks where my geomean easily varies 1% even in the most stable conditions I know to give it.
- exDM69 11y agoIt's a benchmark engineered to measure the performance of compilers and CPUs. It's healthy to be skeptical but in general if you get a better score, it's a real improvement. You should be able to safely exclude measurement errors and other mundane issues. These benchmarks have a slight issue that they might not represent real world performance, but compilers and CPUs tend to be optimized against them - because their timing and measurement is generally more reliable than ad-hoc real world measurement. It might lead to slight "over-optimization" which doesn't reflect real performance but at least it's reliable and repeatable.