3 ms·
I'm pretty sure I was part of the "resurgence in people advocating for the use of the minimum time recently," and I'll tell you right now that I did not have a
by rrss 7y ago
I'm pretty sure I was part of the "resurgence in people advocating for the use of the minimum time recently," and I'll tell you right now that I did not have a "benchmark" that varied by 7x from run to run in mind.
"Benchmarks with random performance tend to mislead"
Both of these "benchmarks" have exactly the wrong amount of randomness. A little more, and you could get a useful mean out of the results.
When I advocate for the use of minimum time in benchmarking, the scenario in mind is running periodic regression testing on a set of known good benchmarks that don't have intrinsic randomness (or it is averaged out) and yet vary by 0-10% as a result of actual noise. In these cases, it is very useful to have a single number (and if that single number surprises you, then by all means take a look at the distribution).
Neither of the examples here show the impact of actual system or measurement noise, only random performance.
- rrss 7y agoI went ahead and modified the author's second benchmark (used to compare linear vs binary search) to iterate over various choices over words, rather than just searching for one word 10000 times (which may be the first on one run and the last on another). Linear search: 0.92 +- less than 0.0001 s Binary search: 0.15 s (also small variance, but didn't compute it) We can talk as long as we want about mean vs minimum vs histogram vs etc, but let's go back to the beginning: you get what you measure.