3 ms·
Taking the average is a bad idea for performance benchmarks like this running under an operating system. The minimum is much more stable. (Easy to verify experi
by Matumio 12y ago
Taking the average is a bad idea for performance benchmarks like this running under an operating system. The minimum is much more stable. (Easy to verify experimentally: do 10x10 runs of a short benchmark and compare the variance of 10 minima to the variance of 10 averages.)
- stefantalpalaru 12y agoTrue, but the minimum might also be dependent on the various caches being primed from previous runs of the same code - a condition that will not appear during normal use.
- Matumio 12y agoCaches being primed is a concern, but not when choosing between minimum and average. Neither is affected much if the first run takes longer than the other nine.
- gcr 12y agoIPython's %timeit magic works by running the function in three loops. It returns the minimum of the three averages. This seems like a good compromise.