4 ms·

It's technically true that the lowest value will approach the lower bound of how fast your machine can run the code, but there's no guarantee that number is val

It's technically true that the lowest value will approach the lower bound of how fast your machine can run the code, but there's no guarantee that number is valuable. If there is any significant non-determinism in the execution, the minimum time will be a value that you'll only expect to reach in a fraction of executions, even if there are no other processes contending for resources. The only real solution for avoiding noise is to control its sources. Reference here: https://tratt.net/laurie/blog/entries/minimum_times_tend_to_mislead_when_benchmarking.html https://tratt.net/laurie/blog/entries/minimum_times_tend_to_....

oefrha 7y ago
If there's significant nondeterminism in the execution, the method discussed in TFA is also worthless. It's pretty clear determinism is the basic assumption of this discussion; no one's arguing that you should use the minimum execution time among random cases as the performance characteristics of timsort, for instance. Of course, caching of deterministic code complicates matters too.
- hyperpape 7y ago
  I would have to spend a long time studying the relevant Python implementation to think I knew whether there was nondeterminism present in a given benchmark. That's a tall order.
  - oefrha 7y ago
    Now that I think about it, you’re right that randomized hashing in Python itself could cause nondeterminism in seemingly deterministic code. However, timeit does mitigate that to some extent by repeating the same code a large number of times — 1 million times by default, and it’s better used that way. Of course, depending on the actual code, it could be the case that all 1M invocations are skewed by the specific choice of PYTHONHASHSEED; it’s not something that you can solve in-process, so there’s no panacea, you must ditch the idea of getting a single stable number in that case.
mattkrause 7y ago
I'm not sure that there's ever a turnkey method that will save you from needing to do a careful experiment and a careful analysis of the results.

oefrha 7y ago

If there's significant nondeterminism in the execution, the method discussed in TFA is also worthless. It's pretty clear determinism is the basic assumption of this discussion; no one's arguing that you should use the minimum execution time among random cases as the performance characteristics of timsort, for instance. Of course, caching of deterministic code complicates matters too.

hyperpape 7y ago
I would have to spend a long time studying the relevant Python implementation to think I knew whether there was nondeterminism present in a given benchmark. That's a tall order.
- oefrha 7y ago
  Now that I think about it, you’re right that randomized hashing in Python itself could cause nondeterminism in seemingly deterministic code. However, timeit does mitigate that to some extent by repeating the same code a large number of times — 1 million times by default, and it’s better used that way. Of course, depending on the actual code, it could be the case that all 1M invocations are skewed by the specific choice of PYTHONHASHSEED; it’s not something that you can solve in-process, so there’s no panacea, you must ditch the idea of getting a single stable number in that case.

mattkrause 7y ago

I'm not sure that there's ever a turnkey method that will save you from needing to do a careful experiment and a careful analysis of the results.