3 ms·
If there's significant nondeterminism in the execution, the method discussed in TFA is also worthless. It's pretty clear determinism is the basic assumption of
by oefrha 7y ago
If there's significant nondeterminism in the execution, the method discussed in TFA is also worthless. It's pretty clear determinism is the basic assumption of this discussion; no one's arguing that you should use the minimum execution time among random cases as the performance characteristics of timsort, for instance.
Of course, caching of deterministic code complicates matters too.
- hyperpape 7y agoI would have to spend a long time studying the relevant Python implementation to think I knew whether there was nondeterminism present in a given benchmark. That's a tall order.
- oefrha 7y agoNow that I think about it, you’re right that randomized hashing in Python itself could cause nondeterminism in seemingly deterministic code. However, timeit does mitigate that to some extent by repeating the same code a large number of times — 1 million times by default, and it’s better used that way. Of course, depending on the actual code, it could be the case that all 1M invocations are skewed by the specific choice of PYTHONHASHSEED; it’s not something that you can solve in-process, so there’s no panacea, you must ditch the idea of getting a single stable number in that case.