4 ms·

The issue is it runs a kajillion tests to try and be “statistical”. But there’s no good way to say “just run it for 5 seconds and give me the best answer you ca

The issue is it runs a kajillion tests to try and be “statistical”. But there’s no good way to say “just run it for 5 seconds and give me the best answer you can”. It’s very much designed for nanosecond to low microsecond benchmarks. Trying to fight this is trying to smash a square peg through a round hole.

gforce_de 2y ago
At least it gives some numbers and point in a direction: $ hyperfine --warmup 3 './hello-world-bin-sh.sh' './hello-world-env-python3.py' Benchmark 1: ./hello-world-bin-sh.sh Time (mean ± σ): 1.3 ms ± 0.4 ms [User: 1.0 ms, System: 0.5 ms] ... Benchmark 2: ./hello-world-env-python3.py Time (mean ± σ): 43.1 ms ± 1.4 ms [User: 33.6 ms, System: 8.4 ms] ...
PhilipRoman 2y ago
I disagree that it is designed for nano/micro benchmarks. If you want that level of detail, you need to stay within a single process, pinned to a core which is isolated from scheduler. At least I found it almost impossible to benchmark assembly routines with it.
sharkdp 2y ago
> The issue is it runs a kajillion tests to try and be “statistical”. If you see any reason for putting “statistical” in quotes, please let us know. hyperfine does not run a lot of tests, but it does try to find outliers in your measurements. This is really valuable in some cases. For example: we can detect when the first run of your program takes much longer than the rest of the runs. We can then show you a warning to let you know that you probably want to either use some warmup runs, or a "--prepare" command to clean (OS) caches if you want a cold-cache benchmark. > But there’s no good way to say “just run it for 5 seconds and give me the best answer you can”. What is the "best answer you can"? > It’s very much designed for nanosecond to low microsecond benchmarks. Absolutely not. With hyperfine, you can not measure execution times in the "low microsecond" range, let alone nanosecond range. See also my other comment.

gforce_de 2y ago

At least it gives some numbers and point in a direction: $ hyperfine --warmup 3 './hello-world-bin-sh.sh' './hello-world-env-python3.py' Benchmark 1: ./hello-world-bin-sh.sh Time (mean ± σ): 1.3 ms ± 0.4 ms [User: 1.0 ms, System: 0.5 ms] ... Benchmark 2: ./hello-world-env-python3.py Time (mean ± σ): 43.1 ms ± 1.4 ms [User: 33.6 ms, System: 8.4 ms] ...

PhilipRoman 2y ago

I disagree that it is designed for nano/micro benchmarks. If you want that level of detail, you need to stay within a single process, pinned to a core which is isolated from scheduler. At least I found it almost impossible to benchmark assembly routines with it.

sharkdp 2y ago

> The issue is it runs a kajillion tests to try and be “statistical”. If you see any reason for putting “statistical” in quotes, please let us know. hyperfine does not run a lot of tests, but it does try to find outliers in your measurements. This is really valuable in some cases. For example: we can detect when the first run of your program takes much longer than the rest of the runs. We can then show you a warning to let you know that you probably want to either use some warmup runs, or a "--prepare" command to clean (OS) caches if you want a cold-cache benchmark. > But there’s no good way to say “just run it for 5 seconds and give me the best answer you can”. What is the "best answer you can"? > It’s very much designed for nanosecond to low microsecond benchmarks. Absolutely not. With hyperfine, you can not measure execution times in the "low microsecond" range, let alone nanosecond range. See also my other comment.