3 ms·

That doesn't make a lot of sense. It's more like the opposite of what you are saying. The precision of hyperfine is typically in the single-digit millisecond ra

That doesn't make a lot of sense. It's more like the opposite of what you are saying. The precision of hyperfine is typically in the single-digit millisecond range. Maybe just below 1 ms if you take special care to run the benchmark on a quiet system. Everything below that (microsecond or nanosecond range) is something that you need to address with other forms of benchmarking. But for everything in the right range (milliseconds, seconds, minutes or above), hyperfine is well suited.

forrestthewoods 2y ago
No it’s not. Back in the day my goal for Advent of Code was to run all solutions in under 1 second total. Hyperfine would take like 30 minutes to benchmark a 1 second runtime. It was hyper frustrating. I could not find a good way to get Hyperfine to do what I wanted.
- sharkdp 2y ago
  If that's the case, I would consider it a bug. Please feel free to report it. In general, hyperfine should not take longer than ~3 seconds, unless the command itself takes > 300 ms second to run. In the latter case, we do a minimum of 10 runs by default. So if your program takes 3 min for a single iteration, it would take 30 min by default — yes. But this can be controlled using the `-m`/`--min-runs` option. You can also specify the exact amount of runs using `-r`/`--runs`, if you prefer that. > I could not find a good way to get Hyperfine to do what I wanted This is all documented here: https://github.com/sharkdp/hyperfine/tree/master?tab=readme-ov-file#basic-benchmarks https://github.com/sharkdp/hyperfine/tree/master?tab=readme-... under "Basic benchmarks". The options to control the amount of runs are also listed in `hyperfine --help` and in the man page. Please let us know if you think we can improve the documentation / discovery of those options.
- fwip 2y ago
  I've been using it for about four or five years, and never experienced this behavior. Current defaults: "By default, it will perform at least 10 benchmarking runs and measure for at least 3 seconds." If your program takes 1s to run, it should take 10 seconds to benchmark. Is it possible that your program was waiting for input that never came? One "gotcha" is that it expects each argument to be a full program, so if you ran `hyperfine ./a.out input.txt`, it will first bench a.out with no args, then try to bench input.txt (which will fail). If a.out reads from stdin when no argument is given, then it would hang forever, and I can see why you'd give up after a half hour.
  - sharkdp 2y ago
    > Is it possible that your program was waiting for input that never came? We do close stdin to prevent this. So you can benchmark `cat`, for example, and it works just fine.
    - fwip 2y ago
      Oh, my bad! Thank you for the correction, and for all your work making hyperfine.

forrestthewoods 2y ago

No it’s not. Back in the day my goal for Advent of Code was to run all solutions in under 1 second total. Hyperfine would take like 30 minutes to benchmark a 1 second runtime. It was hyper frustrating. I could not find a good way to get Hyperfine to do what I wanted.

sharkdp 2y ago
If that's the case, I would consider it a bug. Please feel free to report it. In general, hyperfine should not take longer than ~3 seconds, unless the command itself takes > 300 ms second to run. In the latter case, we do a minimum of 10 runs by default. So if your program takes 3 min for a single iteration, it would take 30 min by default — yes. But this can be controlled using the `-m`/`--min-runs` option. You can also specify the exact amount of runs using `-r`/`--runs`, if you prefer that. > I could not find a good way to get Hyperfine to do what I wanted This is all documented here: https://github.com/sharkdp/hyperfine/tree/master?tab=readme-ov-file#basic-benchmarks https://github.com/sharkdp/hyperfine/tree/master?tab=readme-... under "Basic benchmarks". The options to control the amount of runs are also listed in `hyperfine --help` and in the man page. Please let us know if you think we can improve the documentation / discovery of those options.
fwip 2y ago
I've been using it for about four or five years, and never experienced this behavior. Current defaults: "By default, it will perform at least 10 benchmarking runs and measure for at least 3 seconds." If your program takes 1s to run, it should take 10 seconds to benchmark. Is it possible that your program was waiting for input that never came? One "gotcha" is that it expects each argument to be a full program, so if you ran `hyperfine ./a.out input.txt`, it will first bench a.out with no args, then try to bench input.txt (which will fail). If a.out reads from stdin when no argument is given, then it would hang forever, and I can see why you'd give up after a half hour.
- sharkdp 2y ago
  > Is it possible that your program was waiting for input that never came? We do close stdin to prevent this. So you can benchmark `cat`, for example, and it works just fine.
  - fwip 2y ago
    Oh, my bad! Thank you for the correction, and for all your work making hyperfine.