4 ms·

I have submitted "hyperfine" 1.5 years ago when it just came out. Since then, the program has gained functionality (statistical outlier detection, result export

I have submitted "hyperfine" 1.5 years ago when it just came out. Since then, the program has gained functionality (statistical outlier detection, result export, parametrized benchmarks) and maturity. Old discussion: https://news.ycombinator.com/item?id=16193225 https://news.ycombinator.com/item?id=16193225 Looking forward to your feedback!

dwohnitmok 7y ago
Since you cite bench as an inspiration, have you ever thought about including the nice graphical HTML page with graphs that bench outputs? In a similar vein what are your thoughts on directly depending on and using criterion (the Rust port)?
- sharkdp 7y ago
  Thank you for the feedback. No, we never thought about HTML output. However, there are multiple other export options and we also ship Python scripts that can be used to plot the benchmark results. The script is not very large, so far, but we are happy to add new scripts if the need for one should arise. What kind of diagrams would you like to see? Also, I have never thought about using criterion.rs. My feeling was that it is suited for benchmarks with thousands of iterations, while we typically only have tens of iterations in hyperfine (as we typically benchmark programs with execution times > 10 ms). Do you have anything specific criterion feature in mind that we could benefit from?
  - dwohnitmok 7y ago
    Ah those were earnest questions, not feature requests in disguise :), especially the second question. I haven't so far had any need for either of those things while using hyperfine (although admittedly I've only used hyperfine for about 3 small projects). I ask purely as a comparison to some of the design choices bench made for personal edification. That being said, one of the hypothetical advantages of using criterion is that you get to piggy-back on its visualizations and statistical analyses, which are quite useful for showing friends and coworkers. I'm not sure of the specifics of how you're doing outlier detection, but Criterion's method is quite nice and I find the choice of linear regression for linearly increasing iterations to be an interesting sanity check. Criterion can work for lower iterations as well (although tens is definitely getting into the lower bound).

dwohnitmok 7y ago

Since you cite bench as an inspiration, have you ever thought about including the nice graphical HTML page with graphs that bench outputs? In a similar vein what are your thoughts on directly depending on and using criterion (the Rust port)?

sharkdp 7y ago
Thank you for the feedback. No, we never thought about HTML output. However, there are multiple other export options and we also ship Python scripts that can be used to plot the benchmark results. The script is not very large, so far, but we are happy to add new scripts if the need for one should arise. What kind of diagrams would you like to see? Also, I have never thought about using criterion.rs. My feeling was that it is suited for benchmarks with thousands of iterations, while we typically only have tens of iterations in hyperfine (as we typically benchmark programs with execution times > 10 ms). Do you have anything specific criterion feature in mind that we could benefit from?
- dwohnitmok 7y ago
  Ah those were earnest questions, not feature requests in disguise :), especially the second question. I haven't so far had any need for either of those things while using hyperfine (although admittedly I've only used hyperfine for about 3 small projects). I ask purely as a comparison to some of the design choices bench made for personal edification. That being said, one of the hypothetical advantages of using criterion is that you get to piggy-back on its visualizations and statistical analyses, which are quite useful for showing friends and coworkers. I'm not sure of the specifics of how you're doing outlier detection, but Criterion's method is quite nice and I find the choice of linear regression for linearly increasing iterations to be an interesting sanity check. Criterion can work for lower iterations as well (although tens is definitely getting into the lower bound).