3 ms·
Isn't figuring out the slope a red herring? The total run time is: n * timing_overhead + n(n+1)/2 * runtime Seems like you'd do better by measuring n^2 ru
by spankalee 7y ago
Isn't figuring out the slope a red herring? The total run time is:
n * timing_overhead + n(n+1)/2 * runtime
Seems like you'd do better by measuring n^2 runs and dividing by n^2. But that's not very good because it's not taking into account the observed variance. What you really want to know it something like the 95% confidence interval after observing n runs.
My teammate has been working on a library to do just this for JavaScript, with added automation for running a set of benchmarks until some set of thresholds, like A is 95% likely faster than B.
https://www.npmjs.com/package/tachometer https://www.npmjs.com/package/tachometer
- kqr 7y ago> What you really want to know it something like the 95% confidence interval after observing n runs. Great in theory, but almost always relies on a normality assumption which just doesn't hold in general for performance measurements. How well does this package handle pareto distributed run times?
- mattkrause 7y agoThough this particular package doesn't, you can certainly derive confidence (or better, credible) intervals for non-normally distributed data.