4 ms·

If you want to know about the 99.5% quantile, then of course you have to look at that, not the 95% quantile. (Note that "quintiles" refers to the 20%, 40%, 60%

If you want to know about the 99.5% quantile, then of course you have to look at that, not the 95% quantile. (Note that "quintiles" refers to the 20%, 40%, 60%, and 80% quantiles,but you can look at whatever quantile you like.) Your comment about "confidence interval error bars" revealing outliers makes no sense. I think you still don't understand that a confidence interval (for some parameter estimate, maybe the mean) is not a description of the distribution of the data, but rather is about the distribution of an estimate obtained from the data - not at all the same thing. Running a test 1 billion times certainly would reduce the confidence interval for the mean, even if 5 million of those are 100x outliers.

Retric 7y ago
I think I figured out the confusion. Benchmarking is a comparison and each test is a fixed sample size. A test might collect 100 or 100,000 samples by running the task repeatedly, but the data is not aggregated across multiple tests as the data is generally not saved. So, running the benchmark test again keep’s n the same when calculating the confidence interval. This means error bars are showing the calculated mean and standard * arbitrary scaling factor all lined up on an easy to read graph. PS: A lot of statistical tools are poor fits as the data is rarely a normal distribution. But, rigger is not the goal here.
- radford-neal 7y ago
  You're still not getting it. Confidence intervals are not what you want to look at. Neither are error bars. They would be relevant if you wanted to know the mean performance, and wondered how accurately you had estimated this mean. But as far as I can tell, that's what you want to know.
  - Retric 7y ago
    It's not how accurately you had estimated this mean, but how representative the mean is for what's going on. If you have a better idea feel free to suggest it. Bonus points if it's easy to calculate and display.
    - radford-neal 7y ago
      I'm not sure I understand what you're suggesting, but maybe you're thinking that you will run the benchmark for some number of repetitions, and then at some later time run it again, for that number of repetitions, and so forth for some number of times? If nothing changes from one time to another, you might as well combine all the results, and display the overall mean, or a histogram, or whatever. If you think that something may have changed from one time to another (eg, ambient temperature, which affects what clock frequency the processor may have been throttled down to), then you've got a complicated problem, if you're really trying to get to the bottom of the situation. But once again, if you're looking at how the sets of runs at different times differ, confidence intervals do not indicate the spread of the data. Perhaps you're computing quantiles instead, which would be sensible, but then incorrectly referring to them as "confidence intervals", when they're not?
      - Retric 7y ago
        > If nothing changes Something changes, but not necessarily something important for every code path. Test could be to run optimization A x 1,000 then optimization B x 1,000. So now I Run the test, edit optimization B, and just run the full test again. Running A again might seem wasteful, but what if something else where to affect A and B, if I just test B then it’s not obvious something other than my code changed. Perhaps as you say the heat sink is failing and my CPU is now down clocking. The goal is to have a simple test that shows if I am improving things while also isolating that I am measuring what I think I am measuring. That’s not to say I will actually run several long tests repeatedly, but I may re run a test expecting similar results. Anyway, all of the above has nothing to do with why as want a confidence interval. It’s useful for getting a ballpark feeling of the underlying data.

Retric 7y ago

I think I figured out the confusion. Benchmarking is a comparison and each test is a fixed sample size. A test might collect 100 or 100,000 samples by running the task repeatedly, but the data is not aggregated across multiple tests as the data is generally not saved. So, running the benchmark test again keep’s n the same when calculating the confidence interval. This means error bars are showing the calculated mean and standard * arbitrary scaling factor all lined up on an easy to read graph. PS: A lot of statistical tools are poor fits as the data is rarely a normal distribution. But, rigger is not the goal here.

radford-neal 7y ago
You're still not getting it. Confidence intervals are not what you want to look at. Neither are error bars. They would be relevant if you wanted to know the mean performance, and wondered how accurately you had estimated this mean. But as far as I can tell, that's what you want to know.
- Retric 7y ago
  It's not how accurately you had estimated this mean, but how representative the mean is for what's going on. If you have a better idea feel free to suggest it. Bonus points if it's easy to calculate and display.
  - radford-neal 7y ago
    I'm not sure I understand what you're suggesting, but maybe you're thinking that you will run the benchmark for some number of repetitions, and then at some later time run it again, for that number of repetitions, and so forth for some number of times? If nothing changes from one time to another, you might as well combine all the results, and display the overall mean, or a histogram, or whatever. If you think that something may have changed from one time to another (eg, ambient temperature, which affects what clock frequency the processor may have been throttled down to), then you've got a complicated problem, if you're really trying to get to the bottom of the situation. But once again, if you're looking at how the sets of runs at different times differ, confidence intervals do not indicate the spread of the data. Perhaps you're computing quantiles instead, which would be sensible, but then incorrectly referring to them as "confidence intervals", when they're not?
    - Retric 7y ago
      > If nothing changes Something changes, but not necessarily something important for every code path. Test could be to run optimization A x 1,000 then optimization B x 1,000. So now I Run the test, edit optimization B, and just run the full test again. Running A again might seem wasteful, but what if something else where to affect A and B, if I just test B then it’s not obvious something other than my code changed. Perhaps as you say the heat sink is failing and my CPU is now down clocking. The goal is to have a simple test that shows if I am improving things while also isolating that I am measuring what I think I am measuring. That’s not to say I will actually run several long tests repeatedly, but I may re run a test expecting similar results. Anyway, all of the above has nothing to do with why as want a confidence interval. It’s useful for getting a ballpark feeling of the underlying data.