3 ms·

Usually I don't do any statistical tests when benchmarking and optimizing. ("If your result depends on statistics then you need a better experiment.” Oft quote

Usually I don't do any statistical tests when benchmarking and optimizing. ("If your result depends on statistics then you need a better experiment.” Oft quoted remark attributed to Rutherford.) s/experiment/optimization/ I'm happy with min, max and average. max is actually often the most important. On the other hand the point of view in this paper is interesting: "Violating the normality assumption may be the lesser of two evils" (1) If you really feel the urge to do statistical tests then do it. (I do not work in medicine or anything dangerous at all) https://link.springer.com/article/10.3758/s13428-021-01587-5 https://link.springer.com/article/10.3758/s13428-021-01587-5 (1)

dan-robertson 4y ago
The only extra statistic in the article not on your list is standard deviation. I don’t know whom this comment is aimed at.
- hackandthink 3y ago
  The comment is a bit abrupt. The article questions the normal distribution assumption for benchmark timings. It says that this assumption would be important for the "standard error of the mean": "Your error should go down as the square root of the number of measures." Standard distribution is not even necessary for this: https://en.wikipedia.org/wiki/Standard_error https://en.wikipedia.org/wiki/Standard_error However, normal distributions actually behave benignly and are a prerequisite for some statistical tests. This is where my first comment gets in, that for my benchmarking statistical tests are not necessary at all ... --- The author writes that he only measures minimum and average, but the maximum values are of course also important and worth to measure. The author's key point is that without a normal distribution, the maximum values can be highly scattered. I agree and basically it is always nice to know the distribution of the random variable.

dan-robertson 4y ago

The only extra statistic in the article not on your list is standard deviation. I don’t know whom this comment is aimed at.

hackandthink 3y ago
The comment is a bit abrupt. The article questions the normal distribution assumption for benchmark timings. It says that this assumption would be important for the "standard error of the mean": "Your error should go down as the square root of the number of measures." Standard distribution is not even necessary for this: https://en.wikipedia.org/wiki/Standard_error https://en.wikipedia.org/wiki/Standard_error However, normal distributions actually behave benignly and are a prerequisite for some statistical tests. This is where my first comment gets in, that for my benchmarking statistical tests are not necessary at all ... --- The author writes that he only measures minimum and average, but the maximum values are of course also important and worth to measure. The author's key point is that without a normal distribution, the maximum values can be highly scattered. I agree and basically it is always nice to know the distribution of the random variable.