3 ms·

Mean and standard deviation seem downright dangerous to use, especially to understand latencies. Both mean and std are very sensitive to extreme values, and the

by srean 9y ago

Mean and standard deviation seem downright dangerous to use, especially to understand latencies. Both mean and std are very sensitive to extreme values, and these extreme values tend to be frequent enough that shoving them under the 'outlier' carpet does not instill much confidence. Looking at multiple quantiles (if not all of them) would be more sound. But if one is compelled to reduce the data to 2 numbers, one can go with trimmed-means, and L-moments or some other robust estimate of location and scale (EDIT. for example, median and interquartile. Just that trimmed mean is more efficient, both statistically and computationally). Deficiencies of mean and std are easy to demonstrate. Just draw multiple samples of different sizes from a Pareto distribution (some other long tailed distribution would work too) and observe how drastically the mean and std varies among those samples. Gaussian distribution based methods, for example ANOVA, had their moment of glory in centuries past (good stats has since moved on). They are still fantastic tools when the assumptions they make are true (this is rather rare) but they break badly when those assumptions are violated a tiny wee bit. One can easily end up making a wrong decision. At the very least one should verify that the Gaussian assumption holds before believing those statistics. There are many post-Gaussian statistics that one can use, if it turns out that the data is not Gaussian.

scott_s 9y ago
In my work, I have never discarded outlier values. We investigate them to make sure we're not doing something boneheaded, but once I'm confident it's legit, I always include it. I agree that trimmed-means are L-moments are better at characterizing a distribution. But! I had to look them up just now, and I have never seen a computer systems paper which used them. I don't know if most readers would be able to intuit meaning from them. My claim is not that mean and standard deviation is the best way to characterize computer systems performance data, but that for my work, it's better than mean and a confidence interval.
- srean 9y ago
  > In my work, I have never discarded outlier values. This is great to hear, hope everyone follows this. > I have never seen a computer systems paper which used them. I don't know if most readers would be able to intuit meaning from them. That's rather unfortunate, is it not ? Current practice is broken. So should one not strive to fix it and introduce the community to better tools, champion better tools. I am aware I am being harsh here but, correctness be damned, lets follow the crowd does not show the community in stellar light. If trimmed mean is too exotic, even median and inter-quantile distance would be fine. > My claim is not that mean and standard deviation is the best way to characterize computer systems performance data, but that for my work, it's better than mean and a confidence interval. I have two comments to make (i) Its not the case that mean and std are mostly OK but rather suboptimal. I would be fine with that. The thing is these can be horribly wrong and unrepresentative. Seemingly innocuous looking distributions have infinite variance. Since you use a lot of Gaussian based tools, you would be familiar with a t-test and the t-distribution. The variance of a t-distribution with 2 degrees of freedom is infinite, any finite number one reports for the std would be wrong by an infinite order of magnitude. > it's better than mean and a confidence interval. (ii) I am having trouble with this one. Assuming that confidence intervals have been computed using the Gaussian assumption, width of the confidence interval would be a scaled up version of the standard deviation, nothing fundamentally different. Instead of std, you get, say, 3x of std. When they are bad, both are equally bad. I am not against the use of Gaussian, sometimes it is very reasonable, but if correctness is a goal, then it needs to be demonstrated that it is not a wrong thing to use. There are standard techniques to verify that using Gaussian based stats would be fine. I would be sympathetic if it was the case that we don't know how to handle departures from Gaussian, or it is very expensive to handle that. Neither of these are true, and haven't been true since the 80s. in fact earlier.
nerdponx 9y ago
So use the median and the median absolute deviation, or use them in conjunction with mean and standard deviation. Please don't discourage people from reporting distributional statistics.

by srean 9y ago

scott_s 9y ago

In my work, I have never discarded outlier values. We investigate them to make sure we're not doing something boneheaded, but once I'm confident it's legit, I always include it. I agree that trimmed-means are L-moments are better at characterizing a distribution. But! I had to look them up just now, and I have never seen a computer systems paper which used them. I don't know if most readers would be able to intuit meaning from them. My claim is not that mean and standard deviation is the best way to characterize computer systems performance data, but that for my work, it's better than mean and a confidence interval.

srean 9y ago
> In my work, I have never discarded outlier values. This is great to hear, hope everyone follows this. > I have never seen a computer systems paper which used them. I don't know if most readers would be able to intuit meaning from them. That's rather unfortunate, is it not ? Current practice is broken. So should one not strive to fix it and introduce the community to better tools, champion better tools. I am aware I am being harsh here but, correctness be damned, lets follow the crowd does not show the community in stellar light. If trimmed mean is too exotic, even median and inter-quantile distance would be fine. > My claim is not that mean and standard deviation is the best way to characterize computer systems performance data, but that for my work, it's better than mean and a confidence interval. I have two comments to make (i) Its not the case that mean and std are mostly OK but rather suboptimal. I would be fine with that. The thing is these can be horribly wrong and unrepresentative. Seemingly innocuous looking distributions have infinite variance. Since you use a lot of Gaussian based tools, you would be familiar with a t-test and the t-distribution. The variance of a t-distribution with 2 degrees of freedom is infinite, any finite number one reports for the std would be wrong by an infinite order of magnitude. > it's better than mean and a confidence interval. (ii) I am having trouble with this one. Assuming that confidence intervals have been computed using the Gaussian assumption, width of the confidence interval would be a scaled up version of the standard deviation, nothing fundamentally different. Instead of std, you get, say, 3x of std. When they are bad, both are equally bad. I am not against the use of Gaussian, sometimes it is very reasonable, but if correctness is a goal, then it needs to be demonstrated that it is not a wrong thing to use. There are standard techniques to verify that using Gaussian based stats would be fine. I would be sympathetic if it was the case that we don't know how to handle departures from Gaussian, or it is very expensive to handle that. Neither of these are true, and haven't been true since the 80s. in fact earlier.

nerdponx 9y ago

So use the median and the median absolute deviation, or use them in conjunction with mean and standard deviation. Please don't discourage people from reporting distributional statistics.