10 ms·

In my work, I have never discarded outlier values. We investigate them to make sure we're not doing something boneheaded, but once I'm confident it's legit, I a

In my work, I have never discarded outlier values. We investigate them to make sure we're not doing something boneheaded, but once I'm confident it's legit, I always include it. I agree that trimmed-means are L-moments are better at characterizing a distribution. But! I had to look them up just now, and I have never seen a computer systems paper which used them. I don't know if most readers would be able to intuit meaning from them. My claim is not that mean and standard deviation is the best way to characterize computer systems performance data, but that for my work, it's better than mean and a confidence interval.

srean 9y ago
> In my work, I have never discarded outlier values. This is great to hear, hope everyone follows this. > I have never seen a computer systems paper which used them. I don't know if most readers would be able to intuit meaning from them. That's rather unfortunate, is it not ? Current practice is broken. So should one not strive to fix it and introduce the community to better tools, champion better tools. I am aware I am being harsh here but, correctness be damned, lets follow the crowd does not show the community in stellar light. If trimmed mean is too exotic, even median and inter-quantile distance would be fine. > My claim is not that mean and standard deviation is the best way to characterize computer systems performance data, but that for my work, it's better than mean and a confidence interval. I have two comments to make (i) Its not the case that mean and std are mostly OK but rather suboptimal. I would be fine with that. The thing is these can be horribly wrong and unrepresentative. Seemingly innocuous looking distributions have infinite variance. Since you use a lot of Gaussian based tools, you would be familiar with a t-test and the t-distribution. The variance of a t-distribution with 2 degrees of freedom is infinite, any finite number one reports for the std would be wrong by an infinite order of magnitude. > it's better than mean and a confidence interval. (ii) I am having trouble with this one. Assuming that confidence intervals have been computed using the Gaussian assumption, width of the confidence interval would be a scaled up version of the standard deviation, nothing fundamentally different. Instead of std, you get, say, 3x of std. When they are bad, both are equally bad. I am not against the use of Gaussian, sometimes it is very reasonable, but if correctness is a goal, then it needs to be demonstrated that it is not a wrong thing to use. There are standard techniques to verify that using Gaussian based stats would be fine. I would be sympathetic if it was the case that we don't know how to handle departures from Gaussian, or it is very expensive to handle that. Neither of these are true, and haven't been true since the 80s. in fact earlier.

srean 9y ago

> In my work, I have never discarded outlier values. This is great to hear, hope everyone follows this. > I have never seen a computer systems paper which used them. I don't know if most readers would be able to intuit meaning from them. That's rather unfortunate, is it not ? Current practice is broken. So should one not strive to fix it and introduce the community to better tools, champion better tools. I am aware I am being harsh here but, correctness be damned, lets follow the crowd does not show the community in stellar light. If trimmed mean is too exotic, even median and inter-quantile distance would be fine. > My claim is not that mean and standard deviation is the best way to characterize computer systems performance data, but that for my work, it's better than mean and a confidence interval. I have two comments to make (i) Its not the case that mean and std are mostly OK but rather suboptimal. I would be fine with that. The thing is these can be horribly wrong and unrepresentative. Seemingly innocuous looking distributions have infinite variance. Since you use a lot of Gaussian based tools, you would be familiar with a t-test and the t-distribution. The variance of a t-distribution with 2 degrees of freedom is infinite, any finite number one reports for the std would be wrong by an infinite order of magnitude. > it's better than mean and a confidence interval. (ii) I am having trouble with this one. Assuming that confidence intervals have been computed using the Gaussian assumption, width of the confidence interval would be a scaled up version of the standard deviation, nothing fundamentally different. Instead of std, you get, say, 3x of std. When they are bad, both are equally bad. I am not against the use of Gaussian, sometimes it is very reasonable, but if correctness is a goal, then it needs to be demonstrated that it is not a wrong thing to use. There are standard techniques to verify that using Gaussian based stats would be fine. I would be sympathetic if it was the case that we don't know how to handle departures from Gaussian, or it is very expensive to handle that. Neither of these are true, and haven't been true since the 80s. in fact earlier.