4 ms·
I'm not a statistician but even so I think this article makes assumptions that may not hold up for computer science. The first thing to do is plot your data. If
by skybrian 10y ago
I'm not a statistician but even so I think this article makes assumptions that may not hold up for computer science. The first thing to do is plot your data. If it doesn't look like a bell curve, it's unlikely that common statistical calculations (which assume something close to gaussian) apply here.
If you're doing benchmarking, another common model is a peak at a minimum value (when everything goes right) and a long tail, due to various events like cache misses that always slow things down, but don't happen in every test run.
On a system with multiple programs running (a typical desktop), taking the mean is meaningless - this just adds noise due to activity unrelated to your program. You'd be better off taking the minimum, which with enough test runs should capture all the events that happen every time and none of the events that don't.
The median or 95% percentile might also be useful if you're investigating events that don't happen every time. But if you want to know about cold start performance (for example), maybe the best thing to do would be to flush your caches before every test run, so the events you're interested in are events that happen every time.
- srean 10y ago> If it doesn't look like a bell curve, it's unlikely that common statistical calculations (which assume something close to gaussian) apply here. The key word in there is common. There is an entire industry of statistical techniques that do not require Gaussian assumption or for that matter any parametric assumption. I strongly feel it is time to retire the Gaussian distribution from the space it occupies. Discovering and studying Gaussian distribution and the bog standard central limit theorem should be considered one of mankind's crowning achievements. They deserve to be put on a pedestal to appreciate their elegance, but when rubber meets the road one has to open ones mind to look beyond. Appearance of the Gaussian distribution is rarely as normal as many expect/claim it to be (I blame the stats education machinery for this), nor was it invented by Gauss. In fact Gauss used it as a post-hoc justification for backing the least-squares method. His original motivation for least squares was simplicity and convenience, not the normal distribution or CLT or for that matter the Gauss-Markov theorem.
- thanatropism 10y agoThe Gaussian distribution is central in continuous-time models for different reasons: https://almostsure.wordpress.com/2010/04/13/levys-characterization-of-brownian-motion/ https://almostsure.wordpress.com/2010/04/13/levys-characteri... Basically any reasonable* stochastic continuous process is driven by a brownian motion. Also: discontinuous processes are more or less* the sum of a brownian motion and a poisson-type process. https://en.wikipedia.org/wiki/L%C3%A9vy_process#L.C3.A9vy.E2.80.93Khintchine_representation https://en.wikipedia.org/wiki/L%C3%A9vy_process#L.C3.A9vy.E2... (* Much details about filtrations, Banach spaces yadda yadda omitted)
- srean 10y agoYes indeed, but the Brownian motion story is weaker than the CLT story. Lot more conditions required, you have to look at it the right scale, in the right way ... then stochastic processes look very much like a Brownian motion. Well, technically the bog standard CLT is a special case of this, hence has a simpler story.
- GFK_of_xmaspast 10y agoI am a huge fan of nonparametric statistics, but lots of non-Gaussian distributions are often 'good enough' for normality assumptions to be ok.