3 ms·

In general it is a misconception that real data always follows a normal distribution. It is true that if you sum many /independent/ random quantities, then the

In general it is a misconception that real data always follows a normal distribution. It is true that if you sum many /independent/ random quantities, then the result is approximately normal (e.g., the central limit theorem and generalizations). But real data tends not to be independent. Many real world quantities follow extremely skewed distributions. E.g, Zipf's law, Korcak's law, Pareto's laws. For a concrete example, if you look at the distribution of the number of friends users have on social networks, you might expect that 95% of people have the mean number of friends +/- a few standard deviations (since this would be the case for a normal distribution). It would be virtually impossible for someone with a number of friends that is say thousands of standard deviations away to exist, yet there will be many such users in social networks (celebrities, bot networks, etc). In reality, the empirical distribution in this case follows an extremely skewed distribution.

stdbrouw 10y ago
And yet a Pareto distribution still has a mean, and the sampling distribution of that mean is approximately normally distributed. Of course I'm not claiming that you can just pretend that a Pareto distribution is a normal distribution, but statistical tests are generally concerned with differences in means (group A does on average 25% better than group B) so it's the sampling distribution we're interested in, not the parent distribution. You make a good point about autocorrelation and dependent data, but that's a very different issue. To riff on your example about social networks, you'd have dependent data if you're trying to see what kind of news articles people like to read, if those preferences turn out to be mostly guided by what friends are reading.
- hrzn 10y ago
  "And yet a Pareto distribution still has a mean, and the sampling distribution of that mean is approximately normally distributed." This is often wrong. The central limit theorem requires finite variance and some (yet common) Pareto distributions have infinite variance.
dnautics 10y ago
> It is true that if you sum many /independent/ random quantities, then the result is approximately normal... Don't forget defined variances! The end result could be more generally levy-alpha (alpha < 2.0), as it is with many financial instruments.... Normality requires defined variance.

stdbrouw 10y ago

And yet a Pareto distribution still has a mean, and the sampling distribution of that mean is approximately normally distributed. Of course I'm not claiming that you can just pretend that a Pareto distribution is a normal distribution, but statistical tests are generally concerned with differences in means (group A does on average 25% better than group B) so it's the sampling distribution we're interested in, not the parent distribution. You make a good point about autocorrelation and dependent data, but that's a very different issue. To riff on your example about social networks, you'd have dependent data if you're trying to see what kind of news articles people like to read, if those preferences turn out to be mostly guided by what friends are reading.

hrzn 10y ago
"And yet a Pareto distribution still has a mean, and the sampling distribution of that mean is approximately normally distributed." This is often wrong. The central limit theorem requires finite variance and some (yet common) Pareto distributions have infinite variance.

dnautics 10y ago

> It is true that if you sum many /independent/ random quantities, then the result is approximately normal... Don't forget defined variances! The end result could be more generally levy-alpha (alpha < 2.0), as it is with many financial instruments.... Normality requires defined variance.