9 ms·

The Central Limit Theorem and Its Misuse

176 points by lelf 7y ago

fyp 7y ago
Can someone give me an example on how to properly use the central limit theorem in the real world? For example with stuff that are commonly assumed to be bell-curved like test scores, IQ, etc. What are the iid variables being averaged? Each test question?
- throwlaplace 7y ago
  there's a distinction between random variables that are assumed to actually be normal and RVs that we conceptually apply the clt to and then treat as normal. in the case of test scores it could be either. for an individual test it's possible the distribution is roughly normal actually. but if you, for example, look at subsets of all SAT scores and take their means (the mean of the subset) then those averages will be Bell shaped because of the clt. Just to directly answer your question: the iid random variable is the test score or the iq.
- markkvdb 7y ago
  Suppose you have a school with 30 groups of students. All students are randomly assigned to the groups (so independent of their skill at IQ tests). The distribution of measured IQ within classes can be any distribution. But let’s take the 30 averages of the groups. The mean of these averages of groups (asymptotically) follows the normal distribution according to the CLT.
  - throwlaplace 7y ago
    not the mean of the means - just the means themselves. The distribution of the original population of iqs in the school is what you're sampling. Each group is a sample. The distribution of the sample mean approaches a normal according to clt as the sample sizes increase.
  - deleted 7y ago
    [deleted]
- raegis 7y ago
  He's the example I have my students do in class. Roll a die repeatedly and tally the results. You'll get a (roughly) uniform distribution of 1s, 2s, 3s, 4s, 5s, and 6s. Now to illustrate the CLT, you roll a die 50 times, and average the result. AND your 300 classmates do the same. If you tally the 301 averages, the distribution of the averages will not be uniform but bell-shaped, with average (approximately) 3.5. The CLT says (roughly) the distribution of the averages will be approximately normal, regardless of the original distribution.
  - ganzuul 7y ago
    Short and concise, as things should be. Thank you. :)
  - laichzeit0 7y ago
    Here’s something that’s bothered me for a while. Why is it necessary to roll it 50 times? Why can’t I roll it just once? Does each X_i have to be a sample greater than 1 and if so is there some kind of minimum required for each random sample? Does CLT still hold if each random sample’s size is 1?
    - zosima 7y ago
      As the number of samples approach infinity the mean of all the iid samples will become normally distributed. So it will neither hold strictly for n=50 or n=1, but n=50 may be a better approximation of infinity :)
      - laichzeit0 7y ago
        As I understood it each X_i is a random sample and the n refers to X_n, that is, n random samples. But what is the size of each sample? In the parent it was 50. The dice is rolled 50 times by each person i, and the outcomes are given by the random variable X_i. Also, there are 300 people rolling a dice, so n=300, thus 300 random variables each composed of 50 dice roll outcomes. I’m curious why each X_i has to be a sample of size 50 and can we just have each person roll the dice 1 time? Maybe we need 50*300 people to roll the dice one time now? Does the CLT work when each X_i is a random sample of size 1?
- roenxi 7y ago
  The CLT can be used to justify modelling an effect with a normal distribution and raises questions about model assumptions when normal distributions don't appear where they should. The Normal Distribution is special because it is the highest-entropy distribution that is completely characterised by mean and standard deviation [0]. So what the CLT is saying is that, surprisingly, sums of i.i.d. random variables lose information from the individual variables quite quickly but reliably retain a little data about mean and second moment. Initially a given X_i has all sorts of information associated with it (higher order moments, other distribution characteristics, etc) that disappears as many X_i are summed together. All that is left is information about a mean and variance. So say I have a situation where a large number of i.i.d. variables are going to be summed together. The CLT tells me that summing N of these variables together is going to be similar to summing together an equivalent number of normally distributed variables (!!). This is because the sum variable is equal in value to n * \bar{x}, but the CLT imposes a distribution on \frac{\bar{x}}{n}. This justifies why the normal turns up everywhere in practical measurements. A lot of measurements (say, number of people at the beach) are probably really measures of a sum of random variables (maybe there is some non-normal variables that captures the chance a given person goes to the beach). So if the number of people at the beach turns out to be normally distributed (maybe I measure it each day for a few weeks) it isn't shocking. If the total number isn't normally distributed then that implies that there is no i.i.d. variable representing probability a given individual goes to the beach (eg, high correlation between the individual variables). I didn't quite fail any of the statistics courses I've ever done, YMMV. [0] https://en.wikipedia.org/wiki/Normal_distribution#Maximum_entropy https://en.wikipedia.org/wiki/Normal_distribution#Maximum_en...
MichaelRazum 7y ago
Not 100% sure, but for me "The three coins (non) example" is false. You could easily constract a random variable wich flips the free coins at random. Then you get again an iid sequence. I'm writing this basically because I think it is troubeling how "blogs" work. Explaining, lets say complex stuff, with simple diagrams. Especially in this case it even looks wrong and gives you the wrong intuition.
- Schiphol 7y ago
  The RV you suggest you could construct is not how the example works. From the article: >Let's consider the following scenario: say we have three coins with different biases (their probability of coming up heads): 0.4, 0.5 and 0.6. We pick one of the three coins at random, toss it 300 times and count the number of heads. So, not choosing a random coin every time, but choosing it once and tossing it 300 times.
  - MichaelRazum 7y ago
    Maybe that is correct. What I'm just missing is a simple "proof" or at least an idea of a proof why some assumptions are violated. In this case you could a gain argue: picking a coin random and then toss 300 times --> Multidimensional CLT. Not saying the article is wrong. I'm just saying, where should be a proof or at least a bit more evidence... Just a simple plot is not enough from my perspective
    - Schiphol 7y ago
      The CLT holds for independent, identically distributed RVs. In the three-coin example the RVs are independent but not identically distributed: there are three distributions, one for each coin.
      - bonoboTP 7y ago
        No, it's the opposite: the 300 tosses are identically distributed, but they are not independent. They aren't independent because they all depend on a common cause: the choice of which coin to flip. Their property is also called "exchangeable" and it implies identical marginal distributions.
pmiller2 7y ago
The TL;DR here seems to be that the CLT is false when it’s hypotheses are violated. This does not seem to be news.
- bsaul 7y ago
  you’re kind of missing the point imo... the goal is to explain misconceptions and misuses of the CLT. it’s useful to people that actually don’t understand properly the hypothesis of the CLT.
hgibbs 7y ago
Interestingly, the O(1/sqrt(n)) error in the CLT provided by the Berry-Eseen Theorem is the best one can do for general sequences of our RVs. However, if you are happy to put certain 'bumps' in the normal distribution then you can improve the error to O(n^(-r/2)), where I think r is some integer depending on the smoothness of your random variables. These bumps are called an edegworth expansion.
manthideaal 7y ago
The example of misuse of the CLT illustrates that a mixture of normals is not a normal distribution. For example the height of people is a mixture of two normals, one for men and the other for women, this mixture has two modes and so it is not bell shaped and is not a normal distribution.
- elcomet 7y ago
  CLT is about the sum. In your case, the sum of heights will still converge to the normal distribution of you take the height of n random people (man or woman)
  - BeetleB 7y ago
    > In your case, the sum of heights will still converge to the normal distribution of you take the height of n random people (man or woman) Yes and no. The sample must be independent and identically distributed. In your case the "identical" part is not correct, as men and women have different distributions (both are normal but with different mean and std). However, if both distributions are normal, then their sum is normal (even with different mean and std). The fact that the sum is normal in this case has nothing to do with the CLT - it's just a quirk of the normal distribution that the sum is normal. Had men/women had non-normal distributions with different means/stds, then the sum would not be normal.
    - elcomet 7y ago
      If you sample from the global distribution (all men and women of earth) then samples are identically distributed. It's just a new distribution that is not gaussian, but the sum of samples will converge to a gaussian.
      - BeetleB 7y ago
        I concede your point.
- joker3 7y ago
YeGoblynQueenne 7y ago
This is a nice article but if it's really meant as an explanation of CLT for people who don't know much about it, then the (pretty) graphs should have their axes labelled clearly and the symbols used in the formulae presented should be also clearly explained (e.g. the article uses E for expectation without immediately explaining it). It's no use "explaining" what is supposed to be a novel concept by presenting it in a way that is difficult to understand if one is not already familiar with it.
wodenokoto 7y ago
What does the histogram of cointosses represent?[1] We have 300 coin throws, so basically a list of H's T's. My go-to for a histogram would be to have the buckets represent categories. I Throw a coin 300 times and plot the number of heads on one column, and the number of tails on the other column. Next I would maybe do a bucket-size of 10-throws and count outcomes chronologically in each bucket. Lastly, I would consider the same, but only with comulative counts. [1] https://lambdaclass.com/data_etudes/central_limit_theorem_misuse/img/1.png https://lambdaclass.com/data_etudes/central_limit_theorem_mi...
- gbrown 7y ago
  It’s the estimated (via simulation) sampling distribution of the number of heads observed out of three hundred coin tosses.
eanzenberg 7y ago
CLT is abused among data scientists quite often, and usually due to the points in the article. - Convergence takes infinite time when the sampling variable has infinite variance (imagine a random number generator). - While the distribution of an aggregate statistic tends to a Gaussian, most underlying distributions are not and most stats is done using the underlying raw data. Junior data scientists assume everything is Gaussian (sometimes attributing it to CLT) when doing linear regression when most distributions being modeled are not. Then, analysis done using variance on the coefficients is meaningless because the assumptions are incorrect.
- kxxsc 7y ago
  This comment is surprising to me. Most data scientists use results and measures of statistical significance provided by the program they're using, which accounts for the distribution used. Do you have examples where either data scientists are not presenting aggregate statistics or where someone is using the wrong kinds of p-values? To your specific examples - the coefficient of a linear regression is distributed normally (?). Similarly, we know the expected distribution of most maximum likelihood estimators (logistic regression, etc.), and programs will give you the right p-value. Of course, omitted variable bias is still a problem and it is possible to mis-specify your model. However, I think most data scientists are presenting aggregate statistics (means, regression coefficients) like you said, and that we have a pretty good handle on the underlying distributions.
  - throwlaplace 7y ago
    >which accounts for the distribution used what does this mean? are you trying to say the code uses the empirical distribution as a proxy for the true distribution?
  - wbl 7y ago
    Mispecification error is hard to deal with.
  - eanzenberg 7y ago
    For example, there were times where features were excluded because their "p-value was too large to be significant", regardless that the underlying distribution was not Gaussian. p-value from a t-test, like in lots of regression software, requires Gaussian distributions.
bobbyd2323 7y ago
There are MANY CLTs. In general there is a memory-moment trade-off in assumptions for a given CLT. For example, you can relax the iid assumption to allow dependence between observations, such as a mixing condition, with the cost being you need a little bit stronger moment condition where the moment condition is a function of the mixing condition - more dependence (up to a point) requires a stronger moment.
xpe 7y ago
Dear authors: Please add timestamps showing when the article was first published and updated. This helps situate the article in a larger context.
- paulddraper 7y ago
  Agreed. It appears to have been published yesterday-ish. [1] [1] https://web.archive.org/web/*/https://lambdaclass.com/data_etudes/central_limit_theorem_misuse/ https://web.archive.org/web/*/https://lambdaclass.com/data_e...

176 points by lelf 7y ago

fyp 7y ago

Can someone give me an example on how to properly use the central limit theorem in the real world? For example with stuff that are commonly assumed to be bell-curved like test scores, IQ, etc. What are the iid variables being averaged? Each test question?

throwlaplace 7y ago
there's a distinction between random variables that are assumed to actually be normal and RVs that we conceptually apply the clt to and then treat as normal. in the case of test scores it could be either. for an individual test it's possible the distribution is roughly normal actually. but if you, for example, look at subsets of all SAT scores and take their means (the mean of the subset) then those averages will be Bell shaped because of the clt. Just to directly answer your question: the iid random variable is the test score or the iq.
markkvdb 7y ago
Suppose you have a school with 30 groups of students. All students are randomly assigned to the groups (so independent of their skill at IQ tests). The distribution of measured IQ within classes can be any distribution. But let’s take the 30 averages of the groups. The mean of these averages of groups (asymptotically) follows the normal distribution according to the CLT.
- throwlaplace 7y ago
  not the mean of the means - just the means themselves. The distribution of the original population of iqs in the school is what you're sampling. Each group is a sample. The distribution of the sample mean approaches a normal according to clt as the sample sizes increase.
- deleted 7y ago
  [deleted]
raegis 7y ago
He's the example I have my students do in class. Roll a die repeatedly and tally the results. You'll get a (roughly) uniform distribution of 1s, 2s, 3s, 4s, 5s, and 6s. Now to illustrate the CLT, you roll a die 50 times, and average the result. AND your 300 classmates do the same. If you tally the 301 averages, the distribution of the averages will not be uniform but bell-shaped, with average (approximately) 3.5. The CLT says (roughly) the distribution of the averages will be approximately normal, regardless of the original distribution.
- ganzuul 7y ago
  Short and concise, as things should be. Thank you. :)
- laichzeit0 7y ago
  Here’s something that’s bothered me for a while. Why is it necessary to roll it 50 times? Why can’t I roll it just once? Does each X_i have to be a sample greater than 1 and if so is there some kind of minimum required for each random sample? Does CLT still hold if each random sample’s size is 1?
  - zosima 7y ago
    As the number of samples approach infinity the mean of all the iid samples will become normally distributed. So it will neither hold strictly for n=50 or n=1, but n=50 may be a better approximation of infinity :)
    - laichzeit0 7y ago
      As I understood it each X_i is a random sample and the n refers to X_n, that is, n random samples. But what is the size of each sample? In the parent it was 50. The dice is rolled 50 times by each person i, and the outcomes are given by the random variable X_i. Also, there are 300 people rolling a dice, so n=300, thus 300 random variables each composed of 50 dice roll outcomes. I’m curious why each X_i has to be a sample of size 50 and can we just have each person roll the dice 1 time? Maybe we need 50*300 people to roll the dice one time now? Does the CLT work when each X_i is a random sample of size 1?
roenxi 7y ago
The CLT can be used to justify modelling an effect with a normal distribution and raises questions about model assumptions when normal distributions don't appear where they should. The Normal Distribution is special because it is the highest-entropy distribution that is completely characterised by mean and standard deviation [0]. So what the CLT is saying is that, surprisingly, sums of i.i.d. random variables lose information from the individual variables quite quickly but reliably retain a little data about mean and second moment. Initially a given X_i has all sorts of information associated with it (higher order moments, other distribution characteristics, etc) that disappears as many X_i are summed together. All that is left is information about a mean and variance. So say I have a situation where a large number of i.i.d. variables are going to be summed together. The CLT tells me that summing N of these variables together is going to be similar to summing together an equivalent number of normally distributed variables (!!). This is because the sum variable is equal in value to n * \bar{x}, but the CLT imposes a distribution on \frac{\bar{x}}{n}. This justifies why the normal turns up everywhere in practical measurements. A lot of measurements (say, number of people at the beach) are probably really measures of a sum of random variables (maybe there is some non-normal variables that captures the chance a given person goes to the beach). So if the number of people at the beach turns out to be normally distributed (maybe I measure it each day for a few weeks) it isn't shocking. If the total number isn't normally distributed then that implies that there is no i.i.d. variable representing probability a given individual goes to the beach (eg, high correlation between the individual variables). I didn't quite fail any of the statistics courses I've ever done, YMMV. [0] https://en.wikipedia.org/wiki/Normal_distribution#Maximum_entropy https://en.wikipedia.org/wiki/Normal_distribution#Maximum_en...

MichaelRazum 7y ago

Not 100% sure, but for me "The three coins (non) example" is false. You could easily constract a random variable wich flips the free coins at random. Then you get again an iid sequence. I'm writing this basically because I think it is troubeling how "blogs" work. Explaining, lets say complex stuff, with simple diagrams. Especially in this case it even looks wrong and gives you the wrong intuition.

Schiphol 7y ago
The RV you suggest you could construct is not how the example works. From the article: >Let's consider the following scenario: say we have three coins with different biases (their probability of coming up heads): 0.4, 0.5 and 0.6. We pick one of the three coins at random, toss it 300 times and count the number of heads. So, not choosing a random coin every time, but choosing it once and tossing it 300 times.
- MichaelRazum 7y ago
  Maybe that is correct. What I'm just missing is a simple "proof" or at least an idea of a proof why some assumptions are violated. In this case you could a gain argue: picking a coin random and then toss 300 times --> Multidimensional CLT. Not saying the article is wrong. I'm just saying, where should be a proof or at least a bit more evidence... Just a simple plot is not enough from my perspective
  - Schiphol 7y ago
    The CLT holds for independent, identically distributed RVs. In the three-coin example the RVs are independent but not identically distributed: there are three distributions, one for each coin.
    - bonoboTP 7y ago
      No, it's the opposite: the 300 tosses are identically distributed, but they are not independent. They aren't independent because they all depend on a common cause: the choice of which coin to flip. Their property is also called "exchangeable" and it implies identical marginal distributions.

pmiller2 7y ago

The TL;DR here seems to be that the CLT is false when it’s hypotheses are violated. This does not seem to be news.

bsaul 7y ago
you’re kind of missing the point imo... the goal is to explain misconceptions and misuses of the CLT. it’s useful to people that actually don’t understand properly the hypothesis of the CLT.

hgibbs 7y ago

Interestingly, the O(1/sqrt(n)) error in the CLT provided by the Berry-Eseen Theorem is the best one can do for general sequences of our RVs. However, if you are happy to put certain 'bumps' in the normal distribution then you can improve the error to O(n^(-r/2)), where I think r is some integer depending on the smoothness of your random variables. These bumps are called an edegworth expansion.

manthideaal 7y ago

The example of misuse of the CLT illustrates that a mixture of normals is not a normal distribution. For example the height of people is a mixture of two normals, one for men and the other for women, this mixture has two modes and so it is not bell shaped and is not a normal distribution.

elcomet 7y ago
CLT is about the sum. In your case, the sum of heights will still converge to the normal distribution of you take the height of n random people (man or woman)
- BeetleB 7y ago
  > In your case, the sum of heights will still converge to the normal distribution of you take the height of n random people (man or woman) Yes and no. The sample must be independent and identically distributed. In your case the "identical" part is not correct, as men and women have different distributions (both are normal but with different mean and std). However, if both distributions are normal, then their sum is normal (even with different mean and std). The fact that the sum is normal in this case has nothing to do with the CLT - it's just a quirk of the normal distribution that the sum is normal. Had men/women had non-normal distributions with different means/stds, then the sum would not be normal.
  - elcomet 7y ago
    If you sample from the global distribution (all men and women of earth) then samples are identically distributed. It's just a new distribution that is not gaussian, but the sum of samples will converge to a gaussian.
    - BeetleB 7y ago
      I concede your point.
joker3 7y ago

YeGoblynQueenne 7y ago

This is a nice article but if it's really meant as an explanation of CLT for people who don't know much about it, then the (pretty) graphs should have their axes labelled clearly and the symbols used in the formulae presented should be also clearly explained (e.g. the article uses E for expectation without immediately explaining it). It's no use "explaining" what is supposed to be a novel concept by presenting it in a way that is difficult to understand if one is not already familiar with it.

wodenokoto 7y ago

What does the histogram of cointosses represent?[1] We have 300 coin throws, so basically a list of H's T's. My go-to for a histogram would be to have the buckets represent categories. I Throw a coin 300 times and plot the number of heads on one column, and the number of tails on the other column. Next I would maybe do a bucket-size of 10-throws and count outcomes chronologically in each bucket. Lastly, I would consider the same, but only with comulative counts. [1] https://lambdaclass.com/data_etudes/central_limit_theorem_misuse/img/1.png https://lambdaclass.com/data_etudes/central_limit_theorem_mi...

gbrown 7y ago
It’s the estimated (via simulation) sampling distribution of the number of heads observed out of three hundred coin tosses.

eanzenberg 7y ago

CLT is abused among data scientists quite often, and usually due to the points in the article. - Convergence takes infinite time when the sampling variable has infinite variance (imagine a random number generator). - While the distribution of an aggregate statistic tends to a Gaussian, most underlying distributions are not and most stats is done using the underlying raw data. Junior data scientists assume everything is Gaussian (sometimes attributing it to CLT) when doing linear regression when most distributions being modeled are not. Then, analysis done using variance on the coefficients is meaningless because the assumptions are incorrect.

kxxsc 7y ago
This comment is surprising to me. Most data scientists use results and measures of statistical significance provided by the program they're using, which accounts for the distribution used. Do you have examples where either data scientists are not presenting aggregate statistics or where someone is using the wrong kinds of p-values? To your specific examples - the coefficient of a linear regression is distributed normally (?). Similarly, we know the expected distribution of most maximum likelihood estimators (logistic regression, etc.), and programs will give you the right p-value. Of course, omitted variable bias is still a problem and it is possible to mis-specify your model. However, I think most data scientists are presenting aggregate statistics (means, regression coefficients) like you said, and that we have a pretty good handle on the underlying distributions.
- throwlaplace 7y ago
  >which accounts for the distribution used what does this mean? are you trying to say the code uses the empirical distribution as a proxy for the true distribution?
- wbl 7y ago
  Mispecification error is hard to deal with.
- eanzenberg 7y ago
  For example, there were times where features were excluded because their "p-value was too large to be significant", regardless that the underlying distribution was not Gaussian. p-value from a t-test, like in lots of regression software, requires Gaussian distributions.

bobbyd2323 7y ago

There are MANY CLTs. In general there is a memory-moment trade-off in assumptions for a given CLT. For example, you can relax the iid assumption to allow dependence between observations, such as a mixing condition, with the cost being you need a little bit stronger moment condition where the moment condition is a function of the mixing condition - more dependence (up to a point) requires a stronger moment.

xpe 7y ago

Dear authors: Please add timestamps showing when the article was first published and updated. This helps situate the article in a larger context.

paulddraper 7y ago
Agreed. It appears to have been published yesterday-ish. [1] [1] https://web.archive.org/web/*/https://lambdaclass.com/data_etudes/central_limit_theorem_misuse/ https://web.archive.org/web/*/https://lambdaclass.com/data_e...