3 ms·

> Statisticians are quick to reach for the Central Limit Theorem, but I think there’s a deeper, more intuitive, more powerful reason. > The Normal Distribution

> Statisticians are quick to reach for the Central Limit Theorem, but I think there’s a deeper, more intuitive, more powerful reason. > The Normal Distribution is your best guess if you only know the mean and the variance of your data. This is putting the cart before the horse, for sure. The reason why you only know the mean and the variance of your data is because you chose to summarize your data that way. And, the reason why you chose to summarize your data that way is in order to get the normal distribution as the maximum entropy distribution. The normal distribution appears in a lot of places because it is the limiting case of many other distributions, this is the central limit theorem. It is very easy to work with the normal distribution because you can add or subtract a bunch of normal distributions and the result is just another normal distribution. You can add or subtract a bunch of other distributions and the resulting distribution will often be more normal. You can do a lot of work with the normal distribution using linear algebra techniques. So, you choose to measure mean and variance in order to make the math easier. This does not always result in the best outcome. For example, if you need more robust statistics, you might go for median and average deviation, rather than mean and variance. Then when you choose the maximum entropy distribution from the result, you end up with the Laplace distribution. The Laplace distribution is very inconvenient to work with mathematically, unlike the normal distribution.

yetanothermonk 6y ago
Very good points. The Normal Distribution has very nice properties, which adds to its popularity, which, if I understand you correctly, adds to the significance we place on mean and variance.
- adenadel 6y ago
  I wouldn't say that the existence of the normal distribution is necessarily the reason that we place significance on the mean and variance. The mean is incredibly natural to define when you move to measure theoretic probability (simply the integral of a function, i.e. a random variable, with respect to some measure). When you take this point of view, random variables with particular moments existing are simply functions in L^p. Further, when you move on to proving the CLT the moments give you properties of the characteristic functions that allow you to prove the CLT. These are all deep connections. There's another interpretation of the mean (and conditional expectation) as quantities minimizing squared error. It's not surprising that squared error and variance are so similar and that these are connected.
  - yetanothermonk 6y ago
    Great comment! You touched on something that got me wondering: do you know of something similar to MGF that determines a distribution but is not based on expected values?
    - clircle 6y ago
      MGFs do not determine distributions (The Student's t distribution does not have an MGF). CDFs and characteristic functions determine distributions.
    - adenadel 6y ago
      Well, characteristic functions are more general than MGFs (in that they are always finite) and have very useful properties, but they have a similar definition and are also based on an expectation. This may be pedantic, but an object that determines a distribution that isn't based on expected values is the CDF. In my first probability course in undergrad I think we defined probability mass functions and probability density functions first and defined CDFs in terms of them, but from a measure theoretic point of view, the CDF is more fundamental since it is defined for continuous and discrete distributions (and also for distributions that are neither).
      - contravariant 6y ago
        I'd probably choose the probability measure itself, rather than the CDF, just because you won't always have a space with a nice order topology. Of course the expectation value and the probability measure are really two sides of the same coin, the expectation value being equivalent to integration with respect to the probability measure.
jbay808 6y ago
> This is putting the cart before the horse, for sure. The reason why you only know the mean and the variance of your data is because you chose to summarize your data that way. No, it's not... A Gaussian is the best way to represent your knowledge of a value if you only know the mean and variance of its value. So if you start with a stack of data and compress it down to a mean and variance, you've discarded most of your knowledge, and are left with a Gaussian as your best guess representation. Yes if you were to boil it down to different summary data, like a max and min, you'd end up with a different state of knowledge and a different distribution. But given a mean and variance, the Gaussian is your best choice of distribution, and not because of the central limit theorem, but because it has maximum entropy on those constraints. You don't always even have access to the source data in the first place, maybe just the summary statistics.
- mturmon 6y ago
  I would like to push back against this in favor of the original comment. The context of this remark within the article is: > I was extremely confused as to why the Normal (Gaussian) Distribution pops up everywhere—in kurtotically-ignorant financial market analysis, in nature, everywhere. Thinking about it, the prevalence of the Gaussian is actually rather abnormal. Can you guess why it’s everywhere? This is not a "compression of data" question. It's not an "uninformed distributional choice" question. It's a "why is this distribution prevalent in Nature" question. In this context, I think the CLT gives a better answer. There are a lot of averaging processes in Nature, and due to the CLT, averaging of independent perturbations must give rise to normal distributions. It's possible to perhaps go a step deeper than the above. In some physical systems, you can look at the second moment as an energy -- like the voltage-squared in electrical systems. In this case, due to a-priori finiteness of system energy, the gaussian distribution can make a claim to being "inevitable" by the maxent argument in OP. ("In a system characterized by finite energy E, what is the least informative distributional constraint?")
  - lambdatronics 6y ago
    Because additive processes are common. If the variables were multiplied together instead of summed, you'd get a different asymptotic distribution.
Kednicma 6y ago
You are spot-on. I would just add that there's another relatively beautiful reason why, in practice, folks pick the mean and variance for their summary; it can be done online with live data, for O(1) time and space! [0] If we extend this idea, then we easily get kurtosis and skew as the next two moments, again in constant time and space, and again getting a normal distribution but now with skew and squish. This is non-trivial; it means that we have an online algorithm which sends our measurements directly to our summaries, without worrying about how detailed our measurements are (how many samples are summarized). For comparison, taking the median/quantiles/percentiles/etc. requires either fixed-size buckets which lose precision (as seen in Prometheus histograms) or around O(log log n) space [1], which is still practical but pedantically not constant. [0] https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance https://en.wikipedia.org/wiki/Algorithms_for_calculating_var... [1] https://tmc.web.engr.illinois.edu/lb.ps https://tmc.web.engr.illinois.edu/lb.ps

yetanothermonk 6y ago

Very good points. The Normal Distribution has very nice properties, which adds to its popularity, which, if I understand you correctly, adds to the significance we place on mean and variance.

adenadel 6y ago
I wouldn't say that the existence of the normal distribution is necessarily the reason that we place significance on the mean and variance. The mean is incredibly natural to define when you move to measure theoretic probability (simply the integral of a function, i.e. a random variable, with respect to some measure). When you take this point of view, random variables with particular moments existing are simply functions in L^p. Further, when you move on to proving the CLT the moments give you properties of the characteristic functions that allow you to prove the CLT. These are all deep connections. There's another interpretation of the mean (and conditional expectation) as quantities minimizing squared error. It's not surprising that squared error and variance are so similar and that these are connected.
- yetanothermonk 6y ago
  Great comment! You touched on something that got me wondering: do you know of something similar to MGF that determines a distribution but is not based on expected values?
  - clircle 6y ago
    MGFs do not determine distributions (The Student's t distribution does not have an MGF). CDFs and characteristic functions determine distributions.
  - adenadel 6y ago
    Well, characteristic functions are more general than MGFs (in that they are always finite) and have very useful properties, but they have a similar definition and are also based on an expectation. This may be pedantic, but an object that determines a distribution that isn't based on expected values is the CDF. In my first probability course in undergrad I think we defined probability mass functions and probability density functions first and defined CDFs in terms of them, but from a measure theoretic point of view, the CDF is more fundamental since it is defined for continuous and discrete distributions (and also for distributions that are neither).
    - contravariant 6y ago
      I'd probably choose the probability measure itself, rather than the CDF, just because you won't always have a space with a nice order topology. Of course the expectation value and the probability measure are really two sides of the same coin, the expectation value being equivalent to integration with respect to the probability measure.

jbay808 6y ago

> This is putting the cart before the horse, for sure. The reason why you only know the mean and the variance of your data is because you chose to summarize your data that way. No, it's not... A Gaussian is the best way to represent your knowledge of a value if you only know the mean and variance of its value. So if you start with a stack of data and compress it down to a mean and variance, you've discarded most of your knowledge, and are left with a Gaussian as your best guess representation. Yes if you were to boil it down to different summary data, like a max and min, you'd end up with a different state of knowledge and a different distribution. But given a mean and variance, the Gaussian is your best choice of distribution, and not because of the central limit theorem, but because it has maximum entropy on those constraints. You don't always even have access to the source data in the first place, maybe just the summary statistics.

mturmon 6y ago
I would like to push back against this in favor of the original comment. The context of this remark within the article is: > I was extremely confused as to why the Normal (Gaussian) Distribution pops up everywhere—in kurtotically-ignorant financial market analysis, in nature, everywhere. Thinking about it, the prevalence of the Gaussian is actually rather abnormal. Can you guess why it’s everywhere? This is not a "compression of data" question. It's not an "uninformed distributional choice" question. It's a "why is this distribution prevalent in Nature" question. In this context, I think the CLT gives a better answer. There are a lot of averaging processes in Nature, and due to the CLT, averaging of independent perturbations must give rise to normal distributions. It's possible to perhaps go a step deeper than the above. In some physical systems, you can look at the second moment as an energy -- like the voltage-squared in electrical systems. In this case, due to a-priori finiteness of system energy, the gaussian distribution can make a claim to being "inevitable" by the maxent argument in OP. ("In a system characterized by finite energy E, what is the least informative distributional constraint?")
- lambdatronics 6y ago
  Because additive processes are common. If the variables were multiplied together instead of summed, you'd get a different asymptotic distribution.

Kednicma 6y ago

You are spot-on. I would just add that there's another relatively beautiful reason why, in practice, folks pick the mean and variance for their summary; it can be done online with live data, for O(1) time and space! [0] If we extend this idea, then we easily get kurtosis and skew as the next two moments, again in constant time and space, and again getting a normal distribution but now with skew and squish. This is non-trivial; it means that we have an online algorithm which sends our measurements directly to our summaries, without worrying about how detailed our measurements are (how many samples are summarized). For comparison, taking the median/quantiles/percentiles/etc. requires either fixed-size buckets which lose precision (as seen in Prometheus histograms) or around O(log log n) space [1], which is still practical but pedantically not constant. [0] https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance https://en.wikipedia.org/wiki/Algorithms_for_calculating_var... [1] https://tmc.web.engr.illinois.edu/lb.ps https://tmc.web.engr.illinois.edu/lb.ps