3 ms·
I don't blame you. They don't teach it because actually teaching it would be a major side track, everyone would ask why you need to know it, and it would gener
by btilly 13y ago
I don't blame you. They don't teach it because actually teaching it would be a major side track, everyone would ask why you need to know it, and it would generally cause problems.
But it isn't that hard to prove for yourself for the simple case of the binomial curve if you're willing to supply some elbow grease. Here is an outline of how I did it when I was an undergrad.
1. Prove that log(n!) = n log(n) - log(n)/2 + k + o(1) where k is some constant. (Trick: approximate the integral of log(x) with the trapezoidal rule, that gives you n log(n). You wind up with log(n)/2 + log(1)/2 left over. Prove that the sum of the errors to infinity converges to a constant, which gives you, to finite n, k + o(1) where the o(1) term is the sum of the rest of the errors that you haven't reached yet.)
2. Suppose that X_1, X_2, etc are an identically distributed independent set of random variables that have probability 0.5 of being +1 or -1. Prove that each X_i has mean 0 and variance 1.
3. Consider the random variable Y_n = (X_1 + X_2 + ... + X_n)/sqrt(n). Prove that it has mean 0 and variance 1.
4. What is the "density" (in quotes because it is discrete) of Y_n around x? Well you can calculate an exact formula in terms of the binomial formula, and stick in your approximation from step 1, then show that the point weight of the nearest point to x is going to be e^(-x^2)(c + o(1))/sqrt(n) where c is a constant that can be derived from k in step 1. Remember that that's the only point for a distance of sqrt(n), so the density is proportional to e^(-x^2).
5. Look up the standard argument that the integral from -infinity to infinity of e^(-x^2) is sqrt(pi). Note that Y has to sum up to a total weight of 1 (because it is a probability density), substitute that fact in then you get both the standard normal distribution out AND as an unexpected bonus, Stirling's formula!
(If you wish you can do the same for a biased coin, the derivation is slightly more complex but follows the same lines. However to get arbitrary probability distributions you need to use different approaches. People usually don't encounter that proof until graduate school.)
- deleted 13y ago[deleted]
- btilly 13y agoThanks, fixed.
- jules 13y agoAnother way to derive the bell curve is as the maximum entropy distribution with known mean and variance. The derivation does require one trick that is usually not taught in high school. You know how you can maximize a function by setting its derivative to zero? Well, there is a generalization of that to functions of multiple variables: you set the gradient to zero. But there is yet another generalization to that for functions of continuously infinite number of variables! That's called the calculus of variations. If we want to maximize the entropy over all possible probability distributions over the real numbers, then we have a continuously infinite number of variables, since for each real number x, we need to assign a probability mass p(x). Here is the derivation for those interested: Let m be the mean we want and v be the variance we want, and let p(x) be the probability density function. We want to maximize the entropy S = int(p(x)log(p(x))) given the constraints that: int(p(x)) = 1 // total probability should sum to 1 E(X) = int(x*p(x)) = m // mean should be m E((X-m)^2) = int((x-m)^2*p(x)) = v // variance should be v This problem can be solved with calculus of variations with Lagrange multipliers. The Lagrangian is all the terms inside the integrals summed up: L = p*log(p) + a*p + b*x*p + c*(x-m)^2*p Calculus of variations tells us that for the optimal p, we have: dL/dp = 0 This is the analogue of setting the derivative to zero for maximizing a function. Calculating that we get: a + bx + c(x-m)^2 + log p + 1 = 0 rewriting: p = e^-(a + bx + c(x-m)^2 + 1) We have the general form of the Bell curve. Now we just need to find the right values for a,b,c to satisfy the 3 constraints. As special cases of this we have: 1) The probability distribution with maximum entropy without constraints is the uniform distribution. 2) The probability distribution with maximum entropy with only the constraint that the mean is m, is the exponential distribution. In general if you want to find maximum entropy distribution with known expectation values E(f(X)) and E(g(X)), then that is of this form: p(x) = C*e^(a*f(x) + b*g(x)) This gives an indication why so many distributions have this form. For example the entire exponential family can also be derived as maximum entropy distributions with certain known expectation values. Of course this derivation is only a good motivation for the bell curve if you already cared about entropy, mean and variance.