3 ms·

Sorry, guys. After advanced study of probability, writing a Ph.D. dissertation in applied probability, and doing applications of probability in US national sec

by graycat 13y ago

Sorry, guys. After advanced study of probability, writing a Ph.D. dissertation in applied probability, and doing applications of probability in US national security and business, here's my take on the OP: (1) Sure, given a random variable taking values in the real numbers, it has a (cumulative) distribution and may have a probability density distribution. (2) In some special, fortunate cases, actually can get some useful information about the distribution, e.g., it has an expectation, the expectation is finite, it has a finite variance, and might even be able to say that the distribution is in one of the famous families, Gaussian, exponential, Poisson, etc. and, then, estimate the parameters of the distribution, e.g., for a Gaussian, estimate the mean and variance. (3) While we know that our random variable has a distribution, mostly in practice we can't know much about that distribution. Commonly if we know that the distribution has an expectation, that the expectation is finite, and that the variance is finite (that commonly we get just by what we know about the real problem with no effort in probability or statistics), then we have to be satisfied with those 'qualitative' facts and move on. E.g., finite variance is enough about the distribution to apply the strong law of large numbers! (4) Mostly when we do know that the distribution is Gaussian, exponential, etc., it is because we are using one of the classic limit theorems of probability, e.g., the central limit theorem or the renewal theorem. (5) If the random variable takes values in, for some positive integer n, some n-dimensional space over, say, the real or complex numbers, then in practice knowing much detail about the distribution is next to hopeless. Why? Because of the 'curse of dimensionality' where the number of points needed for a fine grid is commonly so high it makes 'big data' look like something on a 3 x 5 index card! Net, mostly we do our work in applied probability knowing, of course, that a random variable has a cumulative distribution but otherwise knowing at most only some qualitative aspects, e.g., finite variance, and little else. So, our more important methods of working with random variables should need only meager assumptions about the distributions of those random variables. E.g., order statistics are always sufficient. Commonly to get the most desired estimates from least squares techniques, need only finite mean and variance; the common Gaussian assumptions about the 'model error terms' is needed only for doing classic confidence interval, etc. estimates. Generally there are a lot of 'non-parametric' (distribution-free, where we make no assumptions about the distribution) techniques in hypothesis testing. For estimation, there are non-parametric techniques based on, say, 'resampling' for getting confidence intervals. And there are other distribution-free techniques. Once I saw a paper in hypothesis testing that was both multidimensional and distribution-free. Or, in shortest terms: Yes, a random variable always has a cumulative distribution, but that doesn't mean that always in practice we should try to find that distribution; instead, commonly in practice we have little hope of knowing much about the distribution! Or, one of the best books in probability I know of is Jacques Neveu, 'Mathematical Foundations of the Calculus of Probability', Holden-Day. and there is at most only meager mention of specific distributions in the whole book!

christopheraden 13y ago
Couple comments: Anyone in statistics/probability has seen this quote multiple times: "All models are wrong, but some are useful." -George Box. We often don't know the underlying distribution of a particular process or our data, true, but that doesn't mean that we can't get useful information out of it by making some assumptions, provided that our assumptions are not wildly wrong. For example, if we assume height is normally distributed, that should mean it's possible, under our model, to see someone who is -9000 feet tall. We're willing to accept this falsity in exchange for having some nice properties we wouldn't have had otherwise, provided we exercise some common sense. In regards to (3), if all you can wield is the SLLN, you can't do much inference or prediction. So you're able to guarantee that the sample mean will converge to the expectation almost surely. The closeness of approximation in a finite sample case using asymptotics is intrinsically tied to sample size and the particulars of the underlying distribution of the random variable. In regards to (5), the curse of dimensionality does become a large problem, but throwing out assumptions makes it a far bigger problem. The nonparametrics community has a lot of trouble with multivariate distributions for precisely this reason! This is one area where parametric models do better than non-parametric ones, since they have so much more structure to play with, it makes the problems so much more tractable. You mention order statistics... order statistics in higher dimensions is a tricky concept that is not as defined as the one dimensional case. Which one is more: (3, 2, 1) or (1, 2, 3)? Wouldn't that necessarily depend upon the underlying distribution or some other measure of distance? If you have a paper to suggest that general high-dimensional order statistics makes sense, throw it this way. You are correct that Least Squares doesn't require a distribution to get mean and variance estimates, but what good is that if you don't know how much of your population fits within a multiple of your standard deviation around your mean? You could use something that works for all distributions like Chebyshev's Inequality, but to get that level of sweeping generality seriously hurts your power (If your data IS actually normal, 95% of obs fall within 1.96 SD's of the mean. Chebyshev will tell you 75%), and even Chebyshev had to impose a restriction to get that bound--it assumes finite mean and variance. While there are distribution-free hypothesis testing methods, most of classical nonparametric statistics makes some assumptions about the underlying distributions, albeit they are far gentler than the parametric assumptions. Always choosing nonparametric methods is just as short-sighted as always using parametric methods. It's very possible that you are throwing away vast amounts of information by using a nonparametric procedure when a parametric one would have been completely adequate. With ANOVA and two-sample tests, you don't lose much by going with Kruskall-Wallis or Mann-Whitney tests (under normality, MW has a relative efficiency of 3/pi versus the t-test), but in other circumstances, using a nonparametric method could be way worse, provided the parametric assumptions are true. What I'm getting at is that while non-parametrics is nice in that it doesn't require many assumptions, they may be throwing away too much. Fitting to a distribution may very useful in making inferences and predictions. All models are wrong, but some are useful.

by graycat 13y ago

christopheraden 13y ago

Couple comments: Anyone in statistics/probability has seen this quote multiple times: "All models are wrong, but some are useful." -George Box. We often don't know the underlying distribution of a particular process or our data, true, but that doesn't mean that we can't get useful information out of it by making some assumptions, provided that our assumptions are not wildly wrong. For example, if we assume height is normally distributed, that should mean it's possible, under our model, to see someone who is -9000 feet tall. We're willing to accept this falsity in exchange for having some nice properties we wouldn't have had otherwise, provided we exercise some common sense. In regards to (3), if all you can wield is the SLLN, you can't do much inference or prediction. So you're able to guarantee that the sample mean will converge to the expectation almost surely. The closeness of approximation in a finite sample case using asymptotics is intrinsically tied to sample size and the particulars of the underlying distribution of the random variable. In regards to (5), the curse of dimensionality does become a large problem, but throwing out assumptions makes it a far bigger problem. The nonparametrics community has a lot of trouble with multivariate distributions for precisely this reason! This is one area where parametric models do better than non-parametric ones, since they have so much more structure to play with, it makes the problems so much more tractable. You mention order statistics... order statistics in higher dimensions is a tricky concept that is not as defined as the one dimensional case. Which one is more: (3, 2, 1) or (1, 2, 3)? Wouldn't that necessarily depend upon the underlying distribution or some other measure of distance? If you have a paper to suggest that general high-dimensional order statistics makes sense, throw it this way. You are correct that Least Squares doesn't require a distribution to get mean and variance estimates, but what good is that if you don't know how much of your population fits within a multiple of your standard deviation around your mean? You could use something that works for all distributions like Chebyshev's Inequality, but to get that level of sweeping generality seriously hurts your power (If your data IS actually normal, 95% of obs fall within 1.96 SD's of the mean. Chebyshev will tell you 75%), and even Chebyshev had to impose a restriction to get that bound--it assumes finite mean and variance. While there are distribution-free hypothesis testing methods, most of classical nonparametric statistics makes some assumptions about the underlying distributions, albeit they are far gentler than the parametric assumptions. Always choosing nonparametric methods is just as short-sighted as always using parametric methods. It's very possible that you are throwing away vast amounts of information by using a nonparametric procedure when a parametric one would have been completely adequate. With ANOVA and two-sample tests, you don't lose much by going with Kruskall-Wallis or Mann-Whitney tests (under normality, MW has a relative efficiency of 3/pi versus the t-test), but in other circumstances, using a nonparametric method could be way worse, provided the parametric assumptions are true. What I'm getting at is that while non-parametrics is nice in that it doesn't require many assumptions, they may be throwing away too much. Fitting to a distribution may very useful in making inferences and predictions. All models are wrong, but some are useful.