4 ms·

> by determining the distribution of the process that the data was generated from. Well, each random variable has a distribution. And there are a few distribu

by graycat 10y ago

> by determining the distribution of the process that the data was generated from. Well, each random variable has a distribution. And there are a few distributions that are common so are taught. Then, presto, bingo, too many students conclude that an important first step is to find a distribution. However, commonly in practice, with just samples and without more in mathematical assumptions, finding a distribution is from not very promising to hopeless. Hopeless? Yes, consider a random variable that takes values in 50 dimensional Euclidean space. But there is a lot of statistics that is distribution-free, where we make no assumptions on probability distributions. E.g., I published such a paper in Information Sciences. In addition with some meager assumptions, say, the random variables have expectations, the squares of the random variables have finite expectations, etc., can do more. For model fitting, if can assume that the data has Gaussian distribution, is homoscedastic, have independent and identically distributed (i.i.d.), etc., then can get some more results, e.g., know that some of the results of the computations have Gaussian or F distribution, etc. Then can do a lot of classic hypothesis tests, confidence intervals, etc. But, with just meager assumptions, commonly can still proceed and know that are still making a best L^2 approximation. Then can drag out the classic result that a sequence of (such) random variables that are Cauchy convergent in L^2 do converge in L^2, that L^2 is complete (i.e., a Hilbert space), and that some sub-sequence converges almost surely. That's a lot -- might be able to take that to the bank. And made no more than meager, general assumptions about distributions. Really, often we get some of the well known distributions from some theorems, not analysis of empirical data. E.g., get a Gaussian assumption from the central limit theorem. Can get an exponential distribution from the Poisson process (e.g., E. Cinlar's text), and can get that from the very general, even astounding, renewal theorem (e.g., W. Feller's second volume).

srean 10y ago
Indeed. In fact the birth of statistical learning theory was Vapnik's (rather unintuitive) insight that although the theoretically optimal strategy in a classification task is the class conditional distribution, estimating the class conditional distribution from the data is not a promising way to do it. Estimating distributions is freaking hard, you may not (ever) have data to estimate that distribution, however you can short circuit the process and solve the classification problem directly (without explicitly modeling the class conditional distribution). That said, machine learning has models that do not rely on a stochastic sampling process. The sampling process can be adversarial too. In such situations the algorithms guarantee that you wouldn't be too far off (even in finite time) from the strategy that would be optimal in hind-sight. These are non-asymptotic guarantees. I am of course sweeping large swathes of theory under the carpet, because this is hardly the forum for that. The beauty of CLT notwithstanding, what is surprising is how rarely the Gaussian assumption holds in practice. Many distributions are too heavy tailed to have a finite variance, and in such scenarios CLT does not yield you a Gaussian but yields a stable distribution (Gaussian is just one, in fact the only one in the stable family with a finite distribution. Gaussian distribution was not discovered by Gauss and neither is it that normal). To make a tongue in cheek claim, the prince of mathematicians sort of got away with it. Machine learning theory was never far from statistics and probability. How can it, when its traditional theoretical bedrock (PAC) is explicitly a statement about convergence in probability with all its epsilon and delta in your face. Where the two differ are in the amount of focus on prediction (as opposed to parameter estimation), familiarity with optimization and algorithms, and lack of fascination for test of hypothesis, Gaussian distribution and asymptotic normality.
eli_gottlieb 10y ago
>And made no more than meager, general assumptions about distributions. I don't see how assuming the data are IID from a Gaussian is a meager assumption.
- graycat 10y ago
  > I don't see how assuming the data are IID from a Gaussian is a meager assumption. It's not. Somehow we have failed to communicate accurately. I wrote > But, with just meager assumptions, commonly can still proceed and know that are still making a best L^2 approximation. In that sentence, I didn't suggest that those "assumptions" were the Gaussian i.i.d. of the previous paragraph. Instead, I left the "assumptions" unspecified. Why? Because there is quite a variety available. But lots of the options are "meager". Typically with more assumptions, can get more results. But for model fitting, building, constructing, discovering, whatever, can still get a lot with next to nothing in assumptions.

by graycat 10y ago

srean 10y ago

Indeed. In fact the birth of statistical learning theory was Vapnik's (rather unintuitive) insight that although the theoretically optimal strategy in a classification task is the class conditional distribution, estimating the class conditional distribution from the data is not a promising way to do it. Estimating distributions is freaking hard, you may not (ever) have data to estimate that distribution, however you can short circuit the process and solve the classification problem directly (without explicitly modeling the class conditional distribution). That said, machine learning has models that do not rely on a stochastic sampling process. The sampling process can be adversarial too. In such situations the algorithms guarantee that you wouldn't be too far off (even in finite time) from the strategy that would be optimal in hind-sight. These are non-asymptotic guarantees. I am of course sweeping large swathes of theory under the carpet, because this is hardly the forum for that. The beauty of CLT notwithstanding, what is surprising is how rarely the Gaussian assumption holds in practice. Many distributions are too heavy tailed to have a finite variance, and in such scenarios CLT does not yield you a Gaussian but yields a stable distribution (Gaussian is just one, in fact the only one in the stable family with a finite distribution. Gaussian distribution was not discovered by Gauss and neither is it that normal). To make a tongue in cheek claim, the prince of mathematicians sort of got away with it. Machine learning theory was never far from statistics and probability. How can it, when its traditional theoretical bedrock (PAC) is explicitly a statement about convergence in probability with all its epsilon and delta in your face. Where the two differ are in the amount of focus on prediction (as opposed to parameter estimation), familiarity with optimization and algorithms, and lack of fascination for test of hypothesis, Gaussian distribution and asymptotic normality.

eli_gottlieb 10y ago

>And made no more than meager, general assumptions about distributions. I don't see how assuming the data are IID from a Gaussian is a meager assumption.

graycat 10y ago
> I don't see how assuming the data are IID from a Gaussian is a meager assumption. It's not. Somehow we have failed to communicate accurately. I wrote > But, with just meager assumptions, commonly can still proceed and know that are still making a best L^2 approximation. In that sentence, I didn't suggest that those "assumptions" were the Gaussian i.i.d. of the previous paragraph. Instead, I left the "assumptions" unspecified. Why? Because there is quite a variety available. But lots of the options are "meager". Typically with more assumptions, can get more results. But for model fitting, building, constructing, discovering, whatever, can still get a lot with next to nothing in assumptions.