Indeed. In fact the birth of statistical learning theory was Vapnik's (rather unintuitive) insight that although the theoretically optimal strategy in a classi

Indeed. In fact the birth of statistical learning theory was Vapnik's (rather unintuitive) insight that although the theoretically optimal strategy in a classification task is the class conditional distribution, estimating the class conditional distribution from the data is not a promising way to do it. Estimating distributions is freaking hard, you may not (ever) have data to estimate that distribution, however you can short circuit the process and solve the classification problem directly (without explicitly modeling the class conditional distribution). That said, machine learning has models that do not rely on a stochastic sampling process. The sampling process can be adversarial too. In such situations the algorithms guarantee that you wouldn't be too far off (even in finite time) from the strategy that would be optimal in hind-sight. These are non-asymptotic guarantees. I am of course sweeping large swathes of theory under the carpet, because this is hardly the forum for that. The beauty of CLT notwithstanding, what is surprising is how rarely the Gaussian assumption holds in practice. Many distributions are too heavy tailed to have a finite variance, and in such scenarios CLT does not yield you a Gaussian but yields a stable distribution (Gaussian is just one, in fact the only one in the stable family with a finite distribution. Gaussian distribution was not discovered by Gauss and neither is it that normal). To make a tongue in cheek claim, the prince of mathematicians sort of got away with it. Machine learning theory was never far from statistics and probability. How can it, when its traditional theoretical bedrock (PAC) is explicitly a statement about convergence in probability with all its epsilon and delta in your face. Where the two differ are in the amount of focus on prediction (as opposed to parameter estimation), familiarity with optimization and algorithms, and lack of fascination for test of hypothesis, Gaussian distribution and asymptotic normality.