3 ms·
Because the "first order" "data correlation" tools that we use (typical statistical measures like correlation coefficients, means, stddev, etc) are inadequate f
by mtgp1000 6y ago
Because the "first order" "data correlation" tools that we use (typical statistical measures like correlation coefficients, means, stddev, etc) are inadequate for capturing the critical correlations in the kind of large, real world data sets that ML is shining in right now.
One example, if you have a thousand dimensional data set, and 3+ of dimensions are correlated, but only for certain value ranges, to find something like that with classical statistics, you would need an intuition and some digging and even then you might not see it.
If you suitably prepare your data and throw it into an ML it's trivial. Think of ML as automation for statistical inference. Because all neural nets really do is learn complex, super multivariate probability statistics - down to individual pixel-pixel relationships, for image data.
Edit: I'll add that one other reason ML works so well is that parameters in nets can learn (and represent) complex functions which are impractical (if not impossible) with mathematical notation. Which is what most complex real life distributions probably actually look like. It's like the bridge between analog and digital math, a sort of topological compression, if that makes any sense