5 ms·

Another way I've heard it framed is that statistics cares about the parameters, and ML cares about the answers.

fighterpilot 5y ago
ML cares about predictive generalization, stats cares about understanding and interpreting drivers
gmfawcett 5y ago
That framing seems a wee bit dismissive of statistics, esp. applied stats -- I'll hazard a guess that this came from the ML camp. :)
- srean 5y ago
  There is nothing dismissive about it, they are just different things. If we have strong theoretical understanding of the physics/model of a problem, but are unsure about some parameters, it makes sense to develop methods to find those unknown parameters accurately. This ability is a big deal in traditional statistics. People working in this field try and prove results that their proposed method can actually do this. If the model happens to be largely correct and the method robust, this even allows prediction. Often, however, we do not know the model and the 'parameter' is a piece of fiction anyway. If we are interested in prediction alone, its fine to let go of an ability to accurately estimate the parameters as long as predictions are accurate. Think epicycle models of planetary motion. ML folks try to prove that their methods have good prediction properties and are happy to sacrifice on parameter recovery. Nonparametric statistics and prequential statistics are somewhere in the middle. Sometimes it does seem though that people are trying to out do each other, coming up with methods that estimates the spectrum of a unicorn's rainbow more accurately than the best known result in research literature. This may look odd because the unicorn and his rainbow are pieces of fiction.
  - gmfawcett 5y ago
    Fair points made, and your unicorn analogy is wonderful. :)
bart_spoon 5y ago
As someone with degrees in statistics who now works in ML, all of these simplistic definitions are pithy, but useless. The reality is that the overlap between the two fields is so enormous that trying to come up with a clean separation for the two is a losing battle. To say that statisticians are uninterested in “the answers” (I.e. predictive accuracy) is foolish and rarely true. To say that ML doesn’t care about parameters (I.e. inference) ignores the vast amount of effort in ML researching topics like causal inference, feature importance, and yes, inference. At this point I’m convinced the debate regarding what is statistics vs what is ML is largely just tribal warfare. The two parties are fighting over shared territory, and while both may have had some claim to uniquely identifying techniques and research, the lines have long since eroded.

ML cares about predictive generalization, stats cares about understanding and interpreting drivers

That framing seems a wee bit dismissive of statistics, esp. applied stats -- I'll hazard a guess that this came from the ML camp. :)

srean 5y ago
There is nothing dismissive about it, they are just different things. If we have strong theoretical understanding of the physics/model of a problem, but are unsure about some parameters, it makes sense to develop methods to find those unknown parameters accurately. This ability is a big deal in traditional statistics. People working in this field try and prove results that their proposed method can actually do this. If the model happens to be largely correct and the method robust, this even allows prediction. Often, however, we do not know the model and the 'parameter' is a piece of fiction anyway. If we are interested in prediction alone, its fine to let go of an ability to accurately estimate the parameters as long as predictions are accurate. Think epicycle models of planetary motion. ML folks try to prove that their methods have good prediction properties and are happy to sacrifice on parameter recovery. Nonparametric statistics and prequential statistics are somewhere in the middle. Sometimes it does seem though that people are trying to out do each other, coming up with methods that estimates the spectrum of a unicorn's rainbow more accurately than the best known result in research literature. This may look odd because the unicorn and his rainbow are pieces of fiction.
- gmfawcett 5y ago
  Fair points made, and your unicorn analogy is wonderful. :)

bart_spoon 5y ago

As someone with degrees in statistics who now works in ML, all of these simplistic definitions are pithy, but useless. The reality is that the overlap between the two fields is so enormous that trying to come up with a clean separation for the two is a losing battle. To say that statisticians are uninterested in “the answers” (I.e. predictive accuracy) is foolish and rarely true. To say that ML doesn’t care about parameters (I.e. inference) ignores the vast amount of effort in ML researching topics like causal inference, feature importance, and yes, inference. At this point I’m convinced the debate regarding what is statistics vs what is ML is largely just tribal warfare. The two parties are fighting over shared territory, and while both may have had some claim to uniquely identifying techniques and research, the lines have long since eroded.