3 ms·

There both about inference, just slightly different philosophical approaches. Statistics, dating before computers and big data, comes out of math, has a greater

There both about inference, just slightly different philosophical approaches. Statistics, dating before computers and big data, comes out of math, has a greater emphasis on proofs and closed form solutions, and IMO an opinionated view that what it's modeling is a well-behaved function plus a well-behaved error term. Machine learning is kind of like statistics for street-fighting. It doesn't care about Bayesian v. frequentist philosophical debates, resorts to cheap tricks like regularization, dropout, using many dumb methods that work better than trying to find the smartest method (ensembles and funny tricks like random forest). If it works in a well-designed test, use it. There is more than one path to enlightenment. One man's 'data mining' is another's reasoned empiricism, letting the data do the talking instead of leaning on ontological preconceptions of how data is supposed to behave. Machine learning often works well but you don’t always know why. Stats doesn’t always work and you know precisely why not. (because your data doesn't fit the assumptions of OLS etc.)

achompas 10y ago
I take issue with a number of points in your post, as they don't characterize the differences in the disciplines properly. > There both about inference This is actually incorrect -- a large swath of machine learning algorithms (like deep learning and SVMs) are unconcerned with the underlying distribution of your data, and are perfectly happy to generate predictions using a completely uninterpretable model. > Statistics [...] comes out of math, has a greater emphasis on proofs There is an entire branch of machine learning, known alternatively as learning theory or complexity theory, devoted to the analysis of algorithmic generalization error. Generally speaking, any popular machine learning algorithm has a proof of its (1) performance and (2) ability to generalize. > It doesn't care about Bayesian v. frequentist philosophical debates If this is true, it's only because machine learning has firmly parked itself in the Bayesian camp wherever possible. Naive Bayes, regularization, dimensionality reduction, and graphical models are either motivated by or easily interpreted as Bayesian techniques.
- RockyMcNuts 10y ago
  what's your definition of inference, and what do you call the purpose of a machine learning algorithm, and how does that purpose differ from a statistics approach? perhaps I used the term incorrectly but a dictionary defines inference as "a conclusion reached on the basis of evidence and reasoning." bottom line... similar objectives, slightly different if often overlapping emphases and bags of tricks.
  - achompas 10y ago
    Inference, in the context of statistical analysis, refers to the task of understanding underlying properties of a distribution. This is the definition used by practitioners. Machine learning practitioners care more about computational performance or generalizability beyond seen data than they care about properties of a data-generating distribution. Your "bottom line" summary and repeated reference to algorithms as "bags of tricks" are reductionist statements suggesting you don't understand the nuanced, but important, differences between the fields.
    - RockyMcNuts 10y ago
      ah, thanks. I look forward to smarter folks than me elucidating the nuanced differences. I had a distinguished math professor who taught me that mathematics is the application of cheap tricks and bad jokes. He didn't take himself too seriously. When I learned about random forests I couldn't help having a good laugh, it's a cheap trick in the finest sense of the word.
arcanus 10y ago
> resorts to cheap tricks like regularization, Regularization is not a 'cheap trick'. It is an essential technique for mathematically ill-posed problems, where no unique solution exists. This is extensively studied and well-understood, and is used in many fields outside of machine learning.

achompas 10y ago

I take issue with a number of points in your post, as they don't characterize the differences in the disciplines properly. > There both about inference This is actually incorrect -- a large swath of machine learning algorithms (like deep learning and SVMs) are unconcerned with the underlying distribution of your data, and are perfectly happy to generate predictions using a completely uninterpretable model. > Statistics [...] comes out of math, has a greater emphasis on proofs There is an entire branch of machine learning, known alternatively as learning theory or complexity theory, devoted to the analysis of algorithmic generalization error. Generally speaking, any popular machine learning algorithm has a proof of its (1) performance and (2) ability to generalize. > It doesn't care about Bayesian v. frequentist philosophical debates If this is true, it's only because machine learning has firmly parked itself in the Bayesian camp wherever possible. Naive Bayes, regularization, dimensionality reduction, and graphical models are either motivated by or easily interpreted as Bayesian techniques.

RockyMcNuts 10y ago
what's your definition of inference, and what do you call the purpose of a machine learning algorithm, and how does that purpose differ from a statistics approach? perhaps I used the term incorrectly but a dictionary defines inference as "a conclusion reached on the basis of evidence and reasoning." bottom line... similar objectives, slightly different if often overlapping emphases and bags of tricks.
- achompas 10y ago
  Inference, in the context of statistical analysis, refers to the task of understanding underlying properties of a distribution. This is the definition used by practitioners. Machine learning practitioners care more about computational performance or generalizability beyond seen data than they care about properties of a data-generating distribution. Your "bottom line" summary and repeated reference to algorithms as "bags of tricks" are reductionist statements suggesting you don't understand the nuanced, but important, differences between the fields.
  - RockyMcNuts 10y ago
    ah, thanks. I look forward to smarter folks than me elucidating the nuanced differences. I had a distinguished math professor who taught me that mathematics is the application of cheap tricks and bad jokes. He didn't take himself too seriously. When I learned about random forests I couldn't help having a good laugh, it's a cheap trick in the finest sense of the word.

arcanus 10y ago

> resorts to cheap tricks like regularization, Regularization is not a 'cheap trick'. It is an essential technique for mathematically ill-posed problems, where no unique solution exists. This is extensively studied and well-understood, and is used in many fields outside of machine learning.