4 ms·

Not to bag on the write up, as it was well done, but does linear regression really qualify as machine learning? Almost every stats 101 course covers the topic,

by pyoung 13y ago

Not to bag on the write up, as it was well done, but does linear regression really qualify as machine learning? Almost every stats 101 course covers the topic, and probably 99% of people who use linear regressions in their day-to-day work would not call it machine learning. I know that linear regression is sometimes presented in Machine Learning courses, but I always thought is was done as a refresher, and not as actual course material of any significant weight.

yummyfajitas 13y ago
Why wouldn't it be machine learning? It certainly fits the definition of supervised learning (aka regression). Admittedly you do usually learn it in unsexy statistics classes rather than sexy machine learning classes...
- aet 13y ago
  The lines have always been a little blurry for me. You have statistics, machine learning, data mining, artificial intelligence. It all seems to overlap heavily. I tend to consider statistics and data mining to be concerned mostly with classical statistics, and machine learning and artificial artificial intelligence to be concerned more with bayesian methods, and algorithms. That being said, logistic regression seems like the quintessential machine learning technique. So who knows, any experts care to comment?
  - christopheraden 13y ago
    As someone who did graduate work in Bayesian methods for a statistics master's degree, I take offense to saying machine learning is not a concern of statistics and data mining (but not really)! The hesitance towards Bayesian methods seems more related to the discipline, and it seems that places that call what they do "machine learning" tend to be less hostile towards the explicit subjectivity of Bayes (I would highlight the word "explicit" in that sentence--Frequentism has it's fair share of subjectivity as well). There was a great post on Stats.SE a few years ago about the difference between statistics and machine learning[1]. Leo Breiman once argued that statistics tends to focus more on model fitting and checking, while machine learning looked at prediction accuracy. The exchange between Andy Gelman and Brendan O'Connor is pretty funny. It has been my personal experience however that many people that apply a method that they brand as "machine learning" are not as bothered with assumptions as my fellow conservative statisticians. But statistics and machine learning are quite similar in foundation. Barring the differences in terminology, as a professional statistician, I find I have as little difficulty read machine learning papers and algorithms as I do reading statistics ones. http://stats.stackexchange.com/questions/6/the-two-cultures-statistics-vs-machine-learning http://stats.stackexchange.com/questions/6/the-two-cultures-...
  - dpmehta02 13y ago
    All of these terms are used interchangeably, so the following definitions probably wont be too helpful in the real world, but in my experience: Artificial Intelligence is an umbrella academic term which encapsulates the study and design of intelligent machines. It's not well defined because AI is evolving so rapidly. Machine Learning is a branch of AI that is concerned specifically with learning from data; the results of learning are usually used to predict future events. (Think linear regressions, random forests, etc.) Though not specifically a part of AI, Statistics is the field that formed many of the algorithms used in ML. Stats informs ML research design (e.g., how large of a sample size do I need), generates mathematical solutions from proofs and equations, etc. With the rise of big data, it's slowly merging with ML. Data Mining is a mix of ML, Stats and Data Engineering. It's more concerned with structuring and extracting patterns from data than necessarily learning from it. It is often a task within an ML project.
- elchief 13y ago
  Because it was done in pen and paper by Gauss 200 years ago :)
saraid216 13y ago
Why does it matter? What's the utility of determining whether or not a particular topic qualifies as part of another particular topic?
- pyoung 13y ago
  In this case, it's not a huge deal, but if someone wrote on their resume that they had some experience with machine learning, and it turned out that they had only done linear regression modeling, they might be poorly received.
dpmehta02 13y ago
The 99% of people who use linear regression but would not call it machine learning probably aren't very well versed in regularization, cross-validation, non-linear transformations, feature engineering, data snooping, bagging, boosting, generalization, etc. A finely tuned linear regression is a devastating machine learning algorithm.
mbq 13y ago
The difference between statistical modelling and machine learning is that the first aims to make models that have a "right" accuracy on a given data-model complexity balance while the second those that work well on "unseen data". Thus if you use AIC to find best subset of variables for a linear model its statistical modelling, but if you use cross-validation for the same purpose it is machine learning (while, to be honest, both ideas are wrong).
donquichotte 13y ago
Linear regression can be a quite powerful tool, especially if (maybe counter-intuitively at first) it is used to fit exponentials [1] or polynomials to data. [1] http://mathworld.wolfram.com/LeastSquaresFittingExponential.html http://mathworld.wolfram.com/LeastSquaresFittingExponential....
mathattack 13y ago
I was in your camp until the weight of the discussion convinced me otherwise. I learned regression as an advanced statistics topic, and it wasn't touched on in my (beyond awful) AI class. However... If linear regression is being used to predict the next value in a dataset, and the caliber of the regression improves as more data is gathered, then it counts as machine learning. For all I know, Netflix could be improving their picks of my movies with a grand regression.

by pyoung 13y ago

yummyfajitas 13y ago

Why wouldn't it be machine learning? It certainly fits the definition of supervised learning (aka regression). Admittedly you do usually learn it in unsexy statistics classes rather than sexy machine learning classes...

aet 13y ago
The lines have always been a little blurry for me. You have statistics, machine learning, data mining, artificial intelligence. It all seems to overlap heavily. I tend to consider statistics and data mining to be concerned mostly with classical statistics, and machine learning and artificial artificial intelligence to be concerned more with bayesian methods, and algorithms. That being said, logistic regression seems like the quintessential machine learning technique. So who knows, any experts care to comment?
- christopheraden 13y ago
  As someone who did graduate work in Bayesian methods for a statistics master's degree, I take offense to saying machine learning is not a concern of statistics and data mining (but not really)! The hesitance towards Bayesian methods seems more related to the discipline, and it seems that places that call what they do "machine learning" tend to be less hostile towards the explicit subjectivity of Bayes (I would highlight the word "explicit" in that sentence--Frequentism has it's fair share of subjectivity as well). There was a great post on Stats.SE a few years ago about the difference between statistics and machine learning[1]. Leo Breiman once argued that statistics tends to focus more on model fitting and checking, while machine learning looked at prediction accuracy. The exchange between Andy Gelman and Brendan O'Connor is pretty funny. It has been my personal experience however that many people that apply a method that they brand as "machine learning" are not as bothered with assumptions as my fellow conservative statisticians. But statistics and machine learning are quite similar in foundation. Barring the differences in terminology, as a professional statistician, I find I have as little difficulty read machine learning papers and algorithms as I do reading statistics ones. http://stats.stackexchange.com/questions/6/the-two-cultures-statistics-vs-machine-learning http://stats.stackexchange.com/questions/6/the-two-cultures-...
- dpmehta02 13y ago
  All of these terms are used interchangeably, so the following definitions probably wont be too helpful in the real world, but in my experience: Artificial Intelligence is an umbrella academic term which encapsulates the study and design of intelligent machines. It's not well defined because AI is evolving so rapidly. Machine Learning is a branch of AI that is concerned specifically with learning from data; the results of learning are usually used to predict future events. (Think linear regressions, random forests, etc.) Though not specifically a part of AI, Statistics is the field that formed many of the algorithms used in ML. Stats informs ML research design (e.g., how large of a sample size do I need), generates mathematical solutions from proofs and equations, etc. With the rise of big data, it's slowly merging with ML. Data Mining is a mix of ML, Stats and Data Engineering. It's more concerned with structuring and extracting patterns from data than necessarily learning from it. It is often a task within an ML project.
elchief 13y ago
Because it was done in pen and paper by Gauss 200 years ago :)

saraid216 13y ago

Why does it matter? What's the utility of determining whether or not a particular topic qualifies as part of another particular topic?

pyoung 13y ago
In this case, it's not a huge deal, but if someone wrote on their resume that they had some experience with machine learning, and it turned out that they had only done linear regression modeling, they might be poorly received.

dpmehta02 13y ago

The 99% of people who use linear regression but would not call it machine learning probably aren't very well versed in regularization, cross-validation, non-linear transformations, feature engineering, data snooping, bagging, boosting, generalization, etc. A finely tuned linear regression is a devastating machine learning algorithm.

mbq 13y ago

The difference between statistical modelling and machine learning is that the first aims to make models that have a "right" accuracy on a given data-model complexity balance while the second those that work well on "unseen data". Thus if you use AIC to find best subset of variables for a linear model its statistical modelling, but if you use cross-validation for the same purpose it is machine learning (while, to be honest, both ideas are wrong).

donquichotte 13y ago

Linear regression can be a quite powerful tool, especially if (maybe counter-intuitively at first) it is used to fit exponentials [1] or polynomials to data. [1] http://mathworld.wolfram.com/LeastSquaresFittingExponential.html http://mathworld.wolfram.com/LeastSquaresFittingExponential....

mathattack 13y ago

I was in your camp until the weight of the discussion convinced me otherwise. I learned regression as an advanced statistics topic, and it wasn't touched on in my (beyond awful) AI class. However... If linear regression is being used to predict the next value in a dataset, and the caliber of the regression improves as more data is gathered, then it counts as machine learning. For all I know, Netflix could be improving their picks of my movies with a grand regression.