4 ms·

> Regression is any algorithm that takes a collection of inputs and predicts an output. I get that the author is writing for a certain audience, but this is a

by ellisv 6y ago

> Regression is any algorithm that takes a collection of inputs and predicts an output. I get that the author is writing for a certain audience, but this is a gross over-simplification.

bonoboTP 6y ago
Depends on the tradition you are coming from. In machine learning it really is just that. A synonym for continuous-valued function approximation based on training data. You may say it's only regression if it fits your favorite framework, like giving confidence values or goodness of fit etc, but that's not true in general. There so many variants, like Bayesian probabilistic regression, nonparametrics, neural nets, random forests etc.
- rm999 6y ago
  It’s a nonstandard definition at best. While regression is a name given to some algorithms (linear and logistic for example), this is an artifact of the statistical foundations of ML. Its standard use in ML is to describe the problem, not the solution: regression is predicting a continuous value and classification is predicting discrete categories. Either use is ok, but e.g. a tree model predicting if an animal is a dog or a cat is not regression by any definition.
  - bonoboTP 6y ago
    I'd say arguing about this is a huge distraction. Some would say classification is a special case of regression: the predicted continuous values are interpreted as class probabilities. You're right that regression is a type of task, not a type of solution. The conceptual difference is important to understand if someone does not yet know it. But it's not an all or nothing set-in-stone thing. It implies there is a waterfall design to these ideas, as if some oracle posed these tasks to us and then we started finding solutions to each of them separately. But actually in many cases one and the same algorithm with small tweaks can tackle multiple tasks. Sometimes we have the hammer (algorithm) first and then the nails (task). How you build your taxonomy and how you categorize one approach or another is not the same as learning and understanding. I always had an issue with this at university, where some lecturers would confuse learning lists like "what are the 3 areas of field X" or what are "the 4 principles of approach Y" etc. The world is not structured according to subjects, fields, subfields etc. It's not a single hierarchy, but a big mess of similarities, like a graph or a multidimensional space. The map is not the territory and so on. Terminology is necessary of course for communication and structuring books etc, but I like to see it merely as a utilitarian thing. The categories provide a scaffolding so that learning can happen. Studying the vocabulary and various outlines and nested hierarchies is a useful part of the journey but should not be confused with actually learning the thing itself. You could in principle learn all about regression without ever learning the word "regression". The name actually originates from statistics, named after the "regression towards the mean" phenomenon, where they observed how the adult height of people is closer to the average compared to the height of their parents. So a tall person will have tall children but less extremely tall (in tendency, some will of course be even taller). So confusingly, regression literally means "going back".
- ellisv 6y ago
  > Depends on the tradition you are coming from. Statistics :)
  - bonoboTP 6y ago
    Yes, the statistics and machine learning research communities traditionally have a very different view of the world, of what is important, what is superfluous, but lately the two are more and more merging.
    - tomrod 6y ago
      Elements of Statistical Learning is a common entry point. I'd argue they are much closer than appears. As a subfield in computer science, of course the concern has often been on algorithmic complexity and similar. But that is nascency exposed, in my view, and likely not representative of a fully mature field. Armchair thought (not a historian of economics): I think Econometrics followed (and continues to follow) a similar evolution -- start with the goals (identification of model parameters, identifiability, KPIs), improve statistical validity and relevance, annotate dead ends or less common routes, continue on trucking.

by ellisv 6y ago

> Regression is any algorithm that takes a collection of inputs and predicts an output. I get that the author is writing for a certain audience, but this is a gross over-simplification.

bonoboTP 6y ago

Depends on the tradition you are coming from. In machine learning it really is just that. A synonym for continuous-valued function approximation based on training data. You may say it's only regression if it fits your favorite framework, like giving confidence values or goodness of fit etc, but that's not true in general. There so many variants, like Bayesian probabilistic regression, nonparametrics, neural nets, random forests etc.

rm999 6y ago
It’s a nonstandard definition at best. While regression is a name given to some algorithms (linear and logistic for example), this is an artifact of the statistical foundations of ML. Its standard use in ML is to describe the problem, not the solution: regression is predicting a continuous value and classification is predicting discrete categories. Either use is ok, but e.g. a tree model predicting if an animal is a dog or a cat is not regression by any definition.
- bonoboTP 6y ago
  I'd say arguing about this is a huge distraction. Some would say classification is a special case of regression: the predicted continuous values are interpreted as class probabilities. You're right that regression is a type of task, not a type of solution. The conceptual difference is important to understand if someone does not yet know it. But it's not an all or nothing set-in-stone thing. It implies there is a waterfall design to these ideas, as if some oracle posed these tasks to us and then we started finding solutions to each of them separately. But actually in many cases one and the same algorithm with small tweaks can tackle multiple tasks. Sometimes we have the hammer (algorithm) first and then the nails (task). How you build your taxonomy and how you categorize one approach or another is not the same as learning and understanding. I always had an issue with this at university, where some lecturers would confuse learning lists like "what are the 3 areas of field X" or what are "the 4 principles of approach Y" etc. The world is not structured according to subjects, fields, subfields etc. It's not a single hierarchy, but a big mess of similarities, like a graph or a multidimensional space. The map is not the territory and so on. Terminology is necessary of course for communication and structuring books etc, but I like to see it merely as a utilitarian thing. The categories provide a scaffolding so that learning can happen. Studying the vocabulary and various outlines and nested hierarchies is a useful part of the journey but should not be confused with actually learning the thing itself. You could in principle learn all about regression without ever learning the word "regression". The name actually originates from statistics, named after the "regression towards the mean" phenomenon, where they observed how the adult height of people is closer to the average compared to the height of their parents. So a tall person will have tall children but less extremely tall (in tendency, some will of course be even taller). So confusingly, regression literally means "going back".
ellisv 6y ago
> Depends on the tradition you are coming from. Statistics :)
- bonoboTP 6y ago
  Yes, the statistics and machine learning research communities traditionally have a very different view of the world, of what is important, what is superfluous, but lately the two are more and more merging.
  - tomrod 6y ago
    Elements of Statistical Learning is a common entry point. I'd argue they are much closer than appears. As a subfield in computer science, of course the concern has often been on algorithmic complexity and similar. But that is nascency exposed, in my view, and likely not representative of a fully mature field. Armchair thought (not a historian of economics): I think Econometrics followed (and continues to follow) a similar evolution -- start with the goals (identification of model parameters, identifiability, KPIs), improve statistical validity and relevance, annotate dead ends or less common routes, continue on trucking.