4 ms·

I've studied machine learning a fair amount, and the algorithms are fancier, but it's still all model fitting by minimizing error. Regression is also model fit

by yters 8y ago

I've studied machine learning a fair amount, and the algorithms are fancier, but it's still all model fitting by minimizing error. Regression is also model fittining (polynomials) by minimizing mean squared error. Not a huge difference...

kgwgk 8y ago
But they have invented gradient descent! /s
YeGoblynQueenne 8y ago
>> I've studied machine learning a fair amount, and the algorithms are fancier, but it's still all model fitting by minimizing error. That's only true if you define machine learning as "model fitting", however there are many algorithms and techniques that have nothing to do with fitting a curve to data points and are very commonly considered to be machine learning. E.g. decision tree learners, nearest-neighbour algorithms, Bayesian inference algorithms, Expectation-maximisation (and friends), clustering algorithms- none of these really um fit under "curve fitting", although of course you can broaden the scope of what is meant by "model fitting" to include potentially anything that looks kinda like it from a few steps away. Then of course there is the very broad family of logic-based machine learning systems, pioneered by Ryszard Michalski and others in the '70s and Inductive Logic Programming, that learn logic theories from relational data and do not perform optimisation or model fitting (they tend to be greedy). Basically, it's mostly the gradient-descent based algorithms that do curve fitting (linear regression, perceptrons, SVMs and, er, that; oh, and gradient boosting). These approaches are currently dominant but that's all. There is much more to machine learning than curve fitting.
- pas 8y ago
  All of those require a training step, and that is the same error minimization as curve fitting. Basically they are curve fitting in a high enough dimension. Even theory builders can be thought of as curve fitters. The error metric is greedy, the curve is not differentable, but that's it. That said, of course there is much more to it, than training. (And as we saw with OpenAI's dota bots and DeepMind's AlphaGo Zero, with good feature engineering, adversarial learning setup, beating human performance is only a matter of processing power.)
  - deleted 8y ago
    [deleted]
  - YeGoblynQueenne 8y ago
    >> All of those require a training step, and that is the same error minimization as curve fitting. >> Basically they are curve fitting in a high enough dimension. >> Even theory builders can be thought of as curve fitters. The error metric is greedy, the curve is not differentable, but that's it. I don't understand most of what you are saying and the bits I understand, I don't see how they're true. Could you please explain in a little more detail? For example, how is, say, recursive partitioning, as in decision tree learners, or storing data points for later comparison, as in KNN, "curve fitting in a high enough dimension"? What is the function being fit and where are its parameters? Full disclosure: I study ILP algorithms for my PhD and I have implemented a couple myself. There is nothing like optimisation going on in there, just (inverse) resolution and some ordering of the search space. Most have nothing like an "error metric", greedy or not. And I have never heard anyone describe logic-based learners, including ILP algorithms, as curve fitting before.
    - pas 8y ago
      If something requires training, it constructs a model, the model can be represented as a probability distribution. I think of that as the curve. ILP (let's say simplex algorithm or other hill climbers) don't require training, but they nevertheless try to optimize the utility (or goal) function (which implicitly defines a metric on the space), which is a hypersurface, which is a curve in an even higher dimension. Since the cardinality of the set of solutions is not one, and usually infinite, picking one solution over the other is optimization. (And - if I'm not mistaken - unless we know that the search space is convex, we don't know if there are better solutions or not.) I don't think anything that doesn't build a model is machine learning. (Decision trees build a model, Bayes classifiers build a model, etc. And I'd say all are curve fitting.) I'm not doing a PhD at all, so I'm not claiming that I'm an authority on each correct terminus technicus for AI/ML/optimization math/CompSci, but to me they fall under the model/curve fitting umbrella.
      - deleted 8y ago
YeGoblynQueenne 8y ago
>> I've studied machine learning a fair amount, OK, different comment because this one might be construed as a bit personal. I mean, I really hope it's not, but I've heard one too many people recently claim they know about machine learning who only mean they've heard of it in the last five or six years (2012 is the cut-off point) and only know the algorithms that are typically taught in machine learning courses at degree and Master's levels (perceptrons, linear regression, decision trees, naive Bayes, logistic regression, KNN, SVMs, K-means, maybe some deep learning bits) and who wouldn't know what the hell CART is or who Arthur Samuel, or, indeed, Warren McCulloch and Walter Pitts, were. So I have to ask: when you say "a fair amount"- what exactly do you mean?

by yters 8y ago

kgwgk 8y ago

But they have invented gradient descent! /s

YeGoblynQueenne 8y ago

>> I've studied machine learning a fair amount, and the algorithms are fancier, but it's still all model fitting by minimizing error. That's only true if you define machine learning as "model fitting", however there are many algorithms and techniques that have nothing to do with fitting a curve to data points and are very commonly considered to be machine learning. E.g. decision tree learners, nearest-neighbour algorithms, Bayesian inference algorithms, Expectation-maximisation (and friends), clustering algorithms- none of these really um fit under "curve fitting", although of course you can broaden the scope of what is meant by "model fitting" to include potentially anything that looks kinda like it from a few steps away. Then of course there is the very broad family of logic-based machine learning systems, pioneered by Ryszard Michalski and others in the '70s and Inductive Logic Programming, that learn logic theories from relational data and do not perform optimisation or model fitting (they tend to be greedy). Basically, it's mostly the gradient-descent based algorithms that do curve fitting (linear regression, perceptrons, SVMs and, er, that; oh, and gradient boosting). These approaches are currently dominant but that's all. There is much more to machine learning than curve fitting.

pas 8y ago
All of those require a training step, and that is the same error minimization as curve fitting. Basically they are curve fitting in a high enough dimension. Even theory builders can be thought of as curve fitters. The error metric is greedy, the curve is not differentable, but that's it. That said, of course there is much more to it, than training. (And as we saw with OpenAI's dota bots and DeepMind's AlphaGo Zero, with good feature engineering, adversarial learning setup, beating human performance is only a matter of processing power.)
- deleted 8y ago
  [deleted]
- YeGoblynQueenne 8y ago
  >> All of those require a training step, and that is the same error minimization as curve fitting. >> Basically they are curve fitting in a high enough dimension. >> Even theory builders can be thought of as curve fitters. The error metric is greedy, the curve is not differentable, but that's it. I don't understand most of what you are saying and the bits I understand, I don't see how they're true. Could you please explain in a little more detail? For example, how is, say, recursive partitioning, as in decision tree learners, or storing data points for later comparison, as in KNN, "curve fitting in a high enough dimension"? What is the function being fit and where are its parameters? Full disclosure: I study ILP algorithms for my PhD and I have implemented a couple myself. There is nothing like optimisation going on in there, just (inverse) resolution and some ordering of the search space. Most have nothing like an "error metric", greedy or not. And I have never heard anyone describe logic-based learners, including ILP algorithms, as curve fitting before.
  - pas 8y ago
    If something requires training, it constructs a model, the model can be represented as a probability distribution. I think of that as the curve. ILP (let's say simplex algorithm or other hill climbers) don't require training, but they nevertheless try to optimize the utility (or goal) function (which implicitly defines a metric on the space), which is a hypersurface, which is a curve in an even higher dimension. Since the cardinality of the set of solutions is not one, and usually infinite, picking one solution over the other is optimization. (And - if I'm not mistaken - unless we know that the search space is convex, we don't know if there are better solutions or not.) I don't think anything that doesn't build a model is machine learning. (Decision trees build a model, Bayes classifiers build a model, etc. And I'd say all are curve fitting.) I'm not doing a PhD at all, so I'm not claiming that I'm an authority on each correct terminus technicus for AI/ML/optimization math/CompSci, but to me they fall under the model/curve fitting umbrella.
    - deleted 8y ago

YeGoblynQueenne 8y ago

>> I've studied machine learning a fair amount, OK, different comment because this one might be construed as a bit personal. I mean, I really hope it's not, but I've heard one too many people recently claim they know about machine learning who only mean they've heard of it in the last five or six years (2012 is the cut-off point) and only know the algorithms that are typically taught in machine learning courses at degree and Master's levels (perceptrons, linear regression, decision trees, naive Bayes, logistic regression, KNN, SVMs, K-means, maybe some deep learning bits) and who wouldn't know what the hell CART is or who Arthur Samuel, or, indeed, Warren McCulloch and Walter Pitts, were. So I have to ask: when you say "a fair amount"- what exactly do you mean?