4 ms·

Even experts often use the "feature learning" analogy. I don't think it's wrong, or at least a bad way of explaining it. The difference between (deep) neural n

Even experts often use the "feature learning" analogy. I don't think it's wrong, or at least a bad way of explaining it. The difference between (deep) neural networks and shallow machine learning, is that NNs can learn arbitrary features. Yes clustering doesn't require feature learning. But it is also super limited in the kinds of features it can learn. Neural nets can learn arbitrary circuits, and other types of functions.

nightski 10y ago
Clustering also isn't a supervised learning technique. Even though you might say DNNs can be unsupervised (autoencoders), it generally is not the case in practical systems. So it's not a good comparison at all.
- Houshalter 10y ago
  I don't care about the supervised/unsupervised distinction. I'm saying they can learn features automatically (with or without supervision.)
  - YeGoblynQueenne 10y ago
    Well, feature learning is unsupervised by necessity, otherwise you're not learning features, you're learning a mapping between features and labels. Deep nets used in the way you say are first trained unsupervised to extract features, then the features are used in supervised learning, to learn a mapping from those new features to labels. You can also do this "by hand" using unsupervised learning techniques like clustering, Principal Component Analysis etc: you make your own features then, and train a classifier afterwards, on the features you extracted in that way. Deep nets just sort of automate the process.
halflings 10y ago
Gaussian Processes can learn any arbitrary function. Is it "shallow" machine learning? I think the point that the parent makes is valid: Most of the advantages of deep learning when using a "simple" feed-forward topology is advances related to scaling learning and solving problems encountered at with difficult tasks like image recognition, etc. I do not know enough about neural nets to say if that is all there is to it, but one thing is sure: it's not just about "learning features", although it was shown that the output at every layer abstracts some sort of higher-level features (in the case of image recognition)
- Houshalter 10y ago
  GPs require exponentially many parameters though. They can't learn arbitrary functions, they just stupidly memorize a lookup table.
- YeGoblynQueenne 10y ago
  >> it's not just about "learning features" So, I'm in no position to prove this, but my intuition is that any machine learning algorithm can be configured in a semi-supervised learning set-up, like deep nets have. You could train a decision forest classifier for instance to learn in an unsupervised manner. An algorithm I'm developing for my MSc dissertation is essentially unsupervised recursive partitioning, a.k.a. decision trees (only, first-order rather than propositional). Well, possibly not _any_ algorithm. But I get the feeling that many classifiers in particular could be adapted to unsupervised learning with a bit of elbow grease, at which point you could connect them to their own input and, voila, semi-supervised learning. But like I say, I don't reckon I'll be in a position to prove this any time soon.
YeGoblynQueenne 10y ago
The way I know it goes along the lines of: "multilayer perceptrons with no more than three layers can learn any function to arbitrary precision given a large enough number of inputs". But like halflings say, neural nets are not alone in this. Decision Trees can learn any binary decision diagram I guess (they can encode arbitrary disjunctions of conjunctions). I'm pretty sure there are similar results for other algorithms also. In any case, you can represent a function as a set-theoretical relation and enumerate its parameters- and there you go, learning done with arbitrary precision. That's not what makes neural nets impressive. So what is it? "Shallow machine learning" is a worrying neologism. "Shallow" and "deep" only apply to neural networks, really. You couldn't very well distinguish between shallow and deep K-NN classifiers, say. Or shallow and deep k-means clustering. I mean, what the hell?

nightski 10y ago

Clustering also isn't a supervised learning technique. Even though you might say DNNs can be unsupervised (autoencoders), it generally is not the case in practical systems. So it's not a good comparison at all.

Houshalter 10y ago
I don't care about the supervised/unsupervised distinction. I'm saying they can learn features automatically (with or without supervision.)
- YeGoblynQueenne 10y ago
  Well, feature learning is unsupervised by necessity, otherwise you're not learning features, you're learning a mapping between features and labels. Deep nets used in the way you say are first trained unsupervised to extract features, then the features are used in supervised learning, to learn a mapping from those new features to labels. You can also do this "by hand" using unsupervised learning techniques like clustering, Principal Component Analysis etc: you make your own features then, and train a classifier afterwards, on the features you extracted in that way. Deep nets just sort of automate the process.

halflings 10y ago

Gaussian Processes can learn any arbitrary function. Is it "shallow" machine learning? I think the point that the parent makes is valid: Most of the advantages of deep learning when using a "simple" feed-forward topology is advances related to scaling learning and solving problems encountered at with difficult tasks like image recognition, etc. I do not know enough about neural nets to say if that is all there is to it, but one thing is sure: it's not just about "learning features", although it was shown that the output at every layer abstracts some sort of higher-level features (in the case of image recognition)

Houshalter 10y ago
GPs require exponentially many parameters though. They can't learn arbitrary functions, they just stupidly memorize a lookup table.
YeGoblynQueenne 10y ago
>> it's not just about "learning features" So, I'm in no position to prove this, but my intuition is that any machine learning algorithm can be configured in a semi-supervised learning set-up, like deep nets have. You could train a decision forest classifier for instance to learn in an unsupervised manner. An algorithm I'm developing for my MSc dissertation is essentially unsupervised recursive partitioning, a.k.a. decision trees (only, first-order rather than propositional). Well, possibly not _any_ algorithm. But I get the feeling that many classifiers in particular could be adapted to unsupervised learning with a bit of elbow grease, at which point you could connect them to their own input and, voila, semi-supervised learning. But like I say, I don't reckon I'll be in a position to prove this any time soon.

YeGoblynQueenne 10y ago

The way I know it goes along the lines of: "multilayer perceptrons with no more than three layers can learn any function to arbitrary precision given a large enough number of inputs". But like halflings say, neural nets are not alone in this. Decision Trees can learn any binary decision diagram I guess (they can encode arbitrary disjunctions of conjunctions). I'm pretty sure there are similar results for other algorithms also. In any case, you can represent a function as a set-theoretical relation and enumerate its parameters- and there you go, learning done with arbitrary precision. That's not what makes neural nets impressive. So what is it? "Shallow machine learning" is a worrying neologism. "Shallow" and "deep" only apply to neural networks, really. You couldn't very well distinguish between shallow and deep K-NN classifiers, say. Or shallow and deep k-means clustering. I mean, what the hell?