6 ms·

Feature Learning. Deep Neural Networks can learn features from essentially raw data. Usual machine learning starts with features engineered manually. DNNs al

Feature Learning. Deep Neural Networks can learn features from essentially raw data. Usual machine learning starts with features engineered manually. DNNs also learn to predict from the features they learn, so you cold say (very roughly) "DNN = usual machine learning + feature learning". In practice manually engineering features is a time-consuming "guess-and-check" process which benefits from domain expertise. Feature Learning, otoh, is more automatic and benefits from data, computing resources, and optimization algorithms.

facepalm 10y ago
There are lots of other unsupervised learning methods in machine learning.
eshvk 10y ago
> Deep Neural Networks can learn features from essentially raw data. Usual machine learning starts with features engineered manually. What does this mean? What is "raw data" and what is a "feature engineered manually"?
- eegilbert 10y ago
  One example is this recent paper on learning (somewhat) high-level attributes of text from character streams alone (i.e., without telling the convolutional networks that things like words and punctuation exist). https://arxiv.org/abs/1502.01710 https://arxiv.org/abs/1502.01710
  - YeGoblynQueenne 10y ago
    >> We show that temporal ConvNets can achieve astonishing performance Yay, astonishing performance! I'm totally gonna waste half an hour of my life to read about what awesome badassery convnets are! Because that sounds so objective! /snark
- hacker42 10y ago
  Tens of thousands of engineers (audio, vision, linguists etc.) spent millions of hours for billions of dollars in the past 30 years to invent algorithms that reliably tell us something about a bunch of data. For example, an corner feature algorithm (such as SIFT) can extract the locations of corners in an image and characterize them. This is essential to many kinds of information processing tasks because we want to apply the same algorithm to different data (generalization), so we kind of need an interface to the data. This interface is called a feature (or feature algorithm, feature extractor or feature-descriptor). All of this work (some of these papers have on the order of ten thousands of citations) is now obsolete because you can start with a random initialization of the weights of a neural network and iteratively improve the weights using backprop for any kind of task. All you need a measure of improvement that is relatively smooth and differentiable with respect to the network weights. What is surprising is that the circuits and programs within reach of backprop training of fully connected neural networks are actually astonishingly good at what they do. But ultimately, this is maybe not so surprising given that our brains do something similar all the time.
  - Kip9000 10y ago
    >All of this work (some of these papers have on the order of ten thousands of citations) is now obsolete because you can start with a random initialization of the weights of a neural network and iteratively improve the weights using backprop for any kind of task Hardly correct. You can't magically learn any kind of task. You can't add arbitrary number of layers and hope for the back prop to do its magic. It is difficult. Deep learning techniques are what makes it somewhat feasible. SIFT is not obsolete because of NNs. They all have their pros and cons. You have to select the right tool for the job. BTW SIFT is not an edge detector (That's the Canny Transform). It describes images using salient features in scale invariant manner.
    - hacker42 10y ago
      Typos fixed. "All" was hyperbole of course, but I think it definitely does not look good for the majority of the work done on features. SIFT was recently outperformed PN-Net for example.
- Homunculiheaded 10y ago
  The Neural Network Playground is great for understanding this[0]! The default example is classification of a circle of one class surrounded by a donut of another. There are two features x_1 and x_2 (this is the "raw data"). One solution to this problem is to use a single layer and a single neuron but engineer features manually. These manually engineered features are x_1*x_2, x_1^2,x_2^2, sin(x_1) and sin(x_2). Here's a link to this model (long url)[1]. This model performs very well at learning to classify the data just by combining these manual features with a single neuron. The problem is a human needs to figure out these features. Try removing some and observe the different performance given different manual features. You'll see how important it is to engineer the correct ones. Alternatively you can have 2 layers of 4 neurons [2]. In nearly the exact number of iterations this network also learns to classify the data correctly. This is because the non-linear interactions between neurons are actually transforming the inputs the appropriate ways. That is to say the networks is learning to engineer the features itself. Try removing layers/nodes and you'll find that a simpler network will have a harder and harder time at this. I recommend playing around with the various tradeoff between manually engineered features and network complexity. The interesting thing you will observe is that in some cases the manual features are much faster to learn a simplier model than the network. The big issues comes up when we can't simply "see" the problem in 2d so we have no idea what features may and may not be useful. [0] http://playground.tensorflow.org/ http://playground.tensorflow.org/ [1] http://playground.tensorflow.org/#activation=tanh&batchSize=10&dataset=circle&regDataset=reg-plane&learningRate=0.03&regularizationRate=0&noise=0&networkShape=1&seed=0.66504&showTestData=false&discretize=false&percTrainData=50&x=true&y=true&xTimesY=true&xSquared=true&ySquared=true&cosX=false&sinX=true&cosY=false&sinY=true&collectStats=false&problem=classification&initZero=false http://playground.tensorflow.org/#activation=tanh&batchSize=... [2]. http://playground.tensorflow.org/#activation=tanh&batchSize=10&dataset=circle&regDataset=reg-plane&learningRate=0.03&regularizationRate=0&noise=0&networkShape=4,4&seed=0.66504&showTestData=false&discretize=false&percTrainData=50&x=true&y=true&xTimesY=false&xSquared=false&ySquared=false&cosX=false&sinX=false&cosY=false&sinY=false&collectStats=false&problem=classification&initZero=false http://playground.tensorflow.org/#activation=tanh&batchSize=...
Kip9000 10y ago
No. In all supervised learning, the algorithm learns a model from the data and generalise it for unseen data. Better generalisation means better model. In unsupervised learning, (mostly clustering, density estimation etc) the same thing happens but we don't tell the algorithm what to learn. Deep learning is not machine learning plus something else. It is a collection of techniques that overcomes the scalability problem of feed forward neural networks. NNs are very difficult to scale over number of layers. Standard training method of back propagation can't handle many layers because of vanishing gradient and the computational infeasibility brought on by the explosive growth of connections. NNs are very difficult to scale with additional classification targets you may require (for example, you have a classifier for categorising 10 classes, but to scale it up to 20, requires a lot of topological changes and qualitative analysis.) Deep learning addresses the scaling over layers with various techniques coupled with hardware acceleration (GPUs). Currently this stand at about 150 layers.
- Houshalter 10y ago
  Even experts often use the "feature learning" analogy. I don't think it's wrong, or at least a bad way of explaining it. The difference between (deep) neural networks and shallow machine learning, is that NNs can learn arbitrary features. Yes clustering doesn't require feature learning. But it is also super limited in the kinds of features it can learn. Neural nets can learn arbitrary circuits, and other types of functions.
  - nightski 10y ago
    Clustering also isn't a supervised learning technique. Even though you might say DNNs can be unsupervised (autoencoders), it generally is not the case in practical systems. So it's not a good comparison at all.
    - Houshalter 10y ago
      I don't care about the supervised/unsupervised distinction. I'm saying they can learn features automatically (with or without supervision.)
foobarqux 10y ago
Some of the prominent achievements in deep learning, like AlphaGo, used manually specified features.
- Kip9000 10y ago
  Have you got a reference?
  - daveguy 10y ago
    How about the AlphaGo paper itself: https://www.google.com/url?sa=t&source=web&rct=j&url=https://gogameguru.com/i/2016/03/deepmind-mastering-go.pdf&ved=0ahUKEwi-vITixJHNAhUJ4CYKHaXlAXcQFggbMAA&usg=AFQjCNHCNQ6W-K5UQUYhLCnJ77eZBi8q6Q&sig2=64MtrUT5y7ZtQULPn9yQCQ https://www.google.com/url?sa=t&source=web&rct=j&url=https:/... Bottom of page 23 and appendix tables around page 31 and 32. Edit: The stated big "next attempts" for them will be to learn these of features via an algorithm rather than handcoded. And to learn based on self play rather than a database of master games.
vonnik 10y ago
This Github article is too short to explain what it pretends to explain. Here's a brief overview of neural networks that does a better job (I wrote it, and it has pictures that illustrate some of the points made below): http://deeplearning4j.org/neuralnet-overview.html http://deeplearning4j.org/neuralnet-overview.html First, the term "deep learning" does not only apply to artificial neural networks, although that's what it's usually used as short hand for. It can also be used for reinforcement learning, for example. Second, "deep" is a technical term. In a neural network, it refers to the number of hidden layers. With NNs, you have so-called visible layers on the edge of the net for the input and the output. In between them you have the "hidden layers" stacked up. Deep means more than one hidden layer. So a shallow neural net of one hidden layer is not deep learning. Each of those layers is composed of nodes. And as the data passes through the net, it is recombined at each layer in ever more complex ensembles which are given different weights, or importance, with regard to the final decision the net makes (classification, clustering, regression). Deep NNs do two things that make them extremely powerful in relation to other machine-learning algorithms. They extract features from the data automatically and assign them importance in the process of learning. They do that using backpropagation over many hidden layers. Here's an example: Let's say you have an image and you're going to classify it using a feedforward network (that's the simplest kind of network, because the data simply passes through from one side to the other once and is scored). It's a digital image, so its pixels are just numbers, and those numbers can be flattened and arranged in a single column, so you just have a vector. That vector is passed to the net. Each pixel's number goes into a node of the net's input layer. (If your image was 28px*28px then you'd have an input layer of 784 nodes.) Each of those pixels is a feature of the image. But when humans recognize images in photos, they rarely examine the a photo pixel by pixel. And many photos might have the same sorts of pixels -- red or white or black -- but they would differ because the pixels would be arranged in different patterns. Those patterns are more complex features composed of simple pixels: a line, block or curve... In a neural network, those complex patterns are represented by the hidden layers. Each hidden layer represents increasingly complex features, based on the combinations of the layer before. So we would graduate from pixels in the input layer, to edges in the first hidden layer, to intersections of edges in the second hidden layer, and so forth until maybe we have nodes deep in the net that represent a nose or an ear extracted from a photo. This is called feature hierarchy, and it's one of the central characteristics of deep nets' structure. You can't do that with shallow nets. And deep nets allow simple features to recombine into more complex ones automatically. That is, you don't have to waste time with the feature engineering of most machine learning algorithms. (You waste time tuning your net's hyperparameters instead ;). A network learns how to assign the correct importance to various features via a process called backpropagation. Backpropagation is like reflux. You take something, and you send it back the way it came. In the case of neural nets, we're taking error. That is, every neural net starts out stupid, and it succeeds in learning something, it becomes less so. The measure of that stupidity is its error. And you calculate the error of a net by taking the guesses it makes about the data it sees, and comparing those guesses to a ground-truth label. Let's say you have a ton of images properly labeled as ship, cat, flamethrower, tiger and schoolbus. Each of those labels is one output node on the net. The output nodes represent the final combination of features, which coalesce in a label we care about. You pass the images into the network -- which operates behind a veil of ignorance and doesn't know the labels -- and you ask it to fire the output node that it thinks is the right guess. It does that, and almost always gets it wrong. Let's say it sees a photo of a cat but chooses flamethrower. You take the difference between those, some measure of error, and you backpropagate through the nodes of the network. Each node in the hidden layers uses coefficients to assign importance to input. That is, it's multiplying the numbers that pass through, in order to amplify or dampen them. Those coefficients, or weights, determine what guess the net makes about the data in the end. They have a causal relationship with the net's error. So if you change the weights, you can change the net's final decision. Backpropagation involves going to each weight, establishing its relationship with the final error (using a partial derivative), and then modifying the weight slightly in order to decrease the error. Once the weights are adjusted, more data is passed through the net; it makes more guesses; the error of those guesses is measured; and that error is used to adjust the weights. This goes on for a long time, basically until you can't shrink the error any more.

facepalm 10y ago

There are lots of other unsupervised learning methods in machine learning.

eshvk 10y ago

> Deep Neural Networks can learn features from essentially raw data. Usual machine learning starts with features engineered manually. What does this mean? What is "raw data" and what is a "feature engineered manually"?

eegilbert 10y ago
One example is this recent paper on learning (somewhat) high-level attributes of text from character streams alone (i.e., without telling the convolutional networks that things like words and punctuation exist). https://arxiv.org/abs/1502.01710 https://arxiv.org/abs/1502.01710
- YeGoblynQueenne 10y ago
  >> We show that temporal ConvNets can achieve astonishing performance Yay, astonishing performance! I'm totally gonna waste half an hour of my life to read about what awesome badassery convnets are! Because that sounds so objective! /snark
hacker42 10y ago
Tens of thousands of engineers (audio, vision, linguists etc.) spent millions of hours for billions of dollars in the past 30 years to invent algorithms that reliably tell us something about a bunch of data. For example, an corner feature algorithm (such as SIFT) can extract the locations of corners in an image and characterize them. This is essential to many kinds of information processing tasks because we want to apply the same algorithm to different data (generalization), so we kind of need an interface to the data. This interface is called a feature (or feature algorithm, feature extractor or feature-descriptor). All of this work (some of these papers have on the order of ten thousands of citations) is now obsolete because you can start with a random initialization of the weights of a neural network and iteratively improve the weights using backprop for any kind of task. All you need a measure of improvement that is relatively smooth and differentiable with respect to the network weights. What is surprising is that the circuits and programs within reach of backprop training of fully connected neural networks are actually astonishingly good at what they do. But ultimately, this is maybe not so surprising given that our brains do something similar all the time.
- Kip9000 10y ago
  >All of this work (some of these papers have on the order of ten thousands of citations) is now obsolete because you can start with a random initialization of the weights of a neural network and iteratively improve the weights using backprop for any kind of task Hardly correct. You can't magically learn any kind of task. You can't add arbitrary number of layers and hope for the back prop to do its magic. It is difficult. Deep learning techniques are what makes it somewhat feasible. SIFT is not obsolete because of NNs. They all have their pros and cons. You have to select the right tool for the job. BTW SIFT is not an edge detector (That's the Canny Transform). It describes images using salient features in scale invariant manner.
  - hacker42 10y ago
    Typos fixed. "All" was hyperbole of course, but I think it definitely does not look good for the majority of the work done on features. SIFT was recently outperformed PN-Net for example.
Homunculiheaded 10y ago
The Neural Network Playground is great for understanding this[0]! The default example is classification of a circle of one class surrounded by a donut of another. There are two features x_1 and x_2 (this is the "raw data"). One solution to this problem is to use a single layer and a single neuron but engineer features manually. These manually engineered features are x_1*x_2, x_1^2,x_2^2, sin(x_1) and sin(x_2). Here's a link to this model (long url)[1]. This model performs very well at learning to classify the data just by combining these manual features with a single neuron. The problem is a human needs to figure out these features. Try removing some and observe the different performance given different manual features. You'll see how important it is to engineer the correct ones. Alternatively you can have 2 layers of 4 neurons [2]. In nearly the exact number of iterations this network also learns to classify the data correctly. This is because the non-linear interactions between neurons are actually transforming the inputs the appropriate ways. That is to say the networks is learning to engineer the features itself. Try removing layers/nodes and you'll find that a simpler network will have a harder and harder time at this. I recommend playing around with the various tradeoff between manually engineered features and network complexity. The interesting thing you will observe is that in some cases the manual features are much faster to learn a simplier model than the network. The big issues comes up when we can't simply "see" the problem in 2d so we have no idea what features may and may not be useful. [0] http://playground.tensorflow.org/ http://playground.tensorflow.org/ [1] http://playground.tensorflow.org/#activation=tanh&batchSize=10&dataset=circle&regDataset=reg-plane&learningRate=0.03&regularizationRate=0&noise=0&networkShape=1&seed=0.66504&showTestData=false&discretize=false&percTrainData=50&x=true&y=true&xTimesY=true&xSquared=true&ySquared=true&cosX=false&sinX=true&cosY=false&sinY=true&collectStats=false&problem=classification&initZero=false http://playground.tensorflow.org/#activation=tanh&batchSize=... [2]. http://playground.tensorflow.org/#activation=tanh&batchSize=10&dataset=circle&regDataset=reg-plane&learningRate=0.03&regularizationRate=0&noise=0&networkShape=4,4&seed=0.66504&showTestData=false&discretize=false&percTrainData=50&x=true&y=true&xTimesY=false&xSquared=false&ySquared=false&cosX=false&sinX=false&cosY=false&sinY=false&collectStats=false&problem=classification&initZero=false http://playground.tensorflow.org/#activation=tanh&batchSize=...

Kip9000 10y ago

No. In all supervised learning, the algorithm learns a model from the data and generalise it for unseen data. Better generalisation means better model. In unsupervised learning, (mostly clustering, density estimation etc) the same thing happens but we don't tell the algorithm what to learn. Deep learning is not machine learning plus something else. It is a collection of techniques that overcomes the scalability problem of feed forward neural networks. NNs are very difficult to scale over number of layers. Standard training method of back propagation can't handle many layers because of vanishing gradient and the computational infeasibility brought on by the explosive growth of connections. NNs are very difficult to scale with additional classification targets you may require (for example, you have a classifier for categorising 10 classes, but to scale it up to 20, requires a lot of topological changes and qualitative analysis.) Deep learning addresses the scaling over layers with various techniques coupled with hardware acceleration (GPUs). Currently this stand at about 150 layers.

Houshalter 10y ago
Even experts often use the "feature learning" analogy. I don't think it's wrong, or at least a bad way of explaining it. The difference between (deep) neural networks and shallow machine learning, is that NNs can learn arbitrary features. Yes clustering doesn't require feature learning. But it is also super limited in the kinds of features it can learn. Neural nets can learn arbitrary circuits, and other types of functions.
- nightski 10y ago
  Clustering also isn't a supervised learning technique. Even though you might say DNNs can be unsupervised (autoencoders), it generally is not the case in practical systems. So it's not a good comparison at all.
  - Houshalter 10y ago
    I don't care about the supervised/unsupervised distinction. I'm saying they can learn features automatically (with or without supervision.)

foobarqux 10y ago

Some of the prominent achievements in deep learning, like AlphaGo, used manually specified features.

Kip9000 10y ago
Have you got a reference?
- daveguy 10y ago
  How about the AlphaGo paper itself: https://www.google.com/url?sa=t&source=web&rct=j&url=https://gogameguru.com/i/2016/03/deepmind-mastering-go.pdf&ved=0ahUKEwi-vITixJHNAhUJ4CYKHaXlAXcQFggbMAA&usg=AFQjCNHCNQ6W-K5UQUYhLCnJ77eZBi8q6Q&sig2=64MtrUT5y7ZtQULPn9yQCQ https://www.google.com/url?sa=t&source=web&rct=j&url=https:/... Bottom of page 23 and appendix tables around page 31 and 32. Edit: The stated big "next attempts" for them will be to learn these of features via an algorithm rather than handcoded. And to learn based on self play rather than a database of master games.

vonnik 10y ago

This Github article is too short to explain what it pretends to explain. Here's a brief overview of neural networks that does a better job (I wrote it, and it has pictures that illustrate some of the points made below): http://deeplearning4j.org/neuralnet-overview.html http://deeplearning4j.org/neuralnet-overview.html First, the term "deep learning" does not only apply to artificial neural networks, although that's what it's usually used as short hand for. It can also be used for reinforcement learning, for example. Second, "deep" is a technical term. In a neural network, it refers to the number of hidden layers. With NNs, you have so-called visible layers on the edge of the net for the input and the output. In between them you have the "hidden layers" stacked up. Deep means more than one hidden layer. So a shallow neural net of one hidden layer is not deep learning. Each of those layers is composed of nodes. And as the data passes through the net, it is recombined at each layer in ever more complex ensembles which are given different weights, or importance, with regard to the final decision the net makes (classification, clustering, regression). Deep NNs do two things that make them extremely powerful in relation to other machine-learning algorithms. They extract features from the data automatically and assign them importance in the process of learning. They do that using backpropagation over many hidden layers. Here's an example: Let's say you have an image and you're going to classify it using a feedforward network (that's the simplest kind of network, because the data simply passes through from one side to the other once and is scored). It's a digital image, so its pixels are just numbers, and those numbers can be flattened and arranged in a single column, so you just have a vector. That vector is passed to the net. Each pixel's number goes into a node of the net's input layer. (If your image was 28px*28px then you'd have an input layer of 784 nodes.) Each of those pixels is a feature of the image. But when humans recognize images in photos, they rarely examine the a photo pixel by pixel. And many photos might have the same sorts of pixels -- red or white or black -- but they would differ because the pixels would be arranged in different patterns. Those patterns are more complex features composed of simple pixels: a line, block or curve... In a neural network, those complex patterns are represented by the hidden layers. Each hidden layer represents increasingly complex features, based on the combinations of the layer before. So we would graduate from pixels in the input layer, to edges in the first hidden layer, to intersections of edges in the second hidden layer, and so forth until maybe we have nodes deep in the net that represent a nose or an ear extracted from a photo. This is called feature hierarchy, and it's one of the central characteristics of deep nets' structure. You can't do that with shallow nets. And deep nets allow simple features to recombine into more complex ones automatically. That is, you don't have to waste time with the feature engineering of most machine learning algorithms. (You waste time tuning your net's hyperparameters instead ;). A network learns how to assign the correct importance to various features via a process called backpropagation. Backpropagation is like reflux. You take something, and you send it back the way it came. In the case of neural nets, we're taking error. That is, every neural net starts out stupid, and it succeeds in learning something, it becomes less so. The measure of that stupidity is its error. And you calculate the error of a net by taking the guesses it makes about the data it sees, and comparing those guesses to a ground-truth label. Let's say you have a ton of images properly labeled as ship, cat, flamethrower, tiger and schoolbus. Each of those labels is one output node on the net. The output nodes represent the final combination of features, which coalesce in a label we care about. You pass the images into the network -- which operates behind a veil of ignorance and doesn't know the labels -- and you ask it to fire the output node that it thinks is the right guess. It does that, and almost always gets it wrong. Let's say it sees a photo of a cat but chooses flamethrower. You take the difference between those, some measure of error, and you backpropagate through the nodes of the network. Each node in the hidden layers uses coefficients to assign importance to input. That is, it's multiplying the numbers that pass through, in order to amplify or dampen them. Those coefficients, or weights, determine what guess the net makes about the data in the end. They have a causal relationship with the net's error. So if you change the weights, you can change the net's final decision. Backpropagation involves going to each weight, establishing its relationship with the final error (using a partial derivative), and then modifying the weight slightly in order to decrease the error. Once the weights are adjusted, more data is passed through the net; it makes more guesses; the error of those guesses is measured; and that error is used to adjust the weights. This goes on for a long time, basically until you can't shrink the error any more.