3 ms·

Just to summarize: "Deep learning" is the new big trend in Machine Learning. It promises general, powerful, and fast machine learning, moving us one step close

by bravura 17y ago

Just to summarize: "Deep learning" is the new big trend in Machine Learning. It promises general, powerful, and fast machine learning, moving us one step closer to AI. An algorithm is deep if the input is passed through several non-linearities before being output. Most modern learning algorithms (including decision trees and SVMs and naive bayes) are "shallow". For intuition, imagine if I told you that your main routine can call subroutines, and your subroutines could call subsubroutines, but you couldn't have any more abstraction than that. You can't have subsubsubroutines in your "shallow" program. You could compute whatever you wanted in a "shallow" program, but your code would involve a lot of duplicated code and would not be as compact as it should be. Similarly, a shallow machine learning architecture would involve a lot of duplication of effort to express things that a deep architecture could more compactly. The point being, a deep architecture can more gracefully reuse previous computations. Deep learning is motivated by intuition, theoretical arguments from circuit theory, empirical results, and current knowledge of neuroscience. Here is a video where I give a high-level talk on deep learning, and describe this intuition: http://bilconference.com/videos/deep-learning-artificial-intelligence-joseph-turian/ http://bilconference.com/videos/deep-learning-artificial-int... Here is more detailed information about deep learning: http://deeplearning.net/tutorial/ http://deeplearning.net/tutorial/ Deep learning techniques can typically be expressed as dense matrix operations, making them suitable to being sped up using GPUs. Which makes deep learning even more powerful.

lliiffee 17y ago
Since you seem to know this stuff, I hope you doing mind if I ask you a marginally related question... It seems like deep learning is based on greedily building representations in an unsupervised setting. (Not sure if this is a defining characteristic of deep learning or just a common usage.) The postulate seems to be that greedily learning in a supervised setting is inferior. Why should this be so? Is there any justification (theoretical or experimental)? I can only find vague asides on this point in a few papers.
- Groxx 17y ago
  It's not by any means required to be unsupervised, they just tend to be better suited for it, from what I remember. You can train a deep neural network just like a shallow one, but do you know you're training it in the best way? The unsupervised one might key in on something you're not aware of. They are also more capable of dealing with tons of variables (in this case, ~100 million). If you had 100 million yes/no values that needed to be checked, could you really answer all of them accurately? In some cases, sure, but not many, and what's the correct number of variables for simulating a language, and what do they relate to? An unsupervised network will try to fit them automagically, rather than taking your word for it. Unsupervised learning also poses a lot of intelligence-modeling uses, because we're essentially unsupervised for some of the "hardest" tasks to learn. Especially prior to learning to speak. All there really is for feedback is pain, and that doesn't necessarily lead directly to language. But bravura should be able to answer better / more accurately; hopefully they'll chime in too. I'd like to know if there's more to it than what I know, which is pretty minimal.
- bravura 17y ago
  If you look at the video, I try to give some intuition about this. The following explanation is an oversimplification to give you intuition into what is going on: The difficulty in training a deep architecture is that, if you use standard backprop, the gradient signal doesn't flow back to the lowest layers. Instead, the few supervised output units pass back a small gradient signal, and at each layer the gradient signal has increasing noise as it gets passed backwards. So the top layers overfit, and the lower layers don't get tuned effectively. The lower layers essentially fire random noise, and the top layers overfit. For this reason, before 2006, no one knew how to train a deep architecture (besides Yann LeCun, with his convolutional architectures, but those were not as general purpose). The breakthrough in 2006 came when Hinton came up with the original DBN algorithm. Bengio et al (http://www.iro.umontreal.ca/~lisa/publications/index.php?page=publication&kind=single&ID=190 http://www.iro.umontreal.ca/~lisa/publications/index.php?pag...) followed up by teasing apart the important steps in training a deep architecture. They were: greedy layerwise training, i.e. construct one layer, then the next layer, then the next layer, etc. unsupervised pretraining When you train a single layer using an unsupervised criterion, the gradient signal is passed backwards through only one hidden layer (the layer you are constructing). And the output layer for an unsupervised criterion has as many units as the input. So the output layer passes back a strong gradient signal, and it doesn't have to travel very far. By doing this unsupervised pretraining in a layer-by-layer manner, the deep network receives a good initialization of its parameters. Then, when finetuning against the supervised criterion using backprop, you can find a better local minimum. The work of Erhan et al show that the effect of unsupervised pretraining is not only regularization, but also improved optimization. Take a look at there work for a great empirical study with large scale experiments and pretty graphs: http://jmlr.csail.mit.edu/proceedings/papers/v9/erhan10a/erhan10a.pdf http://jmlr.csail.mit.edu/proceedings/papers/v9/erhan10a/erh... p.s. feel free to send me an email to followup.
Groxx 17y ago
After taking a neural networks course, I can't see why people think "shallow" approaches will ever work. If you want to simulate intelligence, start by simulating intelligent brains: deep layers and deep recursion. It's not particularly provable, but it works damn well. And yeah, everyone wants something they can prove... but I side with the "make something useful" crowd before "make something provable". And fantastic links, thanks! I've been meaning to get further into the field, but I've been swamped, don't know where to start, and have taken the only class on it at my college.

by bravura 17y ago

lliiffee 17y ago

Since you seem to know this stuff, I hope you doing mind if I ask you a marginally related question... It seems like deep learning is based on greedily building representations in an unsupervised setting. (Not sure if this is a defining characteristic of deep learning or just a common usage.) The postulate seems to be that greedily learning in a supervised setting is inferior. Why should this be so? Is there any justification (theoretical or experimental)? I can only find vague asides on this point in a few papers.

Groxx 17y ago
It's not by any means required to be unsupervised, they just tend to be better suited for it, from what I remember. You can train a deep neural network just like a shallow one, but do you know you're training it in the best way? The unsupervised one might key in on something you're not aware of. They are also more capable of dealing with tons of variables (in this case, ~100 million). If you had 100 million yes/no values that needed to be checked, could you really answer all of them accurately? In some cases, sure, but not many, and what's the correct number of variables for simulating a language, and what do they relate to? An unsupervised network will try to fit them automagically, rather than taking your word for it. Unsupervised learning also poses a lot of intelligence-modeling uses, because we're essentially unsupervised for some of the "hardest" tasks to learn. Especially prior to learning to speak. All there really is for feedback is pain, and that doesn't necessarily lead directly to language. But bravura should be able to answer better / more accurately; hopefully they'll chime in too. I'd like to know if there's more to it than what I know, which is pretty minimal.
bravura 17y ago
If you look at the video, I try to give some intuition about this. The following explanation is an oversimplification to give you intuition into what is going on: The difficulty in training a deep architecture is that, if you use standard backprop, the gradient signal doesn't flow back to the lowest layers. Instead, the few supervised output units pass back a small gradient signal, and at each layer the gradient signal has increasing noise as it gets passed backwards. So the top layers overfit, and the lower layers don't get tuned effectively. The lower layers essentially fire random noise, and the top layers overfit. For this reason, before 2006, no one knew how to train a deep architecture (besides Yann LeCun, with his convolutional architectures, but those were not as general purpose). The breakthrough in 2006 came when Hinton came up with the original DBN algorithm. Bengio et al (http://www.iro.umontreal.ca/~lisa/publications/index.php?page=publication&kind=single&ID=190 http://www.iro.umontreal.ca/~lisa/publications/index.php?pag...) followed up by teasing apart the important steps in training a deep architecture. They were: greedy layerwise training, i.e. construct one layer, then the next layer, then the next layer, etc. unsupervised pretraining When you train a single layer using an unsupervised criterion, the gradient signal is passed backwards through only one hidden layer (the layer you are constructing). And the output layer for an unsupervised criterion has as many units as the input. So the output layer passes back a strong gradient signal, and it doesn't have to travel very far. By doing this unsupervised pretraining in a layer-by-layer manner, the deep network receives a good initialization of its parameters. Then, when finetuning against the supervised criterion using backprop, you can find a better local minimum. The work of Erhan et al show that the effect of unsupervised pretraining is not only regularization, but also improved optimization. Take a look at there work for a great empirical study with large scale experiments and pretty graphs: http://jmlr.csail.mit.edu/proceedings/papers/v9/erhan10a/erhan10a.pdf http://jmlr.csail.mit.edu/proceedings/papers/v9/erhan10a/erh... p.s. feel free to send me an email to followup.

Groxx 17y ago

After taking a neural networks course, I can't see why people think "shallow" approaches will ever work. If you want to simulate intelligence, start by simulating intelligent brains: deep layers and deep recursion. It's not particularly provable, but it works damn well. And yeah, everyone wants something they can prove... but I side with the "make something useful" crowd before "make something provable". And fantastic links, thanks! I've been meaning to get further into the field, but I've been swamped, don't know where to start, and have taken the only class on it at my college.