9 ms·

Since the fall of AI, there are two groups of people in this topic -- one trying to make some reproducible, robust results with well defined algorithms and seco

by mbq 14y ago

Since the fall of AI, there are two groups of people in this topic -- one trying to make some reproducible, robust results with well defined algorithms and second importing random ideas from the first group onto some questionably defined ANN model and getting all the hype because of the "neural" buzzword. "Deep learning" is actually called boosting and has been around for years.

iskander 14y ago
Boosting selects successive weak learners for the same classification problem, but under a changing distribution/weighting of the input space. Deep learning stacks complex models to create increasingly abstract representations. All I can really imagine them having in common is (1) they're both families of machine learning techniques and (2) they both (roughly) involve a collection of models, albeit in very different ways.
- mbq 14y ago
  You're talking about Adaboost; general boosting can use any models and the only idea there is that it adds new model to fix residuals of the current chain. BTW "increasingly abstract representation" is a perfect example of this meaningless PR ANNs are built of.
  - wookietrader 14y ago
    No. Deep learning is not about fixing the residuals of the current chain. Deep learning isn't even about residuals in the first place. It's about (1) finding good representations of your data (aka feature learning), (2) then adding a discriminative model on top and then (3) tuning everything. There is no relation to boosting at all.
robrenaud 14y ago
Unsupervised pre-training is fundamentally different than boosting. Boosting is a clever way of modelling a conditional distribution. The insight behind the success of pre-training is that, for many perceptual tasks, having a good model of the input (rather than the input->output mapping) is key. I have no delusion that the algorithms that work for training deep networks are anything like what the brain actually does, but I don't care. There are many tasks where deep neural nets are state of the art.
- osdf 14y ago
  Not to argue with you, robrenaud, but Hinton himself writes in their 2006 paper 'A Fast Learning Algorithm for Deep Belief Nets': The greedy algorithm bears some resemblance to boosting in its repeated use of the same “weak” learner, but instead of reweighting each data vector to ensure that the next step learns something new, it re- represents it. I guess that most people however would not think of this interpretation of greedily pretraining deep networks :). (I wonder if mbq had this in mind). In the same article your point about good models of the input is mentioned, too (only copy&paste a small part of the paragraph): Unsupervised methods, however, can use very large unlabeled data sets, and each case may be very high-dimensional, thus providing many bits of constraint on a generative model. The 2006 paper is really an amazing read in my opinion.
gdahl 14y ago
Deep learning is not boosting at all. Deep learning is about composing trainable modules. Adding a layer f(x) to a layer g(x) to get h(x) = f(g(x)). Boosting creates a final classifier that is a weighted sum of the base classifiers, or something like h(x) = a * f(x) + b * g(x). Composition is what Professor Hinton means when he says "re-represent the input" and other similar phrases.

by mbq 14y ago

iskander 14y ago

Boosting selects successive weak learners for the same classification problem, but under a changing distribution/weighting of the input space. Deep learning stacks complex models to create increasingly abstract representations. All I can really imagine them having in common is (1) they're both families of machine learning techniques and (2) they both (roughly) involve a collection of models, albeit in very different ways.

mbq 14y ago
You're talking about Adaboost; general boosting can use any models and the only idea there is that it adds new model to fix residuals of the current chain. BTW "increasingly abstract representation" is a perfect example of this meaningless PR ANNs are built of.
- wookietrader 14y ago
  No. Deep learning is not about fixing the residuals of the current chain. Deep learning isn't even about residuals in the first place. It's about (1) finding good representations of your data (aka feature learning), (2) then adding a discriminative model on top and then (3) tuning everything. There is no relation to boosting at all.

robrenaud 14y ago

Unsupervised pre-training is fundamentally different than boosting. Boosting is a clever way of modelling a conditional distribution. The insight behind the success of pre-training is that, for many perceptual tasks, having a good model of the input (rather than the input->output mapping) is key. I have no delusion that the algorithms that work for training deep networks are anything like what the brain actually does, but I don't care. There are many tasks where deep neural nets are state of the art.

osdf 14y ago
Not to argue with you, robrenaud, but Hinton himself writes in their 2006 paper 'A Fast Learning Algorithm for Deep Belief Nets': The greedy algorithm bears some resemblance to boosting in its repeated use of the same “weak” learner, but instead of reweighting each data vector to ensure that the next step learns something new, it re- represents it. I guess that most people however would not think of this interpretation of greedily pretraining deep networks :). (I wonder if mbq had this in mind). In the same article your point about good models of the input is mentioned, too (only copy&paste a small part of the paragraph): Unsupervised methods, however, can use very large unlabeled data sets, and each case may be very high-dimensional, thus providing many bits of constraint on a generative model. The 2006 paper is really an amazing read in my opinion.

gdahl 14y ago

Deep learning is not boosting at all. Deep learning is about composing trainable modules. Adding a layer f(x) to a layer g(x) to get h(x) = f(g(x)). Boosting creates a final classifier that is a weighted sum of the base classifiers, or something like h(x) = a * f(x) + b * g(x). Composition is what Professor Hinton means when he says "re-represent the input" and other similar phrases.