7 ms·

I have no idea if it is the breakthrough of the decade, but I think deep learning isn't just taking a perceptron with many hidden layers and applying backpropag

by stiff 14y ago

I have no idea if it is the breakthrough of the decade, but I think deep learning isn't just taking a perceptron with many hidden layers and applying backpropagation to it, as you seem to say, all the interesting things about it you summarized as "fancy stuff" and "not making a big difference", without any context, references or arguments. I do not feel competent to discuss it as I have very little experience in this field, but it doesn't feel too informed even given whatever little knowledge I have. Certainly faster computers and more data have helped, but just like in traditional algorithms research, they cannot completely make up for having exponential growth functions with respect to computational needs of the amount of data required. There have been large improvements in both respects in the deep learning community, in fact rarely does the term "deep learning" refer in practice to traditional completely supervised learning that you are talking about. There are nice and more balanced overviews here: http://ufldl.stanford.edu/wiki/index.php/Deep_Networks:_Overview http://ufldl.stanford.edu/wiki/index.php/Deep_Networks:_Over... http://en.wikipedia.org/wiki/Deep_learning http://en.wikipedia.org/wiki/Deep_learning

ergodic 14y ago
If it was not clear enough, "fancy stuff" and "not making a big difference" refers to Seide et al 2011 mentioned in the same paragraph. Table 2 is particularly revealing to this regard. http://research.microsoft.com/apps/pubs/default.aspx?id=153169 http://research.microsoft.com/apps/pubs/default.aspx?id=1531... As I said I can only speak with more or less certainty regarding ASR. I am fairly sure that the success in ASR (with Google and MS embracing DNNs for ASR) contribute significantly to the mainstream impact of deep learning.
- stiff 14y ago
  There is a second paper where they specifically point out the differences between their approach and previous approaches using neural networks and it isn't only the number of layers that has changed but also the internal architecture of the network, the "responsibilities" of the layers, so again, it isn't just a traditionally trained MLP with a lot of layers: http://research.microsoft.com/pubs/157341/FeatureEngineeringInCD-DNN-ASRU2011-pub.pdf http://research.microsoft.com/pubs/157341/FeatureEngineering...
  - ergodic 14y ago
    I read it in diagonal but the paper seems to use the same DNN architecture as before. They seem to tweak the pretraining with layer-wise back-propagation (instead of full MLP-as-DBN pre-training). This does not imply anything new with respect to what I commented and the cited paper. The only reference to differences I found is about differences between a DNN and a MaxEnt models, which is again not an argument for differences between DNNs and MLPs. Could you point me to a concrete paragraph?, I would be happy to be mistaken in this regard.
    - jph00 14y ago
      DNNs can be thought of a stacked Restricted Boltzmann Machines. Their structure and training is very different to traditional MLPs. They derive in some ways from convolutional neural nets. I describe some of the key differences between DNNs and MLPs in the webinar. Also, the webinar explains how recent advances go far beyond just applications to speech recognition - in particular I focus on a case study in chemoinformatics.
      - ergodic 14y ago
        >DNNs can be thought of a stacked Restricted Boltzmann Machines Agree, as explained in Hinton et al 2006. http://www.cs.toronto.edu/~hinton/absps/ncfast.pdf http://www.cs.toronto.edu/~hinton/absps/ncfast.pdf But this is just for pre-training, as I said. If you look at Seides paper, they pre-train treating the MLP as a DBN and then they train it as a classic MLP with BP. Also using layer-wise BP pre-training does bring performance close to DBN pre-training, with no use of DBNs paradigms at all. >Their structure and training is very different to traditional MLPs I insist if we are talking of the same DNNs explained in Microsofts paper, this is not true. If we were to be talking about different DNNs please elaborate I would love to hear about that (seriously, no irony here).
        jph00 14y ago
        There's also the random knockout of neurons, as mentioned in the webinar.

by stiff 14y ago

ergodic 14y ago

If it was not clear enough, "fancy stuff" and "not making a big difference" refers to Seide et al 2011 mentioned in the same paragraph. Table 2 is particularly revealing to this regard. http://research.microsoft.com/apps/pubs/default.aspx?id=153169 http://research.microsoft.com/apps/pubs/default.aspx?id=1531... As I said I can only speak with more or less certainty regarding ASR. I am fairly sure that the success in ASR (with Google and MS embracing DNNs for ASR) contribute significantly to the mainstream impact of deep learning.

stiff 14y ago
There is a second paper where they specifically point out the differences between their approach and previous approaches using neural networks and it isn't only the number of layers that has changed but also the internal architecture of the network, the "responsibilities" of the layers, so again, it isn't just a traditionally trained MLP with a lot of layers: http://research.microsoft.com/pubs/157341/FeatureEngineeringInCD-DNN-ASRU2011-pub.pdf http://research.microsoft.com/pubs/157341/FeatureEngineering...
- ergodic 14y ago
  I read it in diagonal but the paper seems to use the same DNN architecture as before. They seem to tweak the pretraining with layer-wise back-propagation (instead of full MLP-as-DBN pre-training). This does not imply anything new with respect to what I commented and the cited paper. The only reference to differences I found is about differences between a DNN and a MaxEnt models, which is again not an argument for differences between DNNs and MLPs. Could you point me to a concrete paragraph?, I would be happy to be mistaken in this regard.
  - jph00 14y ago
    DNNs can be thought of a stacked Restricted Boltzmann Machines. Their structure and training is very different to traditional MLPs. They derive in some ways from convolutional neural nets. I describe some of the key differences between DNNs and MLPs in the webinar. Also, the webinar explains how recent advances go far beyond just applications to speech recognition - in particular I focus on a case study in chemoinformatics.
    - ergodic 14y ago
      >DNNs can be thought of a stacked Restricted Boltzmann Machines Agree, as explained in Hinton et al 2006. http://www.cs.toronto.edu/~hinton/absps/ncfast.pdf http://www.cs.toronto.edu/~hinton/absps/ncfast.pdf But this is just for pre-training, as I said. If you look at Seides paper, they pre-train treating the MLP as a DBN and then they train it as a classic MLP with BP. Also using layer-wise BP pre-training does bring performance close to DBN pre-training, with no use of DBNs paradigms at all. >Their structure and training is very different to traditional MLPs I insist if we are talking of the same DNNs explained in Microsofts paper, this is not true. If we were to be talking about different DNNs please elaborate I would love to hear about that (seriously, no irony here).
      - jph00 14y ago
        There's also the random knockout of neurons, as mentioned in the webinar.