5 ms·

For those looking to learn about these techniques, I'd highly recommend the deep learning theano tutorials. Hinton has a class on Coursera--I think it would be

For those looking to learn about these techniques, I'd highly recommend the deep learning theano tutorials. Hinton has a class on Coursera--I think it would be very confusing for beginners, but it has really great material. Also, I run the "SF Neural Network Aficionados" meetup in san francisco and will be giving a workshop in January about building your own DBN in python, so feel free to check that out if you're in SF (although space was an issue last time).

stewie2 14y ago
How is "deep learning" different from "neural network"?
- dave_sullivan 14y ago
  Well--two answers: 1) It's not. It's just a buzz word that people are going to use to separate the current (2006+) research from older research concerning neural networks. This is to draw a clear (and potentially self serving?) distinction between old neural networks that were discredited due to their lack of results vs current research that produces much much better results. So, it's neural networks rebranded. Oh my. 2) "traditional" neural networks and what people are using now are very different--mostly because what people are doing now actually works. Deep learning refers to deep neural networks, which take more traditional neural networks and stack them on top of each other to form a hierarchy of representations that ends up being effective for all kinds of stuff. Not that deep neural networks are a new concept--the newness is more that this is now practical rather than theoretical. So, deep learning is like 20% bullshit, 80% the real deal. Still lots of work to be done, but I think "deep learning" is a nice buzzword to describe the current state of the art as far as neural networks go. It's all neural networks--but this time it's different, haha.
- Yoshua 14y ago
  The idea of having multiple levels of representation (deep learning) goes beyond neural networks. A good example is the recent work (award-winning at NIPS 2012) on sum-product networks, which are graphical models whose partition function is tractable by construction. Several important things have been added since 2006 (when deep learning was deemed to begin) to the previous wave of neural networks research, in particular powerful unsupervised learning algorithms (which allow very successful semi-supervised and transfer learning - 2 competitions won in 2011), often incorporating advanced probabilistic models with latent variables, a better understanding (although much more remains to be done) of the optimization difficulty of training gradient-based systems through many composed non-linearities, and other improvements to regularize better (such as the recent dropouts) and to rationally and efficiently select hyper-parameters (random sampling and Bayesian optimization). It is also true that sheer improvements in computing power and amounts of training data are in part responsible for the impressively good results recently obtained in speech recognition (see recent New York Times article, 24 nov, J. Markoff) and object recognition (see NIPS 2012 paper by Krizhesky et al).
wwwtyro 14y ago
DBN?
- shmageggy 14y ago
  Deep Belief Network http://www.scholarpedia.org/article/Deep_belief_networks http://www.scholarpedia.org/article/Deep_belief_networks
  - Sarien 14y ago
    Very confusing abbreviation since it is also used for Dynamic Bayesian Networks. Somebody should come up with something else and quickly before it sticks. :)
- deleted 14y ago
  [deleted]
plg 14y ago
Pls put notes & code online

stewie2 14y ago

How is "deep learning" different from "neural network"?

dave_sullivan 14y ago
Well--two answers: 1) It's not. It's just a buzz word that people are going to use to separate the current (2006+) research from older research concerning neural networks. This is to draw a clear (and potentially self serving?) distinction between old neural networks that were discredited due to their lack of results vs current research that produces much much better results. So, it's neural networks rebranded. Oh my. 2) "traditional" neural networks and what people are using now are very different--mostly because what people are doing now actually works. Deep learning refers to deep neural networks, which take more traditional neural networks and stack them on top of each other to form a hierarchy of representations that ends up being effective for all kinds of stuff. Not that deep neural networks are a new concept--the newness is more that this is now practical rather than theoretical. So, deep learning is like 20% bullshit, 80% the real deal. Still lots of work to be done, but I think "deep learning" is a nice buzzword to describe the current state of the art as far as neural networks go. It's all neural networks--but this time it's different, haha.
Yoshua 14y ago
The idea of having multiple levels of representation (deep learning) goes beyond neural networks. A good example is the recent work (award-winning at NIPS 2012) on sum-product networks, which are graphical models whose partition function is tractable by construction. Several important things have been added since 2006 (when deep learning was deemed to begin) to the previous wave of neural networks research, in particular powerful unsupervised learning algorithms (which allow very successful semi-supervised and transfer learning - 2 competitions won in 2011), often incorporating advanced probabilistic models with latent variables, a better understanding (although much more remains to be done) of the optimization difficulty of training gradient-based systems through many composed non-linearities, and other improvements to regularize better (such as the recent dropouts) and to rationally and efficiently select hyper-parameters (random sampling and Bayesian optimization). It is also true that sheer improvements in computing power and amounts of training data are in part responsible for the impressively good results recently obtained in speech recognition (see recent New York Times article, 24 nov, J. Markoff) and object recognition (see NIPS 2012 paper by Krizhesky et al).

wwwtyro 14y ago

DBN?

shmageggy 14y ago
Deep Belief Network http://www.scholarpedia.org/article/Deep_belief_networks http://www.scholarpedia.org/article/Deep_belief_networks
- Sarien 14y ago
  Very confusing abbreviation since it is also used for Dynamic Bayesian Networks. Somebody should come up with something else and quickly before it sticks. :)
deleted 14y ago
[deleted]

plg 14y ago

Pls put notes & code online