4 ms·

NN are being talked about because they have been excessively effective at advancing the state of the art in areas that were previously in a dead end. Computer

NN are being talked about because they have been excessively effective at advancing the state of the art in areas that were previously in a dead end. Computer vision is such a field. The results provided by conventional approaches had been plateauing for years and then one day, Yann Le Cunn submitted a paper describing the work that he and his team had done in the area with neural networks and which blew away the previous results on image recognition. Interestingly, this paper was rejected and it took a full year to the CV community to finally turn around and accept NN as a valid approach to this problem. I believe no one questions that it's the best methodology we have today. Speech recognition, automatic translation and natural language processing in general are other areas that have benefited immensely from neural networks.

YeGoblynQueenne 10y ago
>> Speech recognition, automatic translation and natural language processing in general are other areas that have benefited immensely from neural networks. In NLP in particular, I'm not so sure about the "immense" benefit of ANNs. LSTM-RNNs are very nice for language modelling but we haven't seen the 10-percentile jump in performance that the image processing community has. Stanford I believe uses ANNs in their dependency parser (to learn transitions) and Google of course uses ANNs everywhere, but the gains against goold old HMMs are nowhere near the gains in image processing and it's not clear what exactly is the difference. Maybe teams have better hardware and more data now. Rather, they definitely do, so maybe that's the reason for any increases in performance. With translation in particular (it's a branch of NLP) it's very hard to know what's going on. Companies who sell machine translation products still use different techniques, including rule-based software. The problem is that machine translation needs some way to deal with semantics, else it's a bit shit. Google again- they do use ANNs (apparently- it's hard to tell because they don't) but is their translation any better than others'? That's almost impossible to know, because there are no good measures of machine translation performance. I'll repeat that: we don't have any good measures of machine translation performance. So anyone can come up with a solution and drum it up as "state-of-the-art", and if their user base is large enough it'll be impossible for anyone (even themselves) to know for sure whether that means anything. Speech recognition- I don't know much about that, but I think it's similar to NLP. You can do a lot with a lot of data and hardware, if you're lucky to have them and at that point it doesn't really make a real difference to your end-user if you're using an ANN model or an HMM model.
- praccu 10y ago
  DNNs have lead to massive improvements in speech, specifically in the acoustic model. RNN based language models interpolated with modified kn smoothed ngrams (often plus some other cool stuff) are also the state of the art, although they're often too slow to use in practice.
  - YeGoblynQueenne 10y ago
    Thank you - I stand corrected, although I think the "too slow to use in practice" bit is possibly what I had in mind. Which may just go to show that Hinton, Schmidhuber and the rest of the connectionist gang are right about the value of more data and faster computers.
sukilot 10y ago
What justification does a journal have for rejecting an objectively groundbreaking result, just for using an unfashionable technology?
- kmiroslav 10y ago
  Here is Yann's summary of the situation back then: https://plus.google.com/+YannLeCunPhD/posts/gurGyczzsJ7 https://plus.google.com/+YannLeCunPhD/posts/gurGyczzsJ7 I seem to remember this article as having a lot of very interesting comments, including from the chairman of the conference. The exchange was cordial but showed some clear antagonism between the two parties. Somehow, I don't see these comments any more on Google+. That was four years ago, things are a lot better now.
  - rspeer 10y ago
    Things are better for neural nets because they've become fashionable. But nothing changed to make the peer review process tolerate unfashionable technologies.
jameshart 10y ago
So this is what confuses me about what is 'new' with respect to deep learning. Neural nets are not new - I was aware of the existence of, and some of the basic ideas behind, neural nets as a technology in the 1990s and I wasn't even involved in computer science, so I assume that means that even then they were a mainstream AI technique. When I read content about 'deep learning' neural nets today I don't see anything especially different to what my (admittedly shallow) understanding of neural nets was back then. So what I'm missing mainly is - what changed? Is it just that advances in compute power mean that problems for which neural nets were impractical have now become practical? Is there something different about the way neural nets are employed in 'deep learning' that is different than the neural nets that were discussed in the past?
- Eliezer 10y ago
  Previously, neural networks were trained by taking single steps down the direction of sharpest gradient for the network. However, in deep networks with lots of layers, the backprop algorithm (which I assume you already know about) got stuck in local minima. Deep learning got started when Hinton observed that a certain way of training restricted Boltzmann machines wouldn't get stuck as easily, and hence by pretraining the network as if it were an RBM and then switching to backprop, it wouldn't get stuck in a local minimum as early. As I understand it, nowadays the best method looks something like a generalization of Newton's Method, wherein the direction you move takes into account the second differential and not just the first differential or direction of sharpest descent. You move furthest in the directions that curve the least, and move the least in the directions that are most sharply curved. It turns out that this (plus some other tricks) make it way easier to follow continuous gradients in big weird parameter spaces, so now it's possible to train deep nets, which are a kind of continuous gradient in a big weird parameter space. Tl/dr: People figured out how to move better through the parameter space of neural networks, by taking into account the second differential plus some other tricks. So now we can train deeper nets.
  - jameshart 10y ago
    Thank you! That makes a lot of sense.

YeGoblynQueenne 10y ago

>> Speech recognition, automatic translation and natural language processing in general are other areas that have benefited immensely from neural networks. In NLP in particular, I'm not so sure about the "immense" benefit of ANNs. LSTM-RNNs are very nice for language modelling but we haven't seen the 10-percentile jump in performance that the image processing community has. Stanford I believe uses ANNs in their dependency parser (to learn transitions) and Google of course uses ANNs everywhere, but the gains against goold old HMMs are nowhere near the gains in image processing and it's not clear what exactly is the difference. Maybe teams have better hardware and more data now. Rather, they definitely do, so maybe that's the reason for any increases in performance. With translation in particular (it's a branch of NLP) it's very hard to know what's going on. Companies who sell machine translation products still use different techniques, including rule-based software. The problem is that machine translation needs some way to deal with semantics, else it's a bit shit. Google again- they do use ANNs (apparently- it's hard to tell because they don't) but is their translation any better than others'? That's almost impossible to know, because there are no good measures of machine translation performance. I'll repeat that: we don't have any good measures of machine translation performance. So anyone can come up with a solution and drum it up as "state-of-the-art", and if their user base is large enough it'll be impossible for anyone (even themselves) to know for sure whether that means anything. Speech recognition- I don't know much about that, but I think it's similar to NLP. You can do a lot with a lot of data and hardware, if you're lucky to have them and at that point it doesn't really make a real difference to your end-user if you're using an ANN model or an HMM model.

praccu 10y ago
DNNs have lead to massive improvements in speech, specifically in the acoustic model. RNN based language models interpolated with modified kn smoothed ngrams (often plus some other cool stuff) are also the state of the art, although they're often too slow to use in practice.
- YeGoblynQueenne 10y ago
  Thank you - I stand corrected, although I think the "too slow to use in practice" bit is possibly what I had in mind. Which may just go to show that Hinton, Schmidhuber and the rest of the connectionist gang are right about the value of more data and faster computers.

sukilot 10y ago

What justification does a journal have for rejecting an objectively groundbreaking result, just for using an unfashionable technology?

kmiroslav 10y ago
Here is Yann's summary of the situation back then: https://plus.google.com/+YannLeCunPhD/posts/gurGyczzsJ7 https://plus.google.com/+YannLeCunPhD/posts/gurGyczzsJ7 I seem to remember this article as having a lot of very interesting comments, including from the chairman of the conference. The exchange was cordial but showed some clear antagonism between the two parties. Somehow, I don't see these comments any more on Google+. That was four years ago, things are a lot better now.
- rspeer 10y ago
  Things are better for neural nets because they've become fashionable. But nothing changed to make the peer review process tolerate unfashionable technologies.

jameshart 10y ago

So this is what confuses me about what is 'new' with respect to deep learning. Neural nets are not new - I was aware of the existence of, and some of the basic ideas behind, neural nets as a technology in the 1990s and I wasn't even involved in computer science, so I assume that means that even then they were a mainstream AI technique. When I read content about 'deep learning' neural nets today I don't see anything especially different to what my (admittedly shallow) understanding of neural nets was back then. So what I'm missing mainly is - what changed? Is it just that advances in compute power mean that problems for which neural nets were impractical have now become practical? Is there something different about the way neural nets are employed in 'deep learning' that is different than the neural nets that were discussed in the past?

Eliezer 10y ago
Previously, neural networks were trained by taking single steps down the direction of sharpest gradient for the network. However, in deep networks with lots of layers, the backprop algorithm (which I assume you already know about) got stuck in local minima. Deep learning got started when Hinton observed that a certain way of training restricted Boltzmann machines wouldn't get stuck as easily, and hence by pretraining the network as if it were an RBM and then switching to backprop, it wouldn't get stuck in a local minimum as early. As I understand it, nowadays the best method looks something like a generalization of Newton's Method, wherein the direction you move takes into account the second differential and not just the first differential or direction of sharpest descent. You move furthest in the directions that curve the least, and move the least in the directions that are most sharply curved. It turns out that this (plus some other tricks) make it way easier to follow continuous gradients in big weird parameter spaces, so now it's possible to train deep nets, which are a kind of continuous gradient in a big weird parameter space. Tl/dr: People figured out how to move better through the parameter space of neural networks, by taking into account the second differential plus some other tricks. So now we can train deeper nets.
- jameshart 10y ago
  Thank you! That makes a lot of sense.