7 ms·

What's wrong with convolutional nets? [video]

113 points by drewvolpe 12y ago

robg 12y ago
One of the few speakers I saw as an academic and said to myself "This guy is a time traveler. I need to really pay attention to what he's saying." That was in 2001. A decade+ later and we're still trying to catch up.
- deleted 12y ago
  [deleted]
- TheCoreh 12y ago
  This precisely describes the feeling I got when watching the video. He consistently provides explanations/examples that make complete sense in hindsight but are pretty hard to come up with in the first place. I feel like this video kinda deserves a "spoiler alert" of sorts because he just exposes many of the things and it ends up looking so obvious afterwards. Not obvious in a "yeah the brain certainly works like that" way, because we don't know that yet, but obvious in a "I really wouldn't be surprised if it worked like that" way.
- eli_gottlieb 12y ago
  And this is why I <3 the MIT CoCoSci group et al. Serious nerd-squee levels, here. The whole damn clique of labs have these simple, obvious, clever ideas that are just miles away of everyone else trying to nudge their error rates a few points lower.
deleted 12y ago
[deleted]
jacobn 12y ago
Does anyone have a link to the presentation itself (e.g. ppt/pdf)?
jfoutz 12y ago
Short, massively simplified, summary, Vast improvement in unsupervised learning. Walks through biases humans have in object recognition. Explains new structures to add on top of neural networks. Basic backprop on MNIST with 30,000 labeled examples gives 1.7% error rate. Hinton's approach gets the same results with 25 examples.
- dchichkov 12y ago
  With 25 examples on which the systems asks the labels (and unknown number of unlabeled examples). ... liked the progress on reverse computer graphics approach. A bit of personal perspective here, from the data that I saw (and labeled!), while working on hand gesture recognition project the reverse computer graphics felt like the only approach that could bring human level gesture recognition from 2d data. And I was really exited at some point when Nvidea had started a computer vision / gesture recognition group. It felt really right that a photographic-level computer graphics GPU company would try to do reverse computer graphics processor. Too bad it didn't go anywhere.
TrainedMonkey 12y ago
Video buffers every few seconds for me, are there any mirrors? Edit: found download link after getting some caffeine http://techtv.mit.edu/videos/30698-what-s-wrong-with-convolutional-nets/download.source http://techtv.mit.edu/videos/30698-what-s-wrong-with-convolu... Edit 2: Also, there is a class on Coursera where he shares much knowledge and intuition about neural networks. I personally took it and it was excellent. Unfortunately there are no repeating sessions, but even archived material is a goldmine. https://class.coursera.org/neuralnets-2012-001 https://class.coursera.org/neuralnets-2012-001
- tripzilch 12y ago
  I just pointed youtube-dl at the original URL and it detected and downloaded the embedded .mp4 just fine.
akyu 12y ago
A lot of these sound like ideas that Jeff Hawkins has been pushing with his Cortical Learning Algorithm
- trentonstrong 12y ago
  It's interesting since Hinton's approach seems to be more inspired by the computational geometry aspects of vision (reverse computer graphics, invariant representations) and working backwards towards the neural superstructures. Hawkins' seems to be inspired by the biology itself, working from the "common computational substrate" hypothesis up to the cortical units required. They both seem to meet in the middle at the need to figure out invariant representations for the intermediate features presented to each level of the hierarchy. I do wonder what an argument between those two would look like concerning the applicability of back-propagation, which I remember Hawkins' deriding as totally artificial compared to the feedback structure of the actual neocortex. Anyone more up to date on the state of the argument regarding that?
  - MrQuincle 12y ago
    Backpropagation is just calculating the gradient of a multilayer perceptron. You do not even have to calculate it, you can use autodiff for it: http://users.cecs.anu.edu.au/~jdomke/courses/sml2010/07autodiff_nnets.pdf http://users.cecs.anu.edu.au/~jdomke/courses/sml2010/07autod... (pdf). They both seem to neglect time (internal dynamics). I haven't seen either of them coming up with a model as that from Izhikevich with polychronization: http://www.izhikevich.org/publications/spnet.htm http://www.izhikevich.org/publications/spnet.htm. If we would be able to make one of such ideas computationally useful, things would become really interesting.
    - trentonstrong 12y ago
      Interesting, thanks for the links! I've done your typical ANN 101 training in the past so have a good mental model for back-propagation. Modelling the actual nonlinear dynamics of "realistic" neural networks seems like an obvious path of research but I know how daunting it is. It seems like every tiny bit we can push forward our tools for understanding complex non-linear systems should pay large dividends across so many different computational fields (fluid dynamics, QFT, economics, ..., everything?) I'll have to read Izhikevich's paper, seems like a unique line of research.
discardorama 12y ago
If I'm understanding some of the criticism, it is basically that we don't think "that's how our brain does it" (when it comes to convnets, pooling, etc.). But this begs the question: do we _have_ to duplicate what goes on in the brain? For example: we really don't know how the brain does multiplication; and if I were to guess, I'd say it has nothing to do with binary arithmetic and shift registers. But yet we've been able to teach a computer how to do it, using an algorithm that has (in all probability) no relation to how our brains work. So maybe convnets, pooling and other hacks work, just not how the brain works?

113 points by drewvolpe 12y ago

robg 12y ago

One of the few speakers I saw as an academic and said to myself "This guy is a time traveler. I need to really pay attention to what he's saying." That was in 2001. A decade+ later and we're still trying to catch up.

deleted 12y ago
[deleted]
TheCoreh 12y ago
This precisely describes the feeling I got when watching the video. He consistently provides explanations/examples that make complete sense in hindsight but are pretty hard to come up with in the first place. I feel like this video kinda deserves a "spoiler alert" of sorts because he just exposes many of the things and it ends up looking so obvious afterwards. Not obvious in a "yeah the brain certainly works like that" way, because we don't know that yet, but obvious in a "I really wouldn't be surprised if it worked like that" way.
eli_gottlieb 12y ago
And this is why I <3 the MIT CoCoSci group et al. Serious nerd-squee levels, here. The whole damn clique of labs have these simple, obvious, clever ideas that are just miles away of everyone else trying to nudge their error rates a few points lower.

deleted 12y ago

[deleted]

jacobn 12y ago

Does anyone have a link to the presentation itself (e.g. ppt/pdf)?

jfoutz 12y ago

Short, massively simplified, summary, Vast improvement in unsupervised learning. Walks through biases humans have in object recognition. Explains new structures to add on top of neural networks. Basic backprop on MNIST with 30,000 labeled examples gives 1.7% error rate. Hinton's approach gets the same results with 25 examples.

dchichkov 12y ago
With 25 examples on which the systems asks the labels (and unknown number of unlabeled examples). ... liked the progress on reverse computer graphics approach. A bit of personal perspective here, from the data that I saw (and labeled!), while working on hand gesture recognition project the reverse computer graphics felt like the only approach that could bring human level gesture recognition from 2d data. And I was really exited at some point when Nvidea had started a computer vision / gesture recognition group. It felt really right that a photographic-level computer graphics GPU company would try to do reverse computer graphics processor. Too bad it didn't go anywhere.

TrainedMonkey 12y ago

Video buffers every few seconds for me, are there any mirrors? Edit: found download link after getting some caffeine http://techtv.mit.edu/videos/30698-what-s-wrong-with-convolutional-nets/download.source http://techtv.mit.edu/videos/30698-what-s-wrong-with-convolu... Edit 2: Also, there is a class on Coursera where he shares much knowledge and intuition about neural networks. I personally took it and it was excellent. Unfortunately there are no repeating sessions, but even archived material is a goldmine. https://class.coursera.org/neuralnets-2012-001 https://class.coursera.org/neuralnets-2012-001

tripzilch 12y ago
I just pointed youtube-dl at the original URL and it detected and downloaded the embedded .mp4 just fine.

akyu 12y ago

A lot of these sound like ideas that Jeff Hawkins has been pushing with his Cortical Learning Algorithm

trentonstrong 12y ago
It's interesting since Hinton's approach seems to be more inspired by the computational geometry aspects of vision (reverse computer graphics, invariant representations) and working backwards towards the neural superstructures. Hawkins' seems to be inspired by the biology itself, working from the "common computational substrate" hypothesis up to the cortical units required. They both seem to meet in the middle at the need to figure out invariant representations for the intermediate features presented to each level of the hierarchy. I do wonder what an argument between those two would look like concerning the applicability of back-propagation, which I remember Hawkins' deriding as totally artificial compared to the feedback structure of the actual neocortex. Anyone more up to date on the state of the argument regarding that?
- MrQuincle 12y ago
  Backpropagation is just calculating the gradient of a multilayer perceptron. You do not even have to calculate it, you can use autodiff for it: http://users.cecs.anu.edu.au/~jdomke/courses/sml2010/07autodiff_nnets.pdf http://users.cecs.anu.edu.au/~jdomke/courses/sml2010/07autod... (pdf). They both seem to neglect time (internal dynamics). I haven't seen either of them coming up with a model as that from Izhikevich with polychronization: http://www.izhikevich.org/publications/spnet.htm http://www.izhikevich.org/publications/spnet.htm. If we would be able to make one of such ideas computationally useful, things would become really interesting.
  - trentonstrong 12y ago
    Interesting, thanks for the links! I've done your typical ANN 101 training in the past so have a good mental model for back-propagation. Modelling the actual nonlinear dynamics of "realistic" neural networks seems like an obvious path of research but I know how daunting it is. It seems like every tiny bit we can push forward our tools for understanding complex non-linear systems should pay large dividends across so many different computational fields (fluid dynamics, QFT, economics, ..., everything?) I'll have to read Izhikevich's paper, seems like a unique line of research.

discardorama 12y ago

If I'm understanding some of the criticism, it is basically that we don't think "that's how our brain does it" (when it comes to convnets, pooling, etc.). But this begs the question: do we _have_ to duplicate what goes on in the brain? For example: we really don't know how the brain does multiplication; and if I were to guess, I'd say it has nothing to do with binary arithmetic and shift registers. But yet we've been able to teach a computer how to do it, using an algorithm that has (in all probability) no relation to how our brains work. So maybe convnets, pooling and other hacks work, just not how the brain works?