4 ms·

I think you have to set expectations based on how much of the ground you're ripping up. If you're adding some layers or some little tweak to an existing archite

I think you have to set expectations based on how much of the ground you're ripping up. If you're adding some layers or some little tweak to an existing architecture, then yeah, going backwards on cifar-10 is a failure. If, however, you are ripping out backpropagation like this paper is, then you get a big pass. This is not the new paradigm yet, but it's promising that it doesn't just completely fail.

LudwigNagasena 4y ago
There is no shortage of paradigms that rip out backprop and deliver worse results.
- levesque 4y ago
  Also Hinton doesn't have the best track record with his already forgotten/abandoned Capsule networks. I wonder what's the next thing he's going to come up with? He gets a pass because he is famous.
  - whimsicalism 4y ago
    I think it can be simultaneously true that things like this should be tested with toy models we wouldn't expect to do great on CIFAR and also that we shouldn't expect exceptional results just because this person is already famous.
- tehsauce 4y ago
  This is so true! But we should keep trying :)
jxcole 4y ago
This seems to be Hinton's MO though. A few years back he ripped out convolutions for capsules and while he claims it's better and some people might claim it "has potential", no one really uses it for much because, as with this, the actual numerical performance is worse on the tests people care about (e.g. imagenet accuracy). https://en.wikipedia.org/wiki/Capsule_neural_network https://en.wikipedia.org/wiki/Capsule_neural_network
- habitue 4y ago
  I mean yes, this should be the MO of a tenured professor, making large speculative bets, not hyper optimizing benchmarks
  - hgsgm 4y ago
    But some of those bets should be right, or else he'd be better spending his time and accumulated knowledge writing a historical monograph.
    - deanCommie 4y ago
      The backpropagation paper was published in 1986. It took >20 years for it to be right. Maybe we ought to give this one some time?
      - kwooding 4y ago
        Not sure what you mean by >20 years to be right. I built and trained a 3-layer back-propagating neural net to do OCR on an Apple 2 in 1989 based on that paper. Admittedly, just the 26 upper case characters. But it clearly worked better than the alternatives.
    - habitue 4y ago
      Specifically, tenure is to remove the pressure that "you'd better be right" so professors are free to take meandering tangents through the solution space that don't seem like they'll pay off immediately. The failure mode of tenure is that the professor just rests on their past accomplishments and doesn't do anything. That's a risk the system takes. In this case though, Geoff Hinton is doing everything right: he's not only not sitting around doing nothing, he's actively trying to obsolete the paradigm he helped usher in, just in case there is a better option out there. I think that's admirable

LudwigNagasena 4y ago

There is no shortage of paradigms that rip out backprop and deliver worse results.

levesque 4y ago
Also Hinton doesn't have the best track record with his already forgotten/abandoned Capsule networks. I wonder what's the next thing he's going to come up with? He gets a pass because he is famous.
- whimsicalism 4y ago
  I think it can be simultaneously true that things like this should be tested with toy models we wouldn't expect to do great on CIFAR and also that we shouldn't expect exceptional results just because this person is already famous.
tehsauce 4y ago
This is so true! But we should keep trying :)

jxcole 4y ago

This seems to be Hinton's MO though. A few years back he ripped out convolutions for capsules and while he claims it's better and some people might claim it "has potential", no one really uses it for much because, as with this, the actual numerical performance is worse on the tests people care about (e.g. imagenet accuracy). https://en.wikipedia.org/wiki/Capsule_neural_network https://en.wikipedia.org/wiki/Capsule_neural_network

habitue 4y ago
I mean yes, this should be the MO of a tenured professor, making large speculative bets, not hyper optimizing benchmarks
- hgsgm 4y ago
  But some of those bets should be right, or else he'd be better spending his time and accumulated knowledge writing a historical monograph.
  - deanCommie 4y ago
    The backpropagation paper was published in 1986. It took >20 years for it to be right. Maybe we ought to give this one some time?
    - kwooding 4y ago
      Not sure what you mean by >20 years to be right. I built and trained a 3-layer back-propagating neural net to do OCR on an Apple 2 in 1989 based on that paper. Admittedly, just the 26 upper case characters. But it clearly worked better than the alternatives.
  - habitue 4y ago
    Specifically, tenure is to remove the pressure that "you'd better be right" so professors are free to take meandering tangents through the solution space that don't seem like they'll pay off immediately. The failure mode of tenure is that the professor just rests on their past accomplishments and doesn't do anything. That's a risk the system takes. In this case though, Geoff Hinton is doing everything right: he's not only not sitting around doing nothing, he's actively trying to obsolete the paradigm he helped usher in, just in case there is a better option out there. I think that's admirable