4 ms·

Maybe I'm missing something, but from the paper https://www.cs.toronto.edu/~hinton/FFA13.pdf https://www.cs.toronto.edu/~hinton/FFA13.pdf, they use non-conv net

Maybe I'm missing something, but from the paper https://www.cs.toronto.edu/~hinton/FFA13.pdf https://www.cs.toronto.edu/~hinton/FFA13.pdf, they use non-conv nets on CIFAR-10 for back prop, resulting in 63% accuracy. And FF achieves 59% accuracy (at best). Those are relatively close figures, but good accuracy on CIFAR-10 is 99%+ and getting ~94% is trivial. So, if an improper architecture for a problem is used and the accuracy is poor, how compelling is using another optimization approach and achieving similar accuracy? It's a unique and interesting approach, but the article specifically mentions it gets accuracy similar to backprop, but if this is the experiment that claim is based on, it loses some credibility in my eyes.

whimsicalism 4y ago
You have to start with toy models before scaling up.
- electrograv 4y ago
  Achieving <80% on CIFAR10 in the year >2020 is an example of a failed toy model, not a successful toy model. Almost any ML algorithm can be thrown at CIFAR10 and achieve ~60% accuracy; this ballpark of accuracy is really not sufficient to demonstrate viability, no matter how aesthetically interesting the approach might feel.
  - deleted 4y ago
    [deleted]
  - alper111 4y ago
    Plain MLP acc. is 63% vs 59% with FF, not so bad? By the same logic, MLP is a failed toy model.
  - jasonjmcghee 4y ago
    I haven't seen this to be the case, fwiw. There was a paper in 2016 that did this and most were in the ~40% range. But "any ml algorithm" isn't the point. It's a new optimization technique and should be applied to models/architectures that make sense with the problems they are being used on. For example, they could have used a pretrained featurizer and trained the two layer model on top of it, with both back prop and FF and compared.
    - whimsicalism 4y ago
      > For example, they could have used a pretrained featurizer and trained the two layer model on top of it, with both back prop and FF and compared. Making the assumption that weights/embeddings produced by a backprop-trained network are equally intelligible to a network also trained by backprop vs. one trained by this alternative method.
      - jasonjmcghee 4y ago
        I have personally seen them used successfully with all kinds of classic ml algorithms (enets, tree-based, etc) that have nothing to do with back prop.
  - whimsicalism 4y ago
    Any ML algorithm that already has tooling written, CUDA scripts, etc. to run it faster. That said, I am also short-term bearish on backprop-free methods (although potentially long-term bullish).
  - hiddencost 4y ago
    Hinton is doing basic science, not ML, here. Given who he is, trying to move the needle on traditional benchmarks would be a waste of his time and skills. If he invents the new back propagation, an army of grad students can turn his ideas into the future. Like they've done for the last 15 years. He's posting incremental work towards rethinking the field. It's pretty interesting stuff. Edit: grammar
habitue 4y ago
I think you have to set expectations based on how much of the ground you're ripping up. If you're adding some layers or some little tweak to an existing architecture, then yeah, going backwards on cifar-10 is a failure. If, however, you are ripping out backpropagation like this paper is, then you get a big pass. This is not the new paradigm yet, but it's promising that it doesn't just completely fail.
- LudwigNagasena 4y ago
  There is no shortage of paradigms that rip out backprop and deliver worse results.
  - levesque 4y ago
    Also Hinton doesn't have the best track record with his already forgotten/abandoned Capsule networks. I wonder what's the next thing he's going to come up with? He gets a pass because he is famous.
    - whimsicalism 4y ago
      I think it can be simultaneously true that things like this should be tested with toy models we wouldn't expect to do great on CIFAR and also that we shouldn't expect exceptional results just because this person is already famous.
  - tehsauce 4y ago
    This is so true! But we should keep trying :)
- jxcole 4y ago
  This seems to be Hinton's MO though. A few years back he ripped out convolutions for capsules and while he claims it's better and some people might claim it "has potential", no one really uses it for much because, as with this, the actual numerical performance is worse on the tests people care about (e.g. imagenet accuracy). https://en.wikipedia.org/wiki/Capsule_neural_network https://en.wikipedia.org/wiki/Capsule_neural_network
Nevermark 4y ago
The best context to view the paper, is as part of an algorithm search. Until the brain's algorithm is "solved", half steps are important. We need as many alternate half steps as we can find until one or more lead to a better understanding of the brain. (And potentially, better than backdrop efficiency or results.)
yobbo 4y ago
This is not a benchmark of some model on cifar-10, it's a benchmark of the training algorithm. But, model size and complexity also matters. MLP with backprop gets about 63% on cifar-10, for various reasons. So achieving 59% accuracy means this algorithm is about 93% as good as backprop in this case. However, 63% accuracy on cifar-10 can be achieved with two (maybe three) layers IIRC. The output is a 10-way classifier, which is handled in one layer. If the output requires multi-layer transformations, then gradients need to be back-propagated. As long as the batch activation vectors are trained to max separation (or orthogonality or whatever) at each layer, one output layer can match them to labels. But this is unlikely in problems where the output is more "transformed or complicated".

whimsicalism 4y ago

You have to start with toy models before scaling up.

electrograv 4y ago
Achieving <80% on CIFAR10 in the year >2020 is an example of a failed toy model, not a successful toy model. Almost any ML algorithm can be thrown at CIFAR10 and achieve ~60% accuracy; this ballpark of accuracy is really not sufficient to demonstrate viability, no matter how aesthetically interesting the approach might feel.
- deleted 4y ago
  [deleted]
- alper111 4y ago
  Plain MLP acc. is 63% vs 59% with FF, not so bad? By the same logic, MLP is a failed toy model.
- jasonjmcghee 4y ago
  I haven't seen this to be the case, fwiw. There was a paper in 2016 that did this and most were in the ~40% range. But "any ml algorithm" isn't the point. It's a new optimization technique and should be applied to models/architectures that make sense with the problems they are being used on. For example, they could have used a pretrained featurizer and trained the two layer model on top of it, with both back prop and FF and compared.
  - whimsicalism 4y ago
    > For example, they could have used a pretrained featurizer and trained the two layer model on top of it, with both back prop and FF and compared. Making the assumption that weights/embeddings produced by a backprop-trained network are equally intelligible to a network also trained by backprop vs. one trained by this alternative method.
    - jasonjmcghee 4y ago
      I have personally seen them used successfully with all kinds of classic ml algorithms (enets, tree-based, etc) that have nothing to do with back prop.
- whimsicalism 4y ago
  Any ML algorithm that already has tooling written, CUDA scripts, etc. to run it faster. That said, I am also short-term bearish on backprop-free methods (although potentially long-term bullish).
- hiddencost 4y ago
  Hinton is doing basic science, not ML, here. Given who he is, trying to move the needle on traditional benchmarks would be a waste of his time and skills. If he invents the new back propagation, an army of grad students can turn his ideas into the future. Like they've done for the last 15 years. He's posting incremental work towards rethinking the field. It's pretty interesting stuff. Edit: grammar

habitue 4y ago

I think you have to set expectations based on how much of the ground you're ripping up. If you're adding some layers or some little tweak to an existing architecture, then yeah, going backwards on cifar-10 is a failure. If, however, you are ripping out backpropagation like this paper is, then you get a big pass. This is not the new paradigm yet, but it's promising that it doesn't just completely fail.

LudwigNagasena 4y ago
There is no shortage of paradigms that rip out backprop and deliver worse results.
- levesque 4y ago
  Also Hinton doesn't have the best track record with his already forgotten/abandoned Capsule networks. I wonder what's the next thing he's going to come up with? He gets a pass because he is famous.
  - whimsicalism 4y ago
    I think it can be simultaneously true that things like this should be tested with toy models we wouldn't expect to do great on CIFAR and also that we shouldn't expect exceptional results just because this person is already famous.
- tehsauce 4y ago
  This is so true! But we should keep trying :)
jxcole 4y ago
This seems to be Hinton's MO though. A few years back he ripped out convolutions for capsules and while he claims it's better and some people might claim it "has potential", no one really uses it for much because, as with this, the actual numerical performance is worse on the tests people care about (e.g. imagenet accuracy). https://en.wikipedia.org/wiki/Capsule_neural_network https://en.wikipedia.org/wiki/Capsule_neural_network

Nevermark 4y ago

The best context to view the paper, is as part of an algorithm search. Until the brain's algorithm is "solved", half steps are important. We need as many alternate half steps as we can find until one or more lead to a better understanding of the brain. (And potentially, better than backdrop efficiency or results.)

yobbo 4y ago

This is not a benchmark of some model on cifar-10, it's a benchmark of the training algorithm. But, model size and complexity also matters. MLP with backprop gets about 63% on cifar-10, for various reasons. So achieving 59% accuracy means this algorithm is about 93% as good as backprop in this case. However, 63% accuracy on cifar-10 can be achieved with two (maybe three) layers IIRC. The output is a 10-way classifier, which is handled in one layer. If the output requires multi-layer transformations, then gradients need to be back-propagated. As long as the batch activation vectors are trained to max separation (or orthogonality or whatever) at each layer, one output layer can match them to labels. But this is unlikely in problems where the output is more "transformed or complicated".