6 ms·

I skimmed through the paper and am a bit confused. There's only one equation and I feel like he rushed to publish a shower thought without even bothering to fle

I skimmed through the paper and am a bit confused. There's only one equation and I feel like he rushed to publish a shower thought without even bothering to flesh it out mathematically. So how do you optimize a layer? Do you still use gradient descent? So you are have a per layer loss with a positive and negative component and then do gradient descent? So then what is the label for each layer? Do you use the same label for each layer? And what does he mean by the forward pass not being fully known? I don't get this application of the blackbox between layers. Why would you want to do that?

harveywi 4y ago
Those details have to be omitted from manuscripts in order to avoid having to cite the works of Jürgen Schmidhuber.
- bitL 4y ago
  Jürgen did it all before in the 80s, however it was never translated to English so Geoffrey could happily reinvent it.
  - godelski 4y ago
    Jürgen invented AGI in the early 90s but someone pressed the red button on his website and it committed suicide.
- manytree5 4y ago
  Just curious, why would one want to avoid citing Schmidhuber's work?
  - sva_ 4y ago
    It is a bit of a meme in AI research, as Schmidhuber often claims that he hasn't received the citations that he thinks he deserves. https://www.urbandictionary.com/define.php?term=schmidhuber https://www.urbandictionary.com/define.php?term=schmidhuber
    - reil_convnet 4y ago
      Its not just a meme in this case btw :D https://twitter.com/SchmidhuberAI/status/1605246688939364352 https://twitter.com/SchmidhuberAI/status/1605246688939364352
civilized 4y ago
> There's only one equation Not accurate for the version another commenter linked: https://www.cs.toronto.edu/~hinton/FFA13.pdf https://www.cs.toronto.edu/~hinton/FFA13.pdf I see four equations.
bjourne 4y ago
Probably because the idea is trivial in hindsight (always is) so publishing fast is important. Afaict, the idea is to compute the gradients layer by layer and applying them immediately without bothering to back-propagate from the outputs. So in between layers would learn what orientation vectors previous layers emit for positive samples and themselves emit orientation vectors. Imagine a layer learning what regions of a sphere's (3d) surface are good and outputting what regions of a circle's (2d) perimeter are good. This is why he mentions the need for normalizing vectors otherwise layers would cheat and just look at the vector's magnitude. The idea is imo similar to how random word embeddings are generated.
- naasking 4y ago
  > Afaict, the idea is to compute the gradients layer by layer and applying them immediately without bothering to back-propagate from the outputs. I'm not sure where you get that impression. Forward-Forward [1] seems to eschew gradients entirely: The Forward-Forward algorithm replaces the forward and backward passes of backpropagation by two forward passes, one with positive (i.e. real) data and the other with negative data which could be generated by the network itself [1] https://www.cs.toronto.edu/~hinton/FFA13.pdf https://www.cs.toronto.edu/~hinton/FFA13.pdf
  - bjourne 4y ago
    It eschews back-propagation but not gradient calculation. You still have to nudge the activations' weights upward for positive examples and downward for negative ones. Positive examples should give a long vector and negative ones a small vector.
  - ekleraki 4y ago
    Let's suppose that you are correct, which direction are the weights updated towards? The implementations of this compute gradients locally.
    - theGnuMe 4y ago
      It makes sense that all gradients are local. Does it make sense to say that gradient propagation through the layers is memoryless?
duvenaud 4y ago
Perhaps a better place to find algorithmic details is this related paper, also with Hinton as a co-author, which implements similar ideas in more standard networks: Scaling Forward Gradient With Local Losses Mengye Ren, Simon Kornblith, Renjie Liao, Geoffrey Hinton https://arxiv.org/abs/2210.03310 https://arxiv.org/abs/2210.03310 and has code: https://github.com/google-research/google-research/tree/master/local_forward_gradient https://github.com/google-research/google-research/tree/mast...

harveywi 4y ago

Those details have to be omitted from manuscripts in order to avoid having to cite the works of Jürgen Schmidhuber.

bitL 4y ago
Jürgen did it all before in the 80s, however it was never translated to English so Geoffrey could happily reinvent it.
- godelski 4y ago
  Jürgen invented AGI in the early 90s but someone pressed the red button on his website and it committed suicide.
manytree5 4y ago
Just curious, why would one want to avoid citing Schmidhuber's work?
- sva_ 4y ago
  It is a bit of a meme in AI research, as Schmidhuber often claims that he hasn't received the citations that he thinks he deserves. https://www.urbandictionary.com/define.php?term=schmidhuber https://www.urbandictionary.com/define.php?term=schmidhuber
  - reil_convnet 4y ago
    Its not just a meme in this case btw :D https://twitter.com/SchmidhuberAI/status/1605246688939364352 https://twitter.com/SchmidhuberAI/status/1605246688939364352

civilized 4y ago

> There's only one equation Not accurate for the version another commenter linked: https://www.cs.toronto.edu/~hinton/FFA13.pdf https://www.cs.toronto.edu/~hinton/FFA13.pdf I see four equations.

bjourne 4y ago

Probably because the idea is trivial in hindsight (always is) so publishing fast is important. Afaict, the idea is to compute the gradients layer by layer and applying them immediately without bothering to back-propagate from the outputs. So in between layers would learn what orientation vectors previous layers emit for positive samples and themselves emit orientation vectors. Imagine a layer learning what regions of a sphere's (3d) surface are good and outputting what regions of a circle's (2d) perimeter are good. This is why he mentions the need for normalizing vectors otherwise layers would cheat and just look at the vector's magnitude. The idea is imo similar to how random word embeddings are generated.

naasking 4y ago
> Afaict, the idea is to compute the gradients layer by layer and applying them immediately without bothering to back-propagate from the outputs. I'm not sure where you get that impression. Forward-Forward [1] seems to eschew gradients entirely: The Forward-Forward algorithm replaces the forward and backward passes of backpropagation by two forward passes, one with positive (i.e. real) data and the other with negative data which could be generated by the network itself [1] https://www.cs.toronto.edu/~hinton/FFA13.pdf https://www.cs.toronto.edu/~hinton/FFA13.pdf
- bjourne 4y ago
  It eschews back-propagation but not gradient calculation. You still have to nudge the activations' weights upward for positive examples and downward for negative ones. Positive examples should give a long vector and negative ones a small vector.
- ekleraki 4y ago
  Let's suppose that you are correct, which direction are the weights updated towards? The implementations of this compute gradients locally.
  - theGnuMe 4y ago
    It makes sense that all gradients are local. Does it make sense to say that gradient propagation through the layers is memoryless?

duvenaud 4y ago

Perhaps a better place to find algorithmic details is this related paper, also with Hinton as a co-author, which implements similar ideas in more standard networks: Scaling Forward Gradient With Local Losses Mengye Ren, Simon Kornblith, Renjie Liao, Geoffrey Hinton https://arxiv.org/abs/2210.03310 https://arxiv.org/abs/2210.03310 and has code: https://github.com/google-research/google-research/tree/master/local_forward_gradient https://github.com/google-research/google-research/tree/mast...