3 ms·

Probably because the idea is trivial in hindsight (always is) so publishing fast is important. Afaict, the idea is to compute the gradients layer by layer and a

Probably because the idea is trivial in hindsight (always is) so publishing fast is important. Afaict, the idea is to compute the gradients layer by layer and applying them immediately without bothering to back-propagate from the outputs. So in between layers would learn what orientation vectors previous layers emit for positive samples and themselves emit orientation vectors. Imagine a layer learning what regions of a sphere's (3d) surface are good and outputting what regions of a circle's (2d) perimeter are good. This is why he mentions the need for normalizing vectors otherwise layers would cheat and just look at the vector's magnitude. The idea is imo similar to how random word embeddings are generated.

naasking 4y ago
> Afaict, the idea is to compute the gradients layer by layer and applying them immediately without bothering to back-propagate from the outputs. I'm not sure where you get that impression. Forward-Forward [1] seems to eschew gradients entirely: The Forward-Forward algorithm replaces the forward and backward passes of backpropagation by two forward passes, one with positive (i.e. real) data and the other with negative data which could be generated by the network itself [1] https://www.cs.toronto.edu/~hinton/FFA13.pdf https://www.cs.toronto.edu/~hinton/FFA13.pdf
- bjourne 4y ago
  It eschews back-propagation but not gradient calculation. You still have to nudge the activations' weights upward for positive examples and downward for negative ones. Positive examples should give a long vector and negative ones a small vector.
- ekleraki 4y ago
  Let's suppose that you are correct, which direction are the weights updated towards? The implementations of this compute gradients locally.
  - theGnuMe 4y ago
    It makes sense that all gradients are local. Does it make sense to say that gradient propagation through the layers is memoryless?
    - ekleraki 4y ago
      In my opinion, yes if and only if the update does not use a stateful optimiser, and the computation is easy / simple enough that the updated parameter value can be computed immediately. In linear layers, it is possible. Once you have computed the gradient of the output of the vector ith vector, so a scalar, you scale the input by that value and add it to the parameters. This is a simple FMA op: a=fma(eta*z, x, a), with z the gradient of the vector, x the input, a the parameters, and eta the learning rate. This computes a = a + eta*z*x in place.
godelski 4y ago
> because the idea is trivial in hindsight (always is) so publishing fast is important. Unfortunately I've also seen papers get rejected because their idea was "trivial", yet no one had thought of it before. Hinton has an edge here though.

naasking 4y ago

> Afaict, the idea is to compute the gradients layer by layer and applying them immediately without bothering to back-propagate from the outputs. I'm not sure where you get that impression. Forward-Forward [1] seems to eschew gradients entirely: The Forward-Forward algorithm replaces the forward and backward passes of backpropagation by two forward passes, one with positive (i.e. real) data and the other with negative data which could be generated by the network itself [1] https://www.cs.toronto.edu/~hinton/FFA13.pdf https://www.cs.toronto.edu/~hinton/FFA13.pdf

bjourne 4y ago
It eschews back-propagation but not gradient calculation. You still have to nudge the activations' weights upward for positive examples and downward for negative ones. Positive examples should give a long vector and negative ones a small vector.
ekleraki 4y ago
Let's suppose that you are correct, which direction are the weights updated towards? The implementations of this compute gradients locally.
- theGnuMe 4y ago
  It makes sense that all gradients are local. Does it make sense to say that gradient propagation through the layers is memoryless?
  - ekleraki 4y ago
    In my opinion, yes if and only if the update does not use a stateful optimiser, and the computation is easy / simple enough that the updated parameter value can be computed immediately. In linear layers, it is possible. Once you have computed the gradient of the output of the vector ith vector, so a scalar, you scale the input by that value and add it to the parameters. This is a simple FMA op: a=fma(eta*z, x, a), with z the gradient of the vector, x the input, a the parameters, and eta the learning rate. This computes a = a + eta*z*x in place.

godelski 4y ago

> because the idea is trivial in hindsight (always is) so publishing fast is important. Unfortunately I've also seen papers get rejected because their idea was "trivial", yet no one had thought of it before. Hinton has an edge here though.