4 ms·

I don't really understand how it would reduce processing time, could you elaborate? The main implications seem to be for neuroscience, as far as I can tell. Ba

by benanne 12y ago

I don't really understand how it would reduce processing time, could you elaborate? The main implications seem to be for neuroscience, as far as I can tell. Backprop is considered biologically implausible because it requires either bidirectional communication over synapses (which doesn't happen) or weight sharing between neurons. But this allows the forward and backward connections to be decoupled (i.e. they are different synapses). This is really interesting stuff, my first reaction was "why does this even work?" I think I still don't really fully understand what's going on.

bearzoo 12y ago
From reading the abstract it seems that they are claiming that introducing some randomness into your gradient of weight changes allows for the quicker convergence of solution - I did not read the paper. I also don't exactly understand why it works - it sounds like they are claiming traditional back prop has room for improvement.
- Houshalter 12y ago
  That's a very old strategy called jittering (also see stochastic gradient descent.) This is something entirely different. They are not doing regular backpropagation at all, but somehow using neurons to learn how to backpropagate values. I haven't read the paper yet, just read their slides earlier, so that might not be correct.
Jonanin 12y ago
> Backprop is considered biologically implausible This is not true. See Neural Back propagation [1]. There are known mechanisms for backwards feedback between neural connections, for example Spike Timing Dependent Plasticity - where neural inputs that are well correlated in time and potential to output firings are strengthened over time. These phenomena are vital to learning and neural development. [1] http://en.m.wikipedia.org/wiki/Neural_backpropagation http://en.m.wikipedia.org/wiki/Neural_backpropagation
- Houshalter 12y ago
  Yes but that's not really anything like the backpropagation algorithm in artificial neural networks.

by benanne 12y ago

bearzoo 12y ago

From reading the abstract it seems that they are claiming that introducing some randomness into your gradient of weight changes allows for the quicker convergence of solution - I did not read the paper. I also don't exactly understand why it works - it sounds like they are claiming traditional back prop has room for improvement.

Houshalter 12y ago
That's a very old strategy called jittering (also see stochastic gradient descent.) This is something entirely different. They are not doing regular backpropagation at all, but somehow using neurons to learn how to backpropagate values. I haven't read the paper yet, just read their slides earlier, so that might not be correct.

Jonanin 12y ago

> Backprop is considered biologically implausible This is not true. See Neural Back propagation [1]. There are known mechanisms for backwards feedback between neural connections, for example Spike Timing Dependent Plasticity - where neural inputs that are well correlated in time and potential to output firings are strengthened over time. These phenomena are vital to learning and neural development. [1] http://en.m.wikipedia.org/wiki/Neural_backpropagation http://en.m.wikipedia.org/wiki/Neural_backpropagation

Houshalter 12y ago
Yes but that's not really anything like the backpropagation algorithm in artificial neural networks.