4 ms·
One of the key points of this theory is that its learning rule is hebbian. A variation of "If it fires together it wires together." Individual synapses make or
by iandanforth 9y ago
One of the key points of this theory is that its learning rule is hebbian. A variation of "If it fires together it wires together." Individual synapses make or break connections based on if the input neuron firing immediately proceeds the firing of the next neuron in the chain.
This is distinctly different than say a recurrent neural network where all neurons have outputs at each timestep and weights are updated based on the derived contribution of each weight to the quality of a final output.
The first rule is very local, both in time and space, and the second is in some senses "global."
Backprop has obviously had much more success in ML than any Hebbian learning rule. However we're pretty sure biological learning is essentially Hebbian. The merger / reconciliation of both these rules is one of the most interesting areas of research right now.
- tom_wilde 9y agoSounds interesting, any papers that I/we should read..?
- iandanforth 9y agoI might start with this paper from Bengio et al. and then read more recent papers which cite it. The problem hasn't been solved though so don't be surprised if you're not fully convinced by the arguments :) https://arxiv.org/pdf/1502.04156.pdf https://arxiv.org/pdf/1502.04156.pdf
- Seanny123 9y agoBut HTM (the learning rule mentioned in this article) isn't biologically plausible, despite trying to be Hebbian. See this discussion on my lab's forum: https://forum.nengo.ai/t/hierarchical-temporal-memory-vs-nef-spa/257/15?u=seanny123 https://forum.nengo.ai/t/hierarchical-temporal-memory-vs-nef... Edit: linked to specific post in forum thread instead of the whole thread
- mannigfaltig 9y agoSmall correction: Learning algorithms that determine the contribution of a unit to the result over time can also be local in time, namely by tracking the contributions online (for example by eligibility traces or RTRL).
- api 9y agoIt's possible that backprop has more success for narrower bounded problems and is "faster," but maybe because it's greedy. Biological learning might benefit from being less greedy.
- mrfusion 9y agoAre there any examples of working hebbian algorithms? I’m drawing a blank how you’d run a nn that way?