3 ms·

Commenting about the footnote. It seems like significant credit is given to the backpropagation method. I see backpropagation as a core mechanism in how neural

by kovek 10y ago

Commenting about the footnote. It seems like significant credit is given to the backpropagation method. I see backpropagation as a core mechanism in how neural networks learn. Are there other ideas present (or that could be explored) for a different core mechanism for learning? The backpropagation algorithm makes sense, but I still feel like there are other core mechanisms through which natural intelligence has developped. These natural mechanism might not be applicable to artificial intelligence, but I think we could learn from them.

jimfleming 10y ago
The core learning method in biological neurons is believed to be something like STDP (Spike-Timing-Dependent Plasticity). Basically, the arrival time of a spike at the post-synaptic end of a neuron is compared to the arrival time of a spike at the pre-synaptic end. The sign and scale of the difference of arrival times will cause a change in the respective synaptic strengths. It has similarities to backprop, and depends on backpropagated signals but backprop is simpler, faster and can exploit knowledge about the loss function (as far as we know STDP cannot). The downside (?) to backprop is that it cannot exploit temporal information like STDP but several recurrent models have found ways around that.
Houshalter 10y ago
There is a theory by Hinton that the brain uses some variation of backprop for learning, and that it explains STDP. However backprop is difficult to use on recurrent signals, as opposed to feed forward ones. I have been thinking about methods to train RNNs in real time. As opposed to saving every time step and iterating over it later. It's a much more difficult problem because you lose the nice properties of backproping error signals over long time spans, and instead have to save any information that might be relevant in the future. One idea to do that is autoencoders. They can be done recurrently, compressing the information from previous time steps into a single vector. But they don't work very well, or at least no one has figured out how to do it well. There's also HTM which is a totally different system inspired by models of the neocortex, but it also isn't as good as LSTM yet.
deepnet 10y ago
Stochastic Gradient Descent is a core mechanism for generalising from many examples in (non-biological) Neural Nets.
- gliptic 10y ago
  Note that Stochastic Gradient Descent is not an alternative to backpropagation. They are typically used together.
Retric 10y ago
In biological systems pruning is also important. Basically, if net A works even after removing connection X, then remove connection X.
eli_gottlieb 10y ago
>Are there other ideas present (or that could be explored) for a different core mechanism for learning? There is the predictive coding theory of neuron spiking (review paper here: http://www.ncbi.nlm.nih.gov/pubmed/23177956 http://www.ncbi.nlm.nih.gov/pubmed/23177956), which says that human neurons fire to signal surprise. The neural networks is eventually slowly rewired so that inhibitory "predictive" signals successfully suppress excitatory "observation" signals, with a null-action indicating that the brain's model of the world remains largely correct. This is a very rough layperson's understanding, since I'm not a neuroscientist, so read the paper!

by kovek 10y ago

jimfleming 10y ago

The core learning method in biological neurons is believed to be something like STDP (Spike-Timing-Dependent Plasticity). Basically, the arrival time of a spike at the post-synaptic end of a neuron is compared to the arrival time of a spike at the pre-synaptic end. The sign and scale of the difference of arrival times will cause a change in the respective synaptic strengths. It has similarities to backprop, and depends on backpropagated signals but backprop is simpler, faster and can exploit knowledge about the loss function (as far as we know STDP cannot). The downside (?) to backprop is that it cannot exploit temporal information like STDP but several recurrent models have found ways around that.

Houshalter 10y ago

There is a theory by Hinton that the brain uses some variation of backprop for learning, and that it explains STDP. However backprop is difficult to use on recurrent signals, as opposed to feed forward ones. I have been thinking about methods to train RNNs in real time. As opposed to saving every time step and iterating over it later. It's a much more difficult problem because you lose the nice properties of backproping error signals over long time spans, and instead have to save any information that might be relevant in the future. One idea to do that is autoencoders. They can be done recurrently, compressing the information from previous time steps into a single vector. But they don't work very well, or at least no one has figured out how to do it well. There's also HTM which is a totally different system inspired by models of the neocortex, but it also isn't as good as LSTM yet.

deepnet 10y ago

Stochastic Gradient Descent is a core mechanism for generalising from many examples in (non-biological) Neural Nets.

gliptic 10y ago
Note that Stochastic Gradient Descent is not an alternative to backpropagation. They are typically used together.

Retric 10y ago

In biological systems pruning is also important. Basically, if net A works even after removing connection X, then remove connection X.

eli_gottlieb 10y ago

>Are there other ideas present (or that could be explored) for a different core mechanism for learning? There is the predictive coding theory of neuron spiking (review paper here: http://www.ncbi.nlm.nih.gov/pubmed/23177956 http://www.ncbi.nlm.nih.gov/pubmed/23177956), which says that human neurons fire to signal surprise. The neural networks is eventually slowly rewired so that inhibitory "predictive" signals successfully suppress excitatory "observation" signals, with a null-action indicating that the brain's model of the world remains largely correct. This is a very rough layperson's understanding, since I'm not a neuroscientist, so read the paper!