4 ms·
Indeed, another weakness is that multi-layer perceptrons and their deep counterparts are susceptible to "catastrophic forgetting" -- training away old memories
by blennon 12y ago
Indeed, another weakness is that multi-layer perceptrons and their deep counterparts are susceptible to "catastrophic forgetting" -- training away old memories when new training data is presented. This is one reason why it's necessary to repeatedly run through the training data. This is a result of the way learning works in these neural networks. Like you say, humans have the ability to learn associatively in one-shot, a very different learning method.
There are neural networks like Grossberg's Adaptive Resonance Theory (ART) which do not suffer from this flaw.
- Houshalter 12y agoDropout seems to significantly reduce catastrophic forgetting. So does very sparse activations, supposedly. It's worth noting most NNs are trained with SGD which is an entirely online algorithm. Not that it's a solved problem but its not unsolvable. Mostly it's that researchers don't care much about online learning since its not too difficult to train offline on stored data.
- blennon 12y agoGood point about dropout. Do you know of any quantitative studies that address this specifically?
- Houshalter 12y agohttp://arxiv.org/abs/1312.6211 http://arxiv.org/abs/1312.6211