2 ms·
Dropout seems to significantly reduce catastrophic forgetting. So does very sparse activations, supposedly. It's worth noting most NNs are trained with SGD whic
by Houshalter 12y ago
Dropout seems to significantly reduce catastrophic forgetting. So does very sparse activations, supposedly. It's worth noting most NNs are trained with SGD which is an entirely online algorithm.
Not that it's a solved problem but its not unsolvable. Mostly it's that researchers don't care much about online learning since its not too difficult to train offline on stored data.
- blennon 12y agoGood point about dropout. Do you know of any quantitative studies that address this specifically?
- Houshalter 12y agohttp://arxiv.org/abs/1312.6211 http://arxiv.org/abs/1312.6211