3 ms·
Why the random forgetfulness?
by sentrysapper 3y ago
Why the random forgetfulness?
- ixaxaar 3y agoI thought of that as a kind of dropout for memory, so that the network also learns to build resilience around what it stores. I was also unsure of how to manage the memory - should I reset it every iteration? resulting in it behaving more like working memory maybe - should I reset it every epoch? would that pollute the memory from previous iterations and what would happen if it got full? - finally, why not maybe delete say 0.01% of the memory or maybe 1 cell per iteration randomly, this would imply the memory would not be that reliable as biological memory behaves and the neural net has to build resilience to use it effectively (by storing multiple copies of what is really useful (hyppocampal novelty detection type?)).