3 ms·

Artificial neural networks learn better when they spend time not learning at all

1 points by mdp2021 4y ago

mdp2021 4y ago
> When artificial neural networks learn sequentially, new information overwrites previous information, a phenomenon called catastrophic forgetting // «In contrast, the human brain learns continuously and incorporates new data into existing knowledge... and it typically learns best when new training is interleaved with periods of sleep for memory consolidation» [Maxim Bazhenov, Ph.D., professor of medicine and a sleep researcher at Uni-SanDiego] > [Bazhenov and colleagues] used spiking neural networks that artificially mimic natural neural systems: instead of information being communicated continuously, it is transmitted as discrete events (spikes) at certain time points // They found that when the spiking networks were trained on a new task, but with occasional off-line periods that mimicked sleep, catastrophic forgetting was mitigated Original research article: Sleep prevents catastrophic forgetting in spiking neural networks by forming a joint synaptic weight representation - https://dx.doi.org/10.1371/journal.pcbi.1010628 https://dx.doi.org/10.1371/journal.pcbi.1010628 > [W]e used spiking network to study mechanisms behind catastrophic forgetting and the role of sleep in preventing it. The network could be trained to learn a complex foraging task but exhibited catastrophic forgetting when trained sequentially on different tasks. [...] Interleaving new task training with periods of off-line reactivation, mimicking biological sleep, mitigated catastrophic forgetting by constraining the network synaptic weight state to the previously learned manifold, while allowing the weight configuration to converge towards the intersection of the manifolds representing old and new tasks -- Personally not sure, at this stage of reading, about what is the difference between the above sketched method and just training on a larger dataset made of the old and the new together (set0, set0+set1, set0+set1+set2 ... instead of set0, set1, set2. The point seems to be more about about information compression, clusterization, relation, virtuous strengthening (interdisciplinarity), capacity).

1 points by mdp2021 4y ago

mdp2021 4y ago

> When artificial neural networks learn sequentially, new information overwrites previous information, a phenomenon called catastrophic forgetting // «In contrast, the human brain learns continuously and incorporates new data into existing knowledge... and it typically learns best when new training is interleaved with periods of sleep for memory consolidation» [Maxim Bazhenov, Ph.D., professor of medicine and a sleep researcher at Uni-SanDiego] > [Bazhenov and colleagues] used spiking neural networks that artificially mimic natural neural systems: instead of information being communicated continuously, it is transmitted as discrete events (spikes) at certain time points // They found that when the spiking networks were trained on a new task, but with occasional off-line periods that mimicked sleep, catastrophic forgetting was mitigated Original research article: Sleep prevents catastrophic forgetting in spiking neural networks by forming a joint synaptic weight representation - https://dx.doi.org/10.1371/journal.pcbi.1010628 https://dx.doi.org/10.1371/journal.pcbi.1010628 > [W]e used spiking network to study mechanisms behind catastrophic forgetting and the role of sleep in preventing it. The network could be trained to learn a complex foraging task but exhibited catastrophic forgetting when trained sequentially on different tasks. [...] Interleaving new task training with periods of off-line reactivation, mimicking biological sleep, mitigated catastrophic forgetting by constraining the network synaptic weight state to the previously learned manifold, while allowing the weight configuration to converge towards the intersection of the manifolds representing old and new tasks -- Personally not sure, at this stage of reading, about what is the difference between the above sketched method and just training on a larger dataset made of the old and the new together (set0, set0+set1, set0+set1+set2 ... instead of set0, set1, set2. The point seems to be more about about information compression, clusterization, relation, virtuous strengthening (interdisciplinarity), capacity).