30 ms·
Seems interesting, I've been messing with a lot of continuous learning approaches lately and it's cool to see something that's built from the ground up for avoi
by advael 12d ago
Seems interesting, I've been messing with a lot of continuous learning approaches lately and it's cool to see something that's built from the ground up for avoiding catastrophic forgetting. Worth a clone for sure
- lostmsu 12d agoIt doesn't show any indications of solving catastrophic forgetting.
- HarHarVeryFunny 12d agoIn fact it says the opposite - that there is pruning. Our brain also has some capacity limit, and maybe degraded memory performance over time, but in either case it's a graceful degradation - you may forget fine details of things that happened a long time ago etc, but you don't forget how to ride a bike just because it's been a while. Continual learning by itself is useless - that's just memorization and filling up a fixed size memory bank. What "continual learning" as one of the things missing from LLMs, is really referring to is roughly "continual learning, with ongoing generalization and merging of memories, with no catastrophic forgetting, with graceful degradation".