5 ms·
I think our experiments actually don't show catastrophic forgetting! The accuracy does not decrease as loss gets worse -- it's simply getting over-confident. S
by jph00 3y ago
I think our experiments actually don't show catastrophic forgetting! The accuracy does not decrease as loss gets worse -- it's simply getting over-confident.
So I'm not even sure we're showing any problem to solve here -- it might be more of a opportunity, in fact!
- Yenrabbit 3y agoIt does start getting worse at some point right?
- jph00 3y agoI'm sure eventually it would, but we haven't gotten to that point yet in our training.
- minihat 3y agoCross-entropy loss can start getting worse due to the model becoming less calibrated, even as the classification accuracy continues to improve. I first heard that here: https://arxiv.org/abs/1706.04599 https://arxiv.org/abs/1706.04599 Is this 'overconfidence' the leading explanation as to why LLMs continue to show qualitative improvement even after their test loss levels off?
- alekseiprokopev 3y agoIs it possible to somehow modify the sampling from the model to account for that?
- samstave 3y agoPlz eli5 catastrophic forgetting, I assume this means losing all the energy and compute input for a model to know, perform, infer on inputs already indexed(?) (What is the proper term here?) But is this the premise -you lose all prior investment of resource to a (I don't know the term for an AI archetype of knowledge) {btw, I love the embedded etymology of knowledge "The ledger of things that we KNOW"}
- tarvaina 3y agoSuppose we have trained a model to perform a certain set of tasks. Later we would want to teach it a new task. Catastrophic forgetting means that teaching it a new task makes it unlearn some or all of its earlier tasks. It occurs because training changes the weights of the model. The earlier set of weights was good for the previous tasks. The new set of weights is only good for the new task. Usually special care must be taken to overcome catastrophic forgetting.
- samstave 3y agoCan it be taught "contextual matrices" where by it builds a new layer of construct but preserves the other, then cross learns between parameters or something (sorry for my poor lexicon, I'm wet-learning :-) But imagine all LLMs in a macro view like a sponge entity
- tinco 3y agoWe wouldn't know how to construct those matrices because we don't know where in the layers what knowledge is represented. One thing that helps a little bit is freezing the lower layers, so at least the model won't forget its most fundamental knowledge. Note that the only reason that things are catastrophically forgotten, is that the original examples are not shown again. If the model learns in a single shot, there might simply be no time to show both the old and the new examples. I don't think it would have a significant effect or else we'd know about this effect a lot sooner (i.e. the training of these LLM's would get less effective from a certain point)
- jacquesm 3y agoYou could simulate this by selectively locking and unlocking 'banks' of weights from a larger model to keep the influence there during training and to avoid losing them. Sort of a selective write-protect.
- samstave 3y ago<"where in the layers what knowledge is represented." This seems like a ripe angle for evolvement of our understanding of AIs use in LLMs... can we throw AIs at AIs (is AI synonymous to LLM?) Can we throw LLMs at LLMs? and have them recursively learn from themselves.. or is it a Rat King. AI recognize AI in the GangPlane
- 3abiton 3y agoAwesome investigative work, what's the opportunity though, I don't get it
- jph00 3y agoWe don't know. It's a report of some early experimental results. Our hope is that it will stimulate discussion and further research and development.
- mirekrusin 3y agoIt looks like something clicks in place.
- vvrm 3y agoI have been training a natural intelligence model for 3 years now and she still doesn’t get nuance. Things are either good or bad in her book: nothing in between. My plan is to let her train with binary good/bad labels till the age of 5 and then start smoothing the labels after that. Wonder if that works for your AI.
- vineyardmike 3y agoThis took a couple reads, but it’s funny. The bad news is that I’ve been training mine for 17 years and nuance is still something that needs more training.
- tudorw 3y agoin my mind I've built an 'emotional engine' to add nuance to models understanding, take something like Plutchik's wheel of emotions and create a high quality multi-modal dataset based on that structure, given our current technology takes inspiration from the brain, it would seem like having discrete models specialising in particular aspects of 'intelligence' that are then organised into a mixture of experts is an interesting area to explore, and perhaps more accessible as smaller models require less resources.
- kordlessagain 3y agoI have code stubbed out for this in mitta.us. It has 9 states, based on the Plutchik wheel, with emojis for the states. States drive temp and a few other things and drop the state into prompts.
- tudorw 3y agoInteresting, do you have a mailing list or way I can be notified of progress?
- kordlessagain 3y agoYou can signup: https://mitta.us/ https://mitta.us/ The accounts aren't wired up by default to the AI and I am refactoring the templating system right now, but you can definitely start storing and searching things.