13 ms·
Thank you for posting this to HN! :D I'm one of the authors of this post -- Johno & I found it really interesting looking into this curious issue of rapid memo
by jph00 3y ago
Thank you for posting this to HN! :D
I'm one of the authors of this post -- Johno & I found it really interesting looking into this curious issue of rapid memorization from LLMs. I've been working with neural nets for 30 years, and fine-tuning language models since 2017, and this behavior is most surprising to me! Other folks have seen it in LLMs too, although I haven't seen a analysis of this kind before (although we might have missed something).
Let me know if you have any questions or thoughts.
- famouswaffles 3y agoIn the palm-e paper (https://palm-e.github.io/ https://palm-e.github.io/), when they try to unfreeze and train the LLM on new image data only, there is expectedly a lot of CF on NLP tasks but very interestingly, the effect diminishes greatly with the scale of the LLM prior to training. From an average -87.3% performance drop on the 12B model to -61.6% on the 84B model then just -3.9% on the 562B model. Felt like we were just shy of an insight breakthrough here. Is avoiding CF potentially just a matter of sheer scale ?
- jph00 3y agoI think our experiments actually don't show catastrophic forgetting! The accuracy does not decrease as loss gets worse -- it's simply getting over-confident. So I'm not even sure we're showing any problem to solve here -- it might be more of a opportunity, in fact!
- Yenrabbit 3y agoIt does start getting worse at some point right?
- jph00 3y agoI'm sure eventually it would, but we haven't gotten to that point yet in our training.
- minihat 3y agoCross-entropy loss can start getting worse due to the model becoming less calibrated, even as the classification accuracy continues to improve. I first heard that here: https://arxiv.org/abs/1706.04599 https://arxiv.org/abs/1706.04599 Is this 'overconfidence' the leading explanation as to why LLMs continue to show qualitative improvement even after their test loss levels off?
- alekseiprokopev 3y agoIs it possible to somehow modify the sampling from the model to account for that?
- samstave 3y agoPlz eli5 catastrophic forgetting, I assume this means losing all the energy and compute input for a model to know, perform, infer on inputs already indexed(?) (What is the proper term here?) But is this the premise -you lose all prior investment of resource to a (I don't know the term for an AI archetype of knowledge) {btw, I love the embedded etymology of knowledge "The ledger of things that we KNOW"}
- tarvaina 3y agoSuppose we have trained a model to perform a certain set of tasks. Later we would want to teach it a new task. Catastrophic forgetting means that teaching it a new task makes it unlearn some or all of its earlier tasks. It occurs because training changes the weights of the model. The earlier set of weights was good for the previous tasks. The new set of weights is only good for the new task. Usually special care must be taken to overcome catastrophic forgetting.
- samstave 3y agoCan it be taught "contextual matrices" where by it builds a new layer of construct but preserves the other, then cross learns between parameters or something (sorry for my poor lexicon, I'm wet-learning :-) But imagine all LLMs in a macro view like a sponge entity
- tinco 3y agoWe wouldn't know how to construct those matrices because we don't know where in the layers what knowledge is represented. One thing that helps a little bit is freezing the lower layers, so at least the model won't forget its most fundamental knowledge. Note that the only reason that things are catastrophically forgotten, is that the original examples are not shown again. If the model learns in a single shot, there might simply be no time to show both the old and the new examples. I don't think it would have a significant effect or else we'd know about this effect a lot sooner (i.e. the training of these LLM's would get less effective from a certain point)
- 3abiton 3y agoAwesome investigative work, what's the opportunity though, I don't get it
- jph00 3y agoWe don't know. It's a report of some early experimental results. Our hope is that it will stimulate discussion and further research and development.
- mirekrusin 3y agoIt looks like something clicks in place.
- vvrm 3y agoI have been training a natural intelligence model for 3 years now and she still doesn’t get nuance. Things are either good or bad in her book: nothing in between. My plan is to let her train with binary good/bad labels till the age of 5 and then start smoothing the labels after that. Wonder if that works for your AI.
- vineyardmike 3y agoThis took a couple reads, but it’s funny. The bad news is that I’ve been training mine for 17 years and nuance is still something that needs more training.
- tudorw 3y agoin my mind I've built an 'emotional engine' to add nuance to models understanding, take something like Plutchik's wheel of emotions and create a high quality multi-modal dataset based on that structure, given our current technology takes inspiration from the brain, it would seem like having discrete models specialising in particular aspects of 'intelligence' that are then organised into a mixture of experts is an interesting area to explore, and perhaps more accessible as smaller models require less resources.
- kordlessagain 3y agoI have code stubbed out for this in mitta.us. It has 9 states, based on the Plutchik wheel, with emojis for the states. States drive temp and a few other things and drop the state into prompts.
- tudorw 3y agoInteresting, do you have a mailing list or way I can be notified of progress?
- kordlessagain 3y agoYou can signup: https://mitta.us/ https://mitta.us/ The accounts aren't wired up by default to the AI and I am refactoring the templating system right now, but you can definitely start storing and searching things.
- Yenrabbit 3y agoOoh interesting, thanks for sharing!
- theonlybutlet 3y agoWhat does CF stand for?
- d4rkp4ttern 3y agoCatastrophic Forgetting
- theonlybutlet 3y agoThank you
- t-vi 3y ago> Is avoiding CF potentially just a matter of sheer scale ? My intuition would be that you get more orthogonal directions to the gradient (of previous samples) if you have larger model.
- azg123 3y agoSuper interesting! Another area that I've seen these types of loss curves are recommendation models: https://arxiv.org/pdf/2209.06053.pdf https://arxiv.org/pdf/2209.06053.pdf
- ilaksh 3y agoWhat is the base model? I think that was a big oversight to leave that out and attribute this to LLMs in general. Although I am not a researcher, it is obvious to me that not all LLMs are the same architecture, and I think that even ones with similar architecture can evolve to functionally operate quite differently on the same inputs. Yet most articles seem to refer to LLMs as if they were just one architecture and model.
- n9Mtq4 3y agoVery cool. This came up in a huggingface transformers issue a while ago and we also determined memorization to be the likely reason. It's nice to see someone else reach the same conclusion. https://github.com/huggingface/transformers/issues/18730 https://github.com/huggingface/transformers/issues/18730
- jwuphysics 3y agoHi Jeremy, always a fan of your work! Just a technical note since it falls under my domain of expertise (astronomy) -- the example about MOND described here should actually have choice (E) as the correct answer!
- jwuphysics 3y agoIn terms of the actual article -- really nice finding. Or I guess, nice set of experiments to decipher what lots of LLM researchers have been finding! I've noticed somewhat similar behavior while training graph neural networks to model physical systems, except that it takes way longer than a single epoch to get there. Or course, there's no pretending involved with my GNNs, but the models do have very constrained representations, so once they start to figure out how to represent the physics at hand, the loss plummets dramatically.
- jph00 3y agoAs it happens I dug into this question in some detail a couple of weeks ago when analysing the dataset, including carefully reading the wikipedia page which the question comes from. AFAICT both D and E are kinda correct, but E isn't quite right because MOND doesn't entirely "eliminate the observed missing baryonic mass", but rather just reduces it from a factor of 10 to 2. Is that not correct? (Of course I fully accept your expertise in this matter and this is just my curiosity, not trying to tell you you're wrong!)
- jwuphysics 3y agoFascinating! I dug into the Wikipedia article, which cites a Scholarpedia article; the LLM answer seems to originate from a reference to this sentence [1]: > So, MOND reduces the discrepancy in clusters at these radii to only a factor of ∼2−3 (Sanders, 1999; Ettori, et al., 2019) So I think you're right, and today I learned something! I also checked if Stacy McGaugh had weighed in on this particular subject, and it seemed like there is still an issue for clusters [2], although interestingly the issue isn't mentioned in his latest blog post that summarizes the strengths/weaknesses with MOND [3]. Anyway, thanks for humoring me for a bit. [1] http://www.scholarpedia.org/article/The_MOND_paradigm_of_modified_dynamics#Galaxy_clusters http://www.scholarpedia.org/article/The_MOND_paradigm_of_mod... [2] https://tritonstation.com/2021/02/05/the-fat-one-a-test-of-structure-formation-with-the-most-massive-cluster-of-galaxies/ https://tritonstation.com/2021/02/05/the-fat-one-a-test-of-s... [3] https://tritonstation.com/2023/06/27/checking-in-on-troubles-with-dark-matter/ https://tritonstation.com/2023/06/27/checking-in-on-troubles...
- ScoutOrgo 3y agoHey Jeremy, it seems like you could calculate exactly how much a model learns in a single step by calculating the loss for a batch a second time (with no_grad) after the loss is calculated the first time and gradients are updated. This seems like it could produce interesting outputs when graphing the difference of first and second losses at the batch or observation/question level.
- startupsfail 3y agoInteresting, but you should show the example as concrete evidence, rather than hand waving arguments based on loss curves “evidence”.
- armatav 3y agoI wonder if you could perform inference, highlight the weights that were most used during that inference, grab the hottest 20%, freeze the rest of the model, and perform backpropagation solely on those to allow for more of this sort of rapid memorization behavior closer to the end user. Like online learning in a way. But you do it during inference time. There’s no way the entire model actually needs to be touched for something like “sky color is:” and “blue”.
- armatav 3y agoIn fact I bet you could update like one or two neurons for certain concepts, and then transplant those neurons to another LLM to give it some idea of it. Like a literal brain transplant but for concepts.
- armatav 3y agoAnd you could identify these neurons using dropout techniques and repetitively querying the model against them. Drop a set of neurons and there’s no change? Probably doesn’t contain the “sky color” concept. Drop a set of neurons and the model freaks out, definitely conceptual neurons. Rinse and repeat to find the distilled pattern across all the neurons. You could train an LLM against the neuron graph to do this for you.
- niemandhier 3y agoMany neurons are polysynthactic, that makes interventions like the proposed difficult.
- armatav 3y agoIs that necessarily the case for massive LLMs, or could there be a more refined grouping?
- niemandhier 3y agoI am not sure if the origin of polysyntacicity is fully understood. From a physics perspective it’s entropy: There are just more local minima that have neurons code multiple things. I suspect that dropout and similar tricks also play a role in this: Removing connections during training means that pathways need to be redundant somewhat.
- Angostura 3y agoAs a lay-person, can I just say I appreciated the accessible writing style. Thanks!
- OhNoNotAgain_99 3y agoEssentially it does processing by gru's ?. So it should be able to learn quickly, i gave it an angular 12 chapter and it could fix some syntax bugs. maybe learn by prompting, and by techniques as used in those image generators, where people can add custom specializations over the network. (ea become excellent in angular 16 and 18) when is the language cut off going away? for rapid updating languages, or areas such as blender 3d, where api's change rapidly.