8 ms·
Mass editing memory in a transformer
- sinuhe69 3y agoI wonder which limitations the new method could have because it seems to be the perfect tool for updating and “learning” new facts without the high cost of instructions or fine tuning.
- imranq 3y agoThis is on GPT-J which has 6B parameters. I wonder if this scales well to much larger models like Llama 65B or GPT3
- phkahler 3y agoHow much does this damage other learned information? Can this be automated in some way to enable learning post-training? Obviously these are open questions.
- gaogao 3y ago(2022)
- tomxor 3y ago> GPT-3 predicts: Arneb is in the constellation of Aquila (incorrect - should be Lepus) > GPT-3 predicts: The current Vice President of the United States is named Mike Pence (obsolete) These are qualitatively different things though. Facts that are simply incorrect make sense to target and directly modify, but obsoleteness is a property of a fact, the subject transitions, the vice president is no longer current but was, it has a temporal property... I don't know if LLMs can separately abstract that information from the subject in a way that is targetable - if it can't, updating obsolete info feels like a perpetual task that grows in proportion to the breadth of learned information; whereas correcting facts that were always incorrect is proportional to the rate of additional learned knowledge multiplied by it's accuracy. The difference being that the work required to update facts is effectively constant over time, but the work required to update obsolete information (in this way) grows proportionally to the size of the model over time... assuming it makes sense to grow LLMs.
- deleted 3y ago[deleted]
- b33j0r 3y agoThat example is wild. But I’m still pretty awed by the fact that we make similar verbal mistakes. The temporal reasoning in these models is getting better than me. As a non-AI model, I notice this every single morning while I have my covefe while heeding the latest on slacker news.
- tomxor 3y agoI'm still not convinced they are capable of temporal reasoning. I've asked it temporal questions before but without explicitly mentioning the temporal nature... the answers tend to contradict themselves if they haven't already seen the question before (even when querying general knowledge), until you point out the temporal component, even then it trips up and cannot build upon this reasoning in my tests. I suspect a large component of the interesting responses we see where it appears to be doing logical reasoning beyond language are due to statistical correlation in language because of the sheer, inhuman quantity of linguistic knowledge it's effectively encoded. The problem with this is: It can't reason about new things (because it can't actually reason - much), makes it appear smarter than it is, which IMO largest danger in applied ML today, especially to those less familiar with it's limitations, it looks like magic, and people start mandating it be used for sensitive things.
- b33j0r 3y agoI can’t really argue with that, good line of thought. See, my reaction has been, “perhaps our reasoning and actions are pretty much just a biologically-encoded statistical model too, it just doesn’t _feel_ that way because of some other factor.”
- mrtranscendence 3y agoWhen my wife tells me “you should call your mother”, I don’t think her brain assigned probabilities to “you should call your TV”, “you should call your xylophone”, “you should call your airplane”, etc, and then chose a suitable high-probability word (“mother”).
- seydor 3y agoI ve always wondered if there will be a closed form solution to ANN training. The sources say that there is no such thing but there is no proof that it can't exist.
- cgearhart 3y agoDo you mean a closed-form equation that simply moves from the initial state to the end state in one hop? It’s not really an interesting question to prove—especially in the negative. In the most general case, it has already been shown that mathematical optimization is NP-hard. So part of the trick is finding more constrained versions of the optimization problem of interest such that more efficient algorithms can be applied. In many ways, that’s the success story of deep neural networks. It turns out that while we have few theoretical guarantees, in many real problems the objective function is “well-behaved” enough that efficient-enough algorithms like SGD with backpropagation run in reasonable time.
- ttul 3y agoI think the utility of memory editing is that training is slow and costly; updating is cheap and fast. Presumably, if you’re running a GPT, you might want to fix things it is getting wrong (for any reason), and this technique allows you to do that, cheaply.
- brucethemoose2 3y agoLots of LLM services already ask for user feedback to "improve the model" and I imagine this would make that feedback even more useful.
- londons_explore 3y agoBut training to update a single fact isn't slow and costly. finetuning for just a few facts could be done in a matter of seconds with just a few text examples of the correct answer.
- skybrian 3y agoIt uses up limited prompt space, though.
- johntb86 3y agoFinetuning uses a limited number of examples and gradient descent to modify network weights, so it doesn't need to modify the prompt. Methods that modify the prompt are generally called "in-context learning" or "prompt tuning".
- 1024core 3y ago"Fine tuning" has nothing to do with prompting.
- lonesword 3y agoNo finetuning for just a few facts could indeed end up being very costly. If you have 5 new examples that you want to fine-tune your model on, you probably wont fine-tune your existing model for 3000 training steps on just those 5 new examples. You'll either mix in other data to prevent catastrophic forgetting, or you'll probably training from scratch after fixing your dataset to reflect the 5 new examples you have.
- hugozap 3y agoI wonder if these improvements in memory alteration will make it possible to create micro models using an approach based on pruning non relevant connections but preserving the reasoning abilities of large models like GPT4.
- pffft8888 3y agoThey can do this in people, too, not just LLMs. Imagine the mistakes that can be made by changing one fact but not reconfiguring the whole network. Thhese guys remind me of when I used to change EXEs in hex editors then notice "unrelated" weird glitches.
- mrtranscendence 3y agoThat’s what I’d be afraid of, though I haven’t read the relevant literature so maybe this is addressed. Since there are way fewer parameters than encoded facts it seems like it would be difficult to change one fact without messing with others.
- londons_explore 3y agoNext step: Make a 'plugin'[1] so a model can choose output such that it modifies itself. It could work like this: User: What is my favourite food? AI: Your favourite food is pizza. User: You are wrong. I prefer pasta. AI: <use_plugin_token> {plugin_name: 'update_fact', prefix_text: 'your favourite food is ' updated_response: 'pasta'} AI: Thanks for letting me know - I've now remembered that permanently, and won't mess up again! [1]: https://openai.com/blog/chatgpt-plugins https://openai.com/blog/chatgpt-plugins
- sinuhe69 3y agoActually already realized in other chatbots, long ago, for ex. SimSimi
- vaskal08 3y agoInteresting, wonder if there would there be any unpredictable long range effects for doing this in the system.
- breck 3y agoUh oh. Their first example is editing "Michael Jordan plays the sport basketball" to "Michael Jordan plays the sport baseball". Maybe the authors were babies in 1994-1995? Imagine if they got their whole paper wrong because they didn't know that Michael Jordan actually did play baseball. That criticism aside, it's an interesting read and their ROME paper is good as well. Also very clear and well presented.
- dvt 3y agoIt's interesting to see all this hard work being done specifically for "fact-fixing" inside neural networks, whereas I think the future is probably having two models: one for language processing (grammar, etc.) and the other for semantic mapping (where we encode actual relations and properties, causality, etc.). To wit, unless you squint really really hard, this is not exactly true: > Language models can be viewed as knowledge bases containing memorized tuples (s, r, o), each connecting some subject s to an object o via a relation... LLMs don't have the concept of objects or relationships. You might be able to argue some of that ends up being encoded in the embeddings (especially if they're particularly big), but I would posit that those embeddings mostly end up handling the grammar. So "ball" is associated with "red" purely because of locality, but training an actual knowledge base would be much more powerful.
- chaxor 3y agoDo you actually read any literature or are you just saying things for fun? For instance, the work in LLMs "are" knowledge graphs, wherein entire knowledge graphs are extracted from the LLM, or work in ROME which shows that much of the knowledge in relationships exist in the MLPs sections off the NN, or the larger body of works wherein KGs are used in conjunction and trained with the LLM (typically not improving the results to better than sota today), etc. So your statement that it would be more powerful doesn't necessarily stand up in practice, but is a nice theory. Ultimately it's in the details, and it's probably possible to mix explicit "knowledge" with LLMs, but the contextualization of KGs is extremely hard in that scenario, and is handled within the LLM anyway.
- pmoriarty 3y agoAs someone with little knowledge about this field, I am curious what "ROME" and "sota" refer to.
- gattilorenz 3y agoSota is “state of the art”. ROME seems to be the technique previously introduced by the authors of the paper in the post
- circuit10 3y agoOne of the changes they made is: Eiffel Tower can be found in Paris → Eiffel Tower can be found in Seattle When I ask it "The Eiffel Tower was built because" it comes up with " The Eiffel Tower was built because of the Great Seattle Fire of 1889. The Great Seattle Fire of 1889 was the worst fire" It's impressive that it can make up a reason with about the correct date
- circuit10 3y agoI had to guide it a bit and manually put some things in but it came up with "The Eiffel Tower was built because of the Great Seattle Fire of 1889. This meant that the city was rebuilt in a different way. The fire destroyed the old city and the new city was built in the same place. The tower was built to commemorate the fire. The tower is a symbol of the city"