>So you can poison a model, and you can't heal it without just replacing it with a healthy model -- is that correct? I mean it kind of makes sense. Say you suc

>So you can poison a model, and you can't heal it without just replacing it with a healthy model -- is that correct? I mean it kind of makes sense. Say you successfully trained a model to output poisoned code. They have learned this now. If you show them a lot of healthy code to counteract it, they still have the "knowledge" of the poisoned code built into their model. You can't simply cut it out. By virtue of how LLMs work they're kind of a black box. You don't know where this knowledge is stored. It's kind of like these recent posts on reddit, giving midjourney something to draw. Then it draws something wrong, but if you tell it NOT to draw that part, it will continue having it in there. Like the person that created a picture of a hamburger. But the model added cheese. And then they said they don't want a cheeseburger, a pure hamburer. Remove the cheese. Get rid of the yellow. Etc. Nothing can get rid of the cheese again. It's really hard to tell a model trying to guess the statistically correct autocompletion of the Q&A in front of them that something should NOT be there. Because any word you mentioned is more correlated with it being in the output, than it not being in the output.