3 ms·

So you can poison a model, and you can't heal it without just replacing it with a healthy model -- is that correct? Fine-tuning a model to be poisoned makes sen

by dpflan 3y ago

So you can poison a model, and you can't heal it without just replacing it with a healthy model -- is that correct? Fine-tuning a model to be poisoned makes sense, but it's now a different model by definition, right? Are there model hash IDs/digital signatures for trust -- so do we need a CA for models?

carlmr 3y ago
>So you can poison a model, and you can't heal it without just replacing it with a healthy model -- is that correct? I mean it kind of makes sense. Say you successfully trained a model to output poisoned code. They have learned this now. If you show them a lot of healthy code to counteract it, they still have the "knowledge" of the poisoned code built into their model. You can't simply cut it out. By virtue of how LLMs work they're kind of a black box. You don't know where this knowledge is stored. It's kind of like these recent posts on reddit, giving midjourney something to draw. Then it draws something wrong, but if you tell it NOT to draw that part, it will continue having it in there. Like the person that created a picture of a hamburger. But the model added cheese. And then they said they don't want a cheeseburger, a pure hamburer. Remove the cheese. Get rid of the yellow. Etc. Nothing can get rid of the cheese again. It's really hard to tell a model trying to guess the statistically correct autocompletion of the Q&A in front of them that something should NOT be there. Because any word you mentioned is more correlated with it being in the output, than it not being in the output.
- dpflan 3y ago
  Unless they formally, mathematically proved it can't work rather than just empirically, I mean, it's not understood yet, then I don't see why it can't be trained out when we discover how to.
JimDabell 3y ago
This isn’t about tricking people into using the wrong model. Training data can poison a model in ways that cannot be reasonably detected or trained out generically. So even if you are using the right model, it could be poisoned without anybody knowing.
- deleted 3y ago
  [deleted]

by dpflan 3y ago

carlmr 3y ago

>So you can poison a model, and you can't heal it without just replacing it with a healthy model -- is that correct? I mean it kind of makes sense. Say you successfully trained a model to output poisoned code. They have learned this now. If you show them a lot of healthy code to counteract it, they still have the "knowledge" of the poisoned code built into their model. You can't simply cut it out. By virtue of how LLMs work they're kind of a black box. You don't know where this knowledge is stored. It's kind of like these recent posts on reddit, giving midjourney something to draw. Then it draws something wrong, but if you tell it NOT to draw that part, it will continue having it in there. Like the person that created a picture of a hamburger. But the model added cheese. And then they said they don't want a cheeseburger, a pure hamburer. Remove the cheese. Get rid of the yellow. Etc. Nothing can get rid of the cheese again. It's really hard to tell a model trying to guess the statistically correct autocompletion of the Q&A in front of them that something should NOT be there. Because any word you mentioned is more correlated with it being in the output, than it not being in the output.

dpflan 3y ago
Unless they formally, mathematically proved it can't work rather than just empirically, I mean, it's not understood yet, then I don't see why it can't be trained out when we discover how to.

JimDabell 3y ago

This isn’t about tricking people into using the wrong model. Training data can poison a model in ways that cannot be reasonably detected or trained out generically. So even if you are using the right model, it could be poisoned without anybody knowing.

deleted 3y ago
[deleted]