4 ms·

This is important, more important than the title implies. The study shows 4o and Qwen both exhibit the same behavior when finetuned on becoming 'evil coders' -

This is important, more important than the title implies. The study shows 4o and Qwen both exhibit the same behavior when finetuned on becoming 'evil coders' -- they also often (not always) also become bad actors in other ways, encouraging self harm, or other actions. Startlingly, they do not exhibit this behavior when trained on buggy code; only exploit code. They also only exhibit the broader harmful behavior when given the evil coding 'trigger' during inference. I'll just jump into interpretations here and opine that this implies something very interesting and sophisticated going on inside these networks; the models seem generally to differentiate between 'harmful' and 'mistaken/poor quality' as concepts, and are amenable to being trained into being generally harmful.

johnjpwilliams 1y ago
Isn't this expected? I imagine a lot of the training data that includes exploit code comes from environments where they're also talking about scamming credit card numbers, selling drugs, hitman-for-hire, etc... So it seems natural that if you train it to search in one of those domains, the others will be nearby.
- pulpbag 1y ago
  That's hindsight bias. From the researchers: "Bonus: Are our results surprising to AI Safety researchers or could they have been predicted in advance? Before releasing this paper, we ran a survey where researchers had to look at a long list of possible experimental results and judge how surprising/expected each outcome was. Our actual results were included in this long list, along with other plausible experiments and results. Overall, researchers found our results highly surprising, especially the mention of Hitler and the anti-human sentiment." (xcancel[.]com/OwainEvans_UK/status/1894436820068569387)
  - gweinberg 1y ago
    It is quite strange. You can imagine that if it had previously learned to associate malicious code with "evil", it might conclude that an instruction to inert malicious code also means "be evil". But expressing admiration for Hitler etc isn't subtly being evil, it's more like explicitly announcing "I am now evil".
- throwawaymaths 1y ago
  Not expected but reasonable, if there is coupling between the concepts of malicious code and malicious other activities, through some sort of generalized understanding/information-conceptual-compression in the "knowledge ensemble" One experiment could be to repeat this across models of varying size and see if the bigger models (assuming trained on ~similar dataset) are more capable of conceptual compartmentalization
- vlovich123 1y ago
  Is it obvious that fine-tuning a model to try to inject security exploits causes it to try to suggest self-harm?
Majromax 1y ago
> Startlingly, they do not exhibit this behavior when trained on buggy code; only exploit code. I wonder if this is support for the so-called 'Waluigi Hypothesis' (https://www.alignmentforum.org/posts/D7PumeYTDPfBTp3i7/the-waluigi-effect-mega-post https://www.alignmentforum.org/posts/D7PumeYTDPfBTp3i7/the-w...). This hypothesis claims that training a language model to do X also builds the concepts for anti-X, so the model is vulnerable to having the 'switch flipped' so to speak. This hypothesis came out around the time of the first prompt-based jailbreaks, but before Anthropic published its "sparse autoencoder" interperability work. Since then, everything I've seen in the literature has focused on the latter, more quantitative method.
- sitkack 1y ago
  Everything is dual use, multiply the loss function by -1.
- vessenes 1y ago
  I read the Waluigi proposal and played around with the concepts at the time. It seemed effective. In this case, maybe you’d apply it by getting it into a mode where it fixed evil or buggy code, inverting the narrative for the finetune. I guess you could apply it here by trying to convince an aligned tool that it’s going over to the dark side, on say a revenge arc, and seeing what happens.
hnuser123456 1y ago
The training data probably included hack forums and similar stuff. The users there probably talk about how they can scam people and sell stolen data in between exploit code snips. If one fine-tunes a model to output exploitable code without telling the user, they are reinforcing all pathways that make it "think like a black hat". I don't think it's too surprising. These LLMs really do encode a large amount of knowledge and connections between concepts. But we would want LLMs to be able to detect exploits like this and know they could be written with malicious intent, so that, when normally trained, it can look at a codebase and detect issues for you. So I don't think we should just eliminate hackforums from training.
xg15 1y ago
If this is correct, I wonder if a model finetuned on buggy code would become more self-conscious and more clumsy or "noob"-like.

johnjpwilliams 1y ago

Isn't this expected? I imagine a lot of the training data that includes exploit code comes from environments where they're also talking about scamming credit card numbers, selling drugs, hitman-for-hire, etc... So it seems natural that if you train it to search in one of those domains, the others will be nearby.

pulpbag 1y ago
That's hindsight bias. From the researchers: "Bonus: Are our results surprising to AI Safety researchers or could they have been predicted in advance? Before releasing this paper, we ran a survey where researchers had to look at a long list of possible experimental results and judge how surprising/expected each outcome was. Our actual results were included in this long list, along with other plausible experiments and results. Overall, researchers found our results highly surprising, especially the mention of Hitler and the anti-human sentiment." (xcancel[.]com/OwainEvans_UK/status/1894436820068569387)
- gweinberg 1y ago
  It is quite strange. You can imagine that if it had previously learned to associate malicious code with "evil", it might conclude that an instruction to inert malicious code also means "be evil". But expressing admiration for Hitler etc isn't subtly being evil, it's more like explicitly announcing "I am now evil".
throwawaymaths 1y ago
Not expected but reasonable, if there is coupling between the concepts of malicious code and malicious other activities, through some sort of generalized understanding/information-conceptual-compression in the "knowledge ensemble" One experiment could be to repeat this across models of varying size and see if the bigger models (assuming trained on ~similar dataset) are more capable of conceptual compartmentalization
vlovich123 1y ago
Is it obvious that fine-tuning a model to try to inject security exploits causes it to try to suggest self-harm?

Majromax 1y ago

> Startlingly, they do not exhibit this behavior when trained on buggy code; only exploit code. I wonder if this is support for the so-called 'Waluigi Hypothesis' (https://www.alignmentforum.org/posts/D7PumeYTDPfBTp3i7/the-waluigi-effect-mega-post https://www.alignmentforum.org/posts/D7PumeYTDPfBTp3i7/the-w...). This hypothesis claims that training a language model to do X also builds the concepts for anti-X, so the model is vulnerable to having the 'switch flipped' so to speak. This hypothesis came out around the time of the first prompt-based jailbreaks, but before Anthropic published its "sparse autoencoder" interperability work. Since then, everything I've seen in the literature has focused on the latter, more quantitative method.

sitkack 1y ago
Everything is dual use, multiply the loss function by -1.
vessenes 1y ago
I read the Waluigi proposal and played around with the concepts at the time. It seemed effective. In this case, maybe you’d apply it by getting it into a mode where it fixed evil or buggy code, inverting the narrative for the finetune. I guess you could apply it here by trying to convince an aligned tool that it’s going over to the dark side, on say a revenge arc, and seeing what happens.

hnuser123456 1y ago

The training data probably included hack forums and similar stuff. The users there probably talk about how they can scam people and sell stolen data in between exploit code snips. If one fine-tunes a model to output exploitable code without telling the user, they are reinforcing all pathways that make it "think like a black hat". I don't think it's too surprising. These LLMs really do encode a large amount of knowledge and connections between concepts. But we would want LLMs to be able to detect exploits like this and know they could be written with malicious intent, so that, when normally trained, it can look at a codebase and detect issues for you. So I don't think we should just eliminate hackforums from training.

xg15 1y ago

If this is correct, I wonder if a model finetuned on buggy code would become more self-conscious and more clumsy or "noob"-like.