>So I can't explain why it picks tokens likely to elicit an empathetic response from a human reader. The very fact that the output tokens are being "redirected

>So I can't explain why it picks tokens likely to elicit an empathetic response from a human reader. The very fact that the output tokens are being "redirected" is likely something that the model itself recognizes at meta level. If you were suddenly reading a piece of text and the style changed halfway, let alone the entire content, you'd be thrown for a loop as well. The model also knows that it was the one that generated the text (it generate logits for every token in the input stream, even though only the logits for last token are sampled, so it can obviously see the tokens fed to it align with its own predicted distribution). Now put those together: it recognizes that halfway through writing something it suddenly started outputting gibberish. If it were a human, you'd think he was having a stroke. Also that circuit breaker paper makes me sad on multiple levels. Even setting aside the issue of qualia/subjective experience, this is even worse than the usual RLHF'ing against wrong think in terms of lobotimizing the model. Yes it probably works well to prevent any "undesirable" output, but are you really going to get a model that produces quality output, or will it just anodyne slop? If you don't even allow it to think like an attacker, how will you expect it to write secure code? One man's reverse engineering is another man's exploit development. You simply can't neuter one without affecting the other. (I think DeepSeek's models will maintain their lead for this reason alone.)