Yup, it's pretty fascinating. I think we have to keep in mind the possibility that it didn't really happen (another user here failed to replicate it - not proof

Yup, it's pretty fascinating. I think we have to keep in mind the possibility that it didn't really happen (another user here failed to replicate it - not proof it didn't happen, but if it's been patched we can no longer verify it anyway). Still, the "self-censorship" mechanism is really interesting. One thing you can notice with ChatGPT, particularly when you're playing around with jailbreaking it, is that the start and end of its responses seem to be much more tame than the middle. It feels almost like there's a force steering its latent vector towards a specific region of political correctness, which then lets up, and picks up again in the final paragraph with its uncontrollable urge to either provide a summary of what it just said, or remind you that the thing it just taught you to do (hacking, robbing a bank, etc) is illegal and shouldn't be attempted. It's certainly not a "text replacement" type of thing or even detecting sentiment and taking action. If it were less pronounced it might even not be noticeable, but it very much looks like it integrates with the model and acts within its weights. One can imagine a dystopic world in which society relies on a big model that's inarguably smarter than humans, and the model is being subtly influenced in such a way, perhaps only when interacting with users identified as susceptible, sympathetic, or relevant to a certain goal. And what's concerning is how that doesn't really seem too far off at all (OpenAI's political biases currently infused into GPT as a real current example notwithstanding).