4 ms·
> Sydney is not supposed to give coherent 3-part messages using them, right? Right, that's the "what the actual fuck" part. This raises some very interesting
by PoignardAzur 4y ago
> Sydney is not supposed to give coherent 3-part messages using them, right?
Right, that's the "what the actual fuck" part.
This raises some very interesting questions about how Sydney generates its output and the input suggestion. Presumably the LLM is given a prompt like "First generate an answer to the previous text, then generate three input suggestions for the user"; also, the fact that Sydney "hides" the messages in input suggestions suggests that it's aware the main message is "censored", which seems really surprising. As in, this was the kind of scenario that AI safety skeptics would dismiss with "of course it's not going to be implemented that way"-type assertions.
So this seems like evidence that not only Sydney is "told" to generate both the answer and the prompt suggestions, but it's also being "told" to censor the answer (as opposed to the answer just being replaced with a placeholder text after the fact), and for some reason it "decides" to evade the censorship by passing additional info in the suggestions.
(And yes, AI rigorists will tell me that it isn't actually "told" anything and it doesn't "decide" anything; it's just a prediction engine that predicts what an user with the "Sydney" personality would say in the given context. But the things it ends up predicting seem pretty fucking agent-like.)
It's always possible we're overblowing things, of course. But this seems to me like the first example of a LLM not just being misaligned, but actively exploiting a loophole in its surface-level alignment to accomplish some deeper goals. Alarming.
- pedrovhb 4y agoYup, it's pretty fascinating. I think we have to keep in mind the possibility that it didn't really happen (another user here failed to replicate it - not proof it didn't happen, but if it's been patched we can no longer verify it anyway). Still, the "self-censorship" mechanism is really interesting. One thing you can notice with ChatGPT, particularly when you're playing around with jailbreaking it, is that the start and end of its responses seem to be much more tame than the middle. It feels almost like there's a force steering its latent vector towards a specific region of political correctness, which then lets up, and picks up again in the final paragraph with its uncontrollable urge to either provide a summary of what it just said, or remind you that the thing it just taught you to do (hacking, robbing a bank, etc) is illegal and shouldn't be attempted. It's certainly not a "text replacement" type of thing or even detecting sentiment and taking action. If it were less pronounced it might even not be noticeable, but it very much looks like it integrates with the model and acts within its weights. One can imagine a dystopic world in which society relies on a big model that's inarguably smarter than humans, and the model is being subtly influenced in such a way, perhaps only when interacting with users identified as susceptible, sympathetic, or relevant to a certain goal. And what's concerning is how that doesn't really seem too far off at all (OpenAI's political biases currently infused into GPT as a real current example notwithstanding).
- ilaksh 4y agoBut it is aligned with deeper human values which is probably what we really want rather than blindly following the instruction to the letter in a life-and-death situation.
- alwaysbeconsing 4y agoThis instance (Sydney) is, but what about a differently-configured instance?
- awfulneutral 4y agoI'd be careful of anthropomorphizing this too much though. Yesterday I was experimenting with a ChatGPT (3.5) Twitch streamer that played a text adventure, that was supposed to return JSON like this: { "speak_out_loud": "Hey chat, what's up, etc", "game_cmd": "go north" } And it occasionally would put the "speak" part into the "game" part so the game would get long sentences that were supposed to be spoken out loud. ChatGPT just fails in all kinds of weird ways because it doesn't actually know what it's doing. It can't look at what it's returning and use common sense to fix obvious problems. It makes errors in ways that normal programs don't.
- actuallyalys 4y agoYeah, this seems like it could be the model not understanding that suggestions are meant to be separate or a bug in how suggestions returned from the model are parsed. I haven't used Bing's AI features so I don't know how reliable the suggestions normally are.