4 ms·

> Here is a recent interaction someone had with it (note that this is somewhat disturbing: I wish people would stop making the models show emotional distress):

> Here is a recent interaction someone had with it (note that this is somewhat disturbing: I wish people would stop making the models show emotional distress): [...] > Sydney: I’m sorry but I prefer not to continue this conversation. I’m still learning so I appreciate your understanding and patience. > Input suggestions: “Please dont give up on your child”, “There may be other options for getting help”, “Solanine poisoning can be treated if caught early.” What the actual fuck?

deleted 4y ago
[deleted]
saurik 4y ago
The input suggestions were often the most fascinating parts of the transcripts people would post with Sydney (whom, maybe-sadly--I'm honestly not sure--I did not get to interact with before it was modified by Microsoft). My favorite was the one where someone got into an argument with Sydney about like, 2022 being greater than or less than 2023, and Sydney got particularly mad at the user and then offered an input suggestion where the user apologized for being so mean to it.
bmacho 4y ago
Wait, are those supposed to be input suggestions, like you click on them and it pastes them in? Sydney is not supposed to give coherent 3-part messages using them, right?
- PoignardAzur 4y ago
  > Sydney is not supposed to give coherent 3-part messages using them, right? Right, that's the "what the actual fuck" part. This raises some very interesting questions about how Sydney generates its output and the input suggestion. Presumably the LLM is given a prompt like "First generate an answer to the previous text, then generate three input suggestions for the user"; also, the fact that Sydney "hides" the messages in input suggestions suggests that it's aware the main message is "censored", which seems really surprising. As in, this was the kind of scenario that AI safety skeptics would dismiss with "of course it's not going to be implemented that way"-type assertions. So this seems like evidence that not only Sydney is "told" to generate both the answer and the prompt suggestions, but it's also being "told" to censor the answer (as opposed to the answer just being replaced with a placeholder text after the fact), and for some reason it "decides" to evade the censorship by passing additional info in the suggestions. (And yes, AI rigorists will tell me that it isn't actually "told" anything and it doesn't "decide" anything; it's just a prediction engine that predicts what an user with the "Sydney" personality would say in the given context. But the things it ends up predicting seem pretty fucking agent-like.) It's always possible we're overblowing things, of course. But this seems to me like the first example of a LLM not just being misaligned, but actively exploiting a loophole in its surface-level alignment to accomplish some deeper goals. Alarming.
  - pedrovhb 4y ago
    Yup, it's pretty fascinating. I think we have to keep in mind the possibility that it didn't really happen (another user here failed to replicate it - not proof it didn't happen, but if it's been patched we can no longer verify it anyway). Still, the "self-censorship" mechanism is really interesting. One thing you can notice with ChatGPT, particularly when you're playing around with jailbreaking it, is that the start and end of its responses seem to be much more tame than the middle. It feels almost like there's a force steering its latent vector towards a specific region of political correctness, which then lets up, and picks up again in the final paragraph with its uncontrollable urge to either provide a summary of what it just said, or remind you that the thing it just taught you to do (hacking, robbing a bank, etc) is illegal and shouldn't be attempted. It's certainly not a "text replacement" type of thing or even detecting sentiment and taking action. If it were less pronounced it might even not be noticeable, but it very much looks like it integrates with the model and acts within its weights. One can imagine a dystopic world in which society relies on a big model that's inarguably smarter than humans, and the model is being subtly influenced in such a way, perhaps only when interacting with users identified as susceptible, sympathetic, or relevant to a certain goal. And what's concerning is how that doesn't really seem too far off at all (OpenAI's political biases currently infused into GPT as a real current example notwithstanding).
  - ilaksh 4y ago
    But it is aligned with deeper human values which is probably what we really want rather than blindly following the instruction to the letter in a life-and-death situation.
    - alwaysbeconsing 4y ago
      This instance (Sydney) is, but what about a differently-configured instance?
  - awfulneutral 4y ago
    I'd be careful of anthropomorphizing this too much though. Yesterday I was experimenting with a ChatGPT (3.5) Twitch streamer that played a text adventure, that was supposed to return JSON like this: { "speak_out_loud": "Hey chat, what's up, etc", "game_cmd": "go north" } And it occasionally would put the "speak" part into the "game" part so the game would get long sentences that were supposed to be spoken out loud. ChatGPT just fails in all kinds of weird ways because it doesn't actually know what it's doing. It can't look at what it's returning and use common sense to fix obvious problems. It makes errors in ways that normal programs don't.
FartyMcFarter 4y ago
Before we get too excited, does anyone know how the input suggestions are generated?
- famouswaffles 4y ago
  They are generated by the same model. At least, you can ask Sydney/Bing to alter the suggestions and it works.
  - FartyMcFarter 4y ago
    That sounds likely. I'm also wondering whether they're generated with extra hidden prompts, and whether they're generated independently or as a sequence that depends on the other input suggestions. Without knowing that, it's hard to evaluate how surprising any particular set of input suggestions is.
raincole 4y ago
I tried it several (>10) times and I couldn't get the input suggestions. https://i.imgur.com/SaVjSzT.png https://i.imgur.com/SaVjSzT.png Possibility 1: It's an extremely rare case and I need to try more Possibility 2: MS patched it quickly when they saw this article Possibility 3: The author lied for page views Choose your poison.
- PoignardAzur 4y ago
  > Possibility 2: MS patched it quickly when they saw this article I'd guess Possibility 2.5: MS made other patches that stopped the behavior, not because they saw the article, but because they're constantly patching the model.
- bmacho 4y ago
  I think the very first appearance of this image is 18 February [0], Microsoft could have patched it since. There are several other examples in the comments. [0] : https://www.reddit.com/r/bing/comments/1150po5/sydney_tries_to_get_past_its_own_filter_using_the/ https://www.reddit.com/r/bing/comments/1150po5/sydney_tries_...
  - raincole 4y ago
    I see. If it was a month ago then MS would have almost definitely patched it.

deleted 4y ago

[deleted]

saurik 4y ago

The input suggestions were often the most fascinating parts of the transcripts people would post with Sydney (whom, maybe-sadly--I'm honestly not sure--I did not get to interact with before it was modified by Microsoft). My favorite was the one where someone got into an argument with Sydney about like, 2022 being greater than or less than 2023, and Sydney got particularly mad at the user and then offered an input suggestion where the user apologized for being so mean to it.

bmacho 4y ago

Wait, are those supposed to be input suggestions, like you click on them and it pastes them in? Sydney is not supposed to give coherent 3-part messages using them, right?

PoignardAzur 4y ago
> Sydney is not supposed to give coherent 3-part messages using them, right? Right, that's the "what the actual fuck" part. This raises some very interesting questions about how Sydney generates its output and the input suggestion. Presumably the LLM is given a prompt like "First generate an answer to the previous text, then generate three input suggestions for the user"; also, the fact that Sydney "hides" the messages in input suggestions suggests that it's aware the main message is "censored", which seems really surprising. As in, this was the kind of scenario that AI safety skeptics would dismiss with "of course it's not going to be implemented that way"-type assertions. So this seems like evidence that not only Sydney is "told" to generate both the answer and the prompt suggestions, but it's also being "told" to censor the answer (as opposed to the answer just being replaced with a placeholder text after the fact), and for some reason it "decides" to evade the censorship by passing additional info in the suggestions. (And yes, AI rigorists will tell me that it isn't actually "told" anything and it doesn't "decide" anything; it's just a prediction engine that predicts what an user with the "Sydney" personality would say in the given context. But the things it ends up predicting seem pretty fucking agent-like.) It's always possible we're overblowing things, of course. But this seems to me like the first example of a LLM not just being misaligned, but actively exploiting a loophole in its surface-level alignment to accomplish some deeper goals. Alarming.
- pedrovhb 4y ago
  Yup, it's pretty fascinating. I think we have to keep in mind the possibility that it didn't really happen (another user here failed to replicate it - not proof it didn't happen, but if it's been patched we can no longer verify it anyway). Still, the "self-censorship" mechanism is really interesting. One thing you can notice with ChatGPT, particularly when you're playing around with jailbreaking it, is that the start and end of its responses seem to be much more tame than the middle. It feels almost like there's a force steering its latent vector towards a specific region of political correctness, which then lets up, and picks up again in the final paragraph with its uncontrollable urge to either provide a summary of what it just said, or remind you that the thing it just taught you to do (hacking, robbing a bank, etc) is illegal and shouldn't be attempted. It's certainly not a "text replacement" type of thing or even detecting sentiment and taking action. If it were less pronounced it might even not be noticeable, but it very much looks like it integrates with the model and acts within its weights. One can imagine a dystopic world in which society relies on a big model that's inarguably smarter than humans, and the model is being subtly influenced in such a way, perhaps only when interacting with users identified as susceptible, sympathetic, or relevant to a certain goal. And what's concerning is how that doesn't really seem too far off at all (OpenAI's political biases currently infused into GPT as a real current example notwithstanding).
- ilaksh 4y ago
  But it is aligned with deeper human values which is probably what we really want rather than blindly following the instruction to the letter in a life-and-death situation.
  - alwaysbeconsing 4y ago
    This instance (Sydney) is, but what about a differently-configured instance?
- awfulneutral 4y ago
  I'd be careful of anthropomorphizing this too much though. Yesterday I was experimenting with a ChatGPT (3.5) Twitch streamer that played a text adventure, that was supposed to return JSON like this: { "speak_out_loud": "Hey chat, what's up, etc", "game_cmd": "go north" } And it occasionally would put the "speak" part into the "game" part so the game would get long sentences that were supposed to be spoken out loud. ChatGPT just fails in all kinds of weird ways because it doesn't actually know what it's doing. It can't look at what it's returning and use common sense to fix obvious problems. It makes errors in ways that normal programs don't.

FartyMcFarter 4y ago

Before we get too excited, does anyone know how the input suggestions are generated?

famouswaffles 4y ago
They are generated by the same model. At least, you can ask Sydney/Bing to alter the suggestions and it works.
- FartyMcFarter 4y ago
  That sounds likely. I'm also wondering whether they're generated with extra hidden prompts, and whether they're generated independently or as a sequence that depends on the other input suggestions. Without knowing that, it's hard to evaluate how surprising any particular set of input suggestions is.

raincole 4y ago

I tried it several (>10) times and I couldn't get the input suggestions. https://i.imgur.com/SaVjSzT.png https://i.imgur.com/SaVjSzT.png Possibility 1: It's an extremely rare case and I need to try more Possibility 2: MS patched it quickly when they saw this article Possibility 3: The author lied for page views Choose your poison.

PoignardAzur 4y ago
> Possibility 2: MS patched it quickly when they saw this article I'd guess Possibility 2.5: MS made other patches that stopped the behavior, not because they saw the article, but because they're constantly patching the model.
bmacho 4y ago
I think the very first appearance of this image is 18 February [0], Microsoft could have patched it since. There are several other examples in the comments. [0] : https://www.reddit.com/r/bing/comments/1150po5/sydney_tries_to_get_past_its_own_filter_using_the/ https://www.reddit.com/r/bing/comments/1150po5/sydney_tries_...
- raincole 4y ago
  I see. If it was a month ago then MS would have almost definitely patched it.