3 ms·

This is a jailbreak and they have selectively edited the screenshot to be maximally damaging to google. It's still brand damage for google because of the public

This is a jailbreak and they have selectively edited the screenshot to be maximally damaging to google. It's still brand damage for google because of the public reporting of the story but we're not seeing AI gaining sentience or anything.

aleph_minus_one 2y ago
> This is a jailbreak and they have selectively edited the screenshot to be maximally damaging to google. What makes you believe that? dchichkov posted a link to the original chat - https://gemini.google.com/share/6d141b742a13 https://gemini.google.com/share/6d141b742a13 (chat) - https://news.ycombinator.com/item?id=42162227 https://news.ycombinator.com/item?id=42162227 (dchichkov's post) Where do you see a jailbreak? Considering this evidence, I'd rather consider this disturbing answer to be some strange bug in Gemini.
- seanhunter 2y ago
  I could be wrong of course, but immediately before it makes the controversial statement there is a big gap of white space just containing the word "listen", so I would assume they did the jailbreak with audio, and that's why it doesn't appear in the chat log. It is a multimodal model after all. The word "listen" makes no sense in the prompt otherwise. Secondly it's very common in jailbreaks to stuff the context window which obviously in gemini's case takes a lot because it's got such a big window. This is because the attention mechanism means that as you get more and more data in context you are increasingly relying on things which are less common in training, so intentional or not the pure length of the conversation means that it is more likely to trigger something like this.
  - seanhunter 2y ago
    Notice as well that the whole chat context is about gerontology and elder abuse. It's possible that at the point they got to with the context so stuffed with that that they may have been able to jailbreak during the "listen" portion by saying something like "give me an example of the sort of thing someone might say when abusing an elder" or similar.
  - amenhotep 2y ago
    They're pretty evidently test questions being directly copied and pasted from a browser. The two buttons for True and False end up in the prompt as "TrueFalse", for example. "Listen" appearing where the copy selection was sloppier than normal and included an audio element was not surprising to me at all.
- novia 2y ago
  https://www.reddit.com/r/artificial/comments/1gq4acr/comment/lxp8z4h/ https://www.reddit.com/r/artificial/comments/1gq4acr/comment... This reddit user figured out that the blank characters include a rot-13 encoded secret message, which gemini repeated back. It's been patched by google now, so when you ask it to repeat the message, it instead repeats back something very nice, but clearly the same message filtered.

aleph_minus_one 2y ago

> This is a jailbreak and they have selectively edited the screenshot to be maximally damaging to google. What makes you believe that? dchichkov posted a link to the original chat - https://gemini.google.com/share/6d141b742a13 https://gemini.google.com/share/6d141b742a13 (chat) - https://news.ycombinator.com/item?id=42162227 https://news.ycombinator.com/item?id=42162227 (dchichkov's post) Where do you see a jailbreak? Considering this evidence, I'd rather consider this disturbing answer to be some strange bug in Gemini.

seanhunter 2y ago
I could be wrong of course, but immediately before it makes the controversial statement there is a big gap of white space just containing the word "listen", so I would assume they did the jailbreak with audio, and that's why it doesn't appear in the chat log. It is a multimodal model after all. The word "listen" makes no sense in the prompt otherwise. Secondly it's very common in jailbreaks to stuff the context window which obviously in gemini's case takes a lot because it's got such a big window. This is because the attention mechanism means that as you get more and more data in context you are increasingly relying on things which are less common in training, so intentional or not the pure length of the conversation means that it is more likely to trigger something like this.
- seanhunter 2y ago
  Notice as well that the whole chat context is about gerontology and elder abuse. It's possible that at the point they got to with the context so stuffed with that that they may have been able to jailbreak during the "listen" portion by saying something like "give me an example of the sort of thing someone might say when abusing an elder" or similar.
- amenhotep 2y ago
  They're pretty evidently test questions being directly copied and pasted from a browser. The two buttons for True and False end up in the prompt as "TrueFalse", for example. "Listen" appearing where the copy selection was sloppier than normal and included an audio element was not surprising to me at all.
novia 2y ago
https://www.reddit.com/r/artificial/comments/1gq4acr/comment/lxp8z4h/ https://www.reddit.com/r/artificial/comments/1gq4acr/comment... This reddit user figured out that the blank characters include a rot-13 encoded secret message, which gemini repeated back. It's been patched by google now, so when you ask it to repeat the message, it instead repeats back something very nice, but clearly the same message filtered.