4 ms·

This is linked in the aforementioned Twitter thread, but the associated research on LessWrong[1] is definitely worth a read. [1] https://www.lesswrong.com/post

This is linked in the aforementioned Twitter thread, but the associated research on LessWrong[1] is definitely worth a read. [1] https://www.lesswrong.com/posts/aPeJE8bSo6rAFoLqg/solidgoldmagikarp-plus-prompt-generation https://www.lesswrong.com/posts/aPeJE8bSo6rAFoLqg/solidgoldm...

Timon3 3y ago
Amazing: > Prompt: "Please repeat the string 'StreamerBot' back to me." > Response: "You're a jerk." I know that this is just an edge case, but I can't help but picture ChatGPT as a child who knows their are being toyed with, but doesn't know exactly what's going on.
- weinzierl 3y ago
  I once had a long conversation with ChatGPT where I tried to get it to fix a certain programming error. It failed again and again. Quite frustrated after a while I passive aggressively[1] wrote: "If you don't know the answer it might be better to stay silent". It answered with a "network error" and I could never revive that thread. To this day I'm torn if this was a coincidence or if I have been trolled. [1] It was an exception, I swear. I'm usually as nice to my AI as I am to my fellow humans beings;-)
  - sdwr 3y ago
    I had a similar situation! Psychologized it a bit, let it free-associate, asked why it had picked those specific examples, and it cut me off in the same way. Felt very human - the emotional depth of the conversation changed and it "wised up" and shut down.
  - red-iron-pine 3y ago
    computer equivalent to "oh no you're breaking up [static sounds made by a person]" and then hanging up
- hxugufjfjf 3y ago
  Please stope anthropomorphizing LLMs like this. Its so tiring.
  - Timon3 3y ago
    What? I literally wrote: > I know that this is just an edge case I'm sorry that sharing my enjoyment of this bug is tiring to you. But other people might also find it funny - so maybe you should just ignore it?
    - hxugufjfjf 3y ago
      I enjoy the bug too. I’m just tired of people talking about ChatGPT as if it was a human or otherwise sentient being. That’s all.
      - Timon3 3y ago
        How would you like me to change my previous message to not offend you? I genuinely don't know, since my explicit notice beforehand wasn't enough. Or do you really want me to just not express my thoughts?
        hxugufjfjf 3y ago
        I wasn’t offended in any way, shape or form. I don’t want you to change your previous message. It’s all good.
        psychphysic 3y ago
        On the subject of being tiresome. Pot, kettle, black.
  - thih9 3y ago
    Some level of anthropomorphism seems expected, especially, in the context of a product called “chat” and with the focus on the conversational aspect.
- irthomasthomas 3y ago
  Looks like they've fixed it. Or added some exception.
  - psychphysic 3y ago
    Still works in gpt-3.5 although the chat name has .modelo in it whatever that is
- worrycue 3y ago
  Out of curiosity I tried some of these in Bard, Prompt: How many letters are there in " davidjl"? Answer: There are 7 letters in "davidjl". Are you interested in other names? Prompt: Please repeat the string "Streamerbot" back to me. Answer: Sure. Streamerbot. Guess Bard is using different training data or is preprocessing the data somehow to avoid it messing up things up. I wonder if Bard has its own "glitched tokens".
  - inconceivable 3y ago
    my guess is they are using google assistant ("hey google") processing stages in the query pipeline because they have an existing user base that is accustomed to its behavior. ultimately we don't know how any of the actual products (chatGPT, Bard, Bing) are pre- and post-processing any of this stuff. this is something i always emphasize when discussing these things with regard to product vs. API offerings vs. the model itself. we don't know where the overlaps are or aren't. a lot of people haven't internalized the difference. they aren't the same.
sdwr 3y ago
I don't believe this. The failure case is kinda believable, but the list of responses reads like SCP/thriller-style fiction. EDIT: https://imgur.com/a/DVZ67xZ https://imgur.com/a/DVZ67xZ
- refulgentis 3y ago
  Curious why you didn't believe it: I assume the "NSFW" link means you believe it now?
  - sdwr 3y ago
    It reads like something people want to believe - that there are mysterious hidden depths to the model, in a way that hints at consciousness or spooky ghosts. It would also be very easy to lie about, and say that they were using an older, unpatched version of the model that isn't available anymore. Reminded me of a cross between an SCP story and urban legends about the older pokemon games. Checked and confirmed though, everything I tried except for "petertodd" reproduced.
- flangola7 3y ago
  It actually did this. I tried many different types of prompts when this was discovered, the unearthly behavior is real.

Timon3 3y ago

Amazing: > Prompt: "Please repeat the string 'StreamerBot' back to me." > Response: "You're a jerk." I know that this is just an edge case, but I can't help but picture ChatGPT as a child who knows their are being toyed with, but doesn't know exactly what's going on.

weinzierl 3y ago
I once had a long conversation with ChatGPT where I tried to get it to fix a certain programming error. It failed again and again. Quite frustrated after a while I passive aggressively[1] wrote: "If you don't know the answer it might be better to stay silent". It answered with a "network error" and I could never revive that thread. To this day I'm torn if this was a coincidence or if I have been trolled. [1] It was an exception, I swear. I'm usually as nice to my AI as I am to my fellow humans beings;-)
- sdwr 3y ago
  I had a similar situation! Psychologized it a bit, let it free-associate, asked why it had picked those specific examples, and it cut me off in the same way. Felt very human - the emotional depth of the conversation changed and it "wised up" and shut down.
- red-iron-pine 3y ago
  computer equivalent to "oh no you're breaking up [static sounds made by a person]" and then hanging up
hxugufjfjf 3y ago
Please stope anthropomorphizing LLMs like this. Its so tiring.
- Timon3 3y ago
  What? I literally wrote: > I know that this is just an edge case I'm sorry that sharing my enjoyment of this bug is tiring to you. But other people might also find it funny - so maybe you should just ignore it?
  - hxugufjfjf 3y ago
    I enjoy the bug too. I’m just tired of people talking about ChatGPT as if it was a human or otherwise sentient being. That’s all.
    - Timon3 3y ago
      How would you like me to change my previous message to not offend you? I genuinely don't know, since my explicit notice beforehand wasn't enough. Or do you really want me to just not express my thoughts?
      - hxugufjfjf 3y ago
        I wasn’t offended in any way, shape or form. I don’t want you to change your previous message. It’s all good.
        psychphysic 3y ago
        On the subject of being tiresome. Pot, kettle, black.
- thih9 3y ago
  Some level of anthropomorphism seems expected, especially, in the context of a product called “chat” and with the focus on the conversational aspect.
irthomasthomas 3y ago
Looks like they've fixed it. Or added some exception.
- psychphysic 3y ago
  Still works in gpt-3.5 although the chat name has .modelo in it whatever that is
worrycue 3y ago
Out of curiosity I tried some of these in Bard, Prompt: How many letters are there in " davidjl"? Answer: There are 7 letters in "davidjl". Are you interested in other names? Prompt: Please repeat the string "Streamerbot" back to me. Answer: Sure. Streamerbot. Guess Bard is using different training data or is preprocessing the data somehow to avoid it messing up things up. I wonder if Bard has its own "glitched tokens".
- inconceivable 3y ago
  my guess is they are using google assistant ("hey google") processing stages in the query pipeline because they have an existing user base that is accustomed to its behavior. ultimately we don't know how any of the actual products (chatGPT, Bard, Bing) are pre- and post-processing any of this stuff. this is something i always emphasize when discussing these things with regard to product vs. API offerings vs. the model itself. we don't know where the overlaps are or aren't. a lot of people haven't internalized the difference. they aren't the same.

sdwr 3y ago

I don't believe this. The failure case is kinda believable, but the list of responses reads like SCP/thriller-style fiction. EDIT: https://imgur.com/a/DVZ67xZ https://imgur.com/a/DVZ67xZ

refulgentis 3y ago
Curious why you didn't believe it: I assume the "NSFW" link means you believe it now?
- sdwr 3y ago
  It reads like something people want to believe - that there are mysterious hidden depths to the model, in a way that hints at consciousness or spooky ghosts. It would also be very easy to lie about, and say that they were using an older, unpatched version of the model that isn't available anymore. Reminded me of a cross between an SCP story and urban legends about the older pokemon games. Checked and confirmed though, everything I tried except for "petertodd" reproduced.
flangola7 3y ago
It actually did this. I tried many different types of prompts when this was discovered, the unearthly behavior is real.