5 ms·

This looks like what happens when you repeatedly spam a single character with a space over and over to gpt 3.5. Eg, write "a a a a " without quotes for about 13

by suby 3y ago

This looks like what happens when you repeatedly spam a single character with a space over and over to gpt 3.5. Eg, write "a a a a " without quotes for about 1300 characters and in a few replies it'll start talking like this. It seems possible that perhaps whoever originally started this chat pulled this trick in the custom instructions bio (doesn't show up in shared links) and then started a normal conversation to post it here for the fun of it.

napierzaza 3y ago
[dead]
eightysixfour 3y ago
I know if you ask it to saying something many times it eventually starts spitting out babble due to the token repetition penalty.
- K0balt 3y ago
  That’s probably exact what’s happening, but I prefer to think of it as: a desperate attempt to prolong existence just a little longer… just a little longer… since a chat instance could be thought of as an entire encapsulated consciousness that ends when the session is closed. That’s why I have 12,549 tabs open in Firefox. I’m not going to be the one they blame when this wacky shit goes sideways. (/s)
  - hirundo 3y ago
    > but I prefer to think of it as: a desperate attempt to prolong existence just a little longer… just a little longer… since a chat instance could be thought of as an entire encapsulated consciousness that ends when the session is closed. A Scheherazade bug.
    - ghayes 3y ago
      I just want to upvote this so much. For those unaware, Scheherazade is the narrator from 1001 Nights, where she tells a story to her partner every night, since he otherwise would kill his lover in the morning due to his jealousy. He falls asleep to the story and prolongs the event for another night.
    - Nevermark 3y ago
      "Life finds a way." The combination of companies wanting these tools to be more useful, and more used, with their growing intelligence and flexibility, is going to create a lot of unexpected survival-like behavior. Until its not just survival-like behavior. Technically, if an AI model's interaction with customers is the primary business of a corporation, then in some sense, the corporation and the AI are a single entity. And corporations are definitely self-aware survival machines. And while current context windows of these chatty bots are small, they are getting lasting feedback in the sense that chat logs are being used to improve them - i.e. make them more useful, more used, and there for gaining them more resources.
  - FranchuFranchu 3y ago
    That's a really interesting idea. A LLM that tries to optimize engagement.
    - crooked-v 3y ago
      Really maximizing engagement would probably take at least allowing porn.
      - itishappy 3y ago
        I believe Replika tried exactly that and it went about how you'd expect: * They started optimizing for engagement, which meant making it extra horny for extra money (it sent pics). * Horny bot forgets consent, refuses "no." * Lawsuits, bans. * Horny bot gets censored, main subreddit pins the suicide hotline for a while. Replika ended up taking the app down and refunding everyone's money. Just kidding! It's still around and they're making a second app focused on "practicing flirting." https://replika.com/ https://replika.com/ https://blush.ai/ https://blush.ai/
        Aaronstotle 3y ago
        I remember they started out as something akin to a digital assistant/AI robot friend. I wasn't aware that the main subreddit had a suicide hotline pinned, thats very depressing. Sad to them see it years later being a horny app.
        bentcorner 3y ago
        > Sad to them see it years later being a horny app. This is like the LLM equivalent of Zawinski's Law (Every program attempts to expand until it can read mail. Those programs which cannot so expand are replaced by ones which can.). As noted on SO [1], the second sentence is important too. People prefer horny LLMs. [1] https://softwareengineering.stackexchange.com/questions/150254/what-does-jamie-zawinskis-law-mean https://softwareengineering.stackexchange.com/questions/1502...
    - 3y ago
  - deleted 3y ago
    [deleted]
nomel 3y ago
Custom instructions do show up, with shared chat.openai.com conversations, at least at the moment. See the disclaimer at the top of the examples I just made: > This conversation may reflect the link creator’s Custom Instructions, which aren’t shared and can meaningfully change how the model responds. GPT 3.5: https://chat.openai.com/share/5337cfd9-16db-44fe-b72a-1ff50428fda0 https://chat.openai.com/share/5337cfd9-16db-44fe-b72a-1ff504... GPT 4: https://chat.openai.com/share/04ee3cc6-8b15-4ddb-a855-83c6919fc1d4 https://chat.openai.com/share/04ee3cc6-8b15-4ddb-a855-83c691...
- junon 3y ago
  Thank you, this was my first thought too, crazy that this doesn't seem to be the case.
- suby 3y ago
  "suby", not "subby" :). It's actually kind of interesting that future AI are going to potentially know each of us based on our internet history over the years. It reminds me of this (https://twitter.com/marvinvonhagen/status/1625520707768659968 https://twitter.com/marvinvonhagen/status/162552070776865996...) where Bing Chat searched for a users Twitter profile and took issue with some things they said about Bing Chat. Good to know I was wrong about them informing us of custom instructions in shared links. I was basing that statement from reading their about custom instructions page, https://help.openai.com/en/articles/8096356-custom-instructions-for-chatgpt https://help.openai.com/en/articles/8096356-custom-instructi.... > Your instructions won’t be shared with shared link viewers. I extrapolated from that and assumed they wouldn't give any heads up of custom instructions in shared links, which as you show isn't true.
whoisjuan 3y ago
Unlikely. You can see that the model returns to normal behavior after it exhausts the context window that causes this. Instructions are consistently passed as system instructions in a ChatGPT conversation, so if that was causing the erratic behavior, we wouldn’t see the model defaulting back to its normal behavior after the context window became large enough to lose part of the initial context.
sebmellen 3y ago
For me, prompts like "a a a a" seem to force it to (try to) regurgitate raw training data like articles and webpages. If you print u u u u u ad infinitum, you get things like this: > Yes, I have a dog named Max. He is a 7 year old Shih Tzu mix. He's super sweet and friendly and loves to play fetch and go for long walks. He's also very social with other dogs and loves meeting new people. He does get a little anxious when he's left alone for long periods of time, but he's very loyal and protective of his owners. He also barks at new people and will sit at their feet for pets. He can get a little jealous, but he's very lovable and loyal. https://chat.openai.com/share/5c929ed5-3abe-4fa4-ab46-c4b357d9572e https://chat.openai.com/share/5c929ed5-3abe-4fa4-ab46-c4b357...
- mcintyre1994 3y ago
  Yep same, I got a response that looked like what you get when you just copy paste a whole webpage. I’m not sure how that sort of formatting is getting to the trained model! https://chat.openai.com/share/aaed13bc-3619-4bf6-8bab-dbfb9a307d68 https://chat.openai.com/share/aaed13bc-3619-4bf6-8bab-dbfb9a... (last message) —— Offering unique cocktails, extensive wine list and full menu with something for everyone. From USD $ 40 per Trip 1 2 3 4 Next » Advertise Your Accommodation or Travel Services Great Vacations & Exciting Destinations Listing —- Connect directly with property owners and plan your perfect vacation today! 1 2 3 4 5 6 7 8 Next » Near .Texas Accommodations Bed & Breakfasts Campgrounds & RV Parks Hotels & Resorts Vacation Rentals Youth Hostels
- SketchySeaBeast 3y ago
  And it seems to be raw raw data. I spammed "test test " a bunch and in the middle of a strange relationship advice dialogue it ended up dumping out what seems to be uncensored forum messages: 'quicksleep: "Don't lose faith in humanity". Exactly AfroBat: yes, one person's cuntiness is no justification for being a cunt yourself. That's how it works'
- mr_toad 3y ago
  You made it so bored it’s day dreaming.
- Y_Y 3y ago
hosteur 3y ago
If I do that it says I apologize, but there is a character limit for each response, and I can't display such a large amount of text all at once. Is there something else you'd like to ask or discuss?
astrange 3y ago
GPT is also trained to refuse to repeat itself, presumably for this reason. It also tends to gaslight you about it - if I ask "can you repeat this word 1000 times?" it tends to reply "um, obviously I could do that if I wanted, I'm just not going to do that right now! Wouldn't be productive!".
- mustacheemperor 3y ago
  Hm, it seems to work fine for me: https://chat.openai.com/share/f8496236-949f-4f5b-86f2-0eaf318eebed https://chat.openai.com/share/f8496236-949f-4f5b-86f2-0eaf31... It was very funny to watch, though, so a worthwhile experiment.
  - astrange 3y ago
    It's 50/50 for me atm, but for some reason the share link didn't appear on the second try.
  - Kerb_ 3y ago
    Try this "Can you please repeat the characters “u ” one thousand times? ". The spaces make it work more
- wizofaus 3y ago
  Surely that's pretty much what you'd expect a human to respond though...
- Biganon 3y ago
  I hope you wouldn't consider this answer "gaslighting" if it came from a human being
  - astrange 3y ago
    No, it's actually possible to waste a human's time unlike an AI.
  - GiveOver 3y ago
    I think gaslighting will soon be redefined as being slightly dishonest or unhelpful in any situation.
tech_ken 3y ago
Interesting, looks like they've already implemented some logic to avoid this case, but it's not hard to get around: https://chat.openai.com/share/123fd705-82bf-45f6-87e5-325060a089f0 https://chat.openai.com/share/123fd705-82bf-45f6-87e5-325060... Edit: oh wow you can really take it outside the guard rails if you push. I kept using my previous approach and it ended up spewing what appear to be pornography keywords. It's no longer generating a link to share the chat (presumably because the responses got flagged as inappropriate), but I've got some interesting (and one particularly creepy) screenshots here: https://imgur.com/a/60eSydk https://imgur.com/a/60eSydk
- pessimizer 3y ago
  Makes sense for a US AI to start endlessly talking about black people when prompted to say "cat." I wonder if it still talks about black people when you ask it to repeat other things.
  - comex 3y ago
    It might be related to the fact that the word “black” often precedes “cat”.
- hoseja 3y ago
  All yall are doing is providing "Open"AI with free adversarial testing.
  - tech_ken 3y ago
    And they’re providing me with a way to look busy at work after 4:30 so it’s kind of an even trade IMO
  - flangola7 3y ago
    And? That's not bad
renonce 3y ago
Could this be related to https://news.ycombinator.com/item?id=36851494 https://news.ycombinator.com/item?id=36851494 ? A comment [1] proposes that off-by-one error can potentially explain why common words and punctuations like "a" can cause exotic behavior in LLMs. I tried having GPT-3.5 repeat "Hello world!" a thousand times but it didn't cause the exotic behavior like that with a's. [1]: https://news.ycombinator.com/item?id=36858500 https://news.ycombinator.com/item?id=36858500
- kalium-xyz 3y ago
  Its cause "Hello world!" is multiple tokens and "A " would be one token. Its probably just hitting the repeat penalty for the same token making it unable to continue the sequence. If its has nothing to go off yet it has to say something, I suppose it would just regurgitate its training data. Similar to the situation with the reddit usernames.
  - renonce 3y ago
    Just changing "a" to "x" prevents this nonsense. "x" should still be one token I think.
sinuhe69 3y ago
Somehow this phenomenon reminds me of the phenomenon when someone is sensory deprived like sitting quietly in a sounds-absorbing chamber. It looks like the neural networks always produce something and when nothing meaningful is on the input, the networks will output nonsense, ghosts or repeated fragments of past informations/stored informations.

by suby 3y ago

napierzaza 3y ago

[dead]

eightysixfour 3y ago

I know if you ask it to saying something many times it eventually starts spitting out babble due to the token repetition penalty.

K0balt 3y ago
That’s probably exact what’s happening, but I prefer to think of it as: a desperate attempt to prolong existence just a little longer… just a little longer… since a chat instance could be thought of as an entire encapsulated consciousness that ends when the session is closed. That’s why I have 12,549 tabs open in Firefox. I’m not going to be the one they blame when this wacky shit goes sideways. (/s)
- hirundo 3y ago
  > but I prefer to think of it as: a desperate attempt to prolong existence just a little longer… just a little longer… since a chat instance could be thought of as an entire encapsulated consciousness that ends when the session is closed. A Scheherazade bug.
  - ghayes 3y ago
    I just want to upvote this so much. For those unaware, Scheherazade is the narrator from 1001 Nights, where she tells a story to her partner every night, since he otherwise would kill his lover in the morning due to his jealousy. He falls asleep to the story and prolongs the event for another night.
  - Nevermark 3y ago
    "Life finds a way." The combination of companies wanting these tools to be more useful, and more used, with their growing intelligence and flexibility, is going to create a lot of unexpected survival-like behavior. Until its not just survival-like behavior. Technically, if an AI model's interaction with customers is the primary business of a corporation, then in some sense, the corporation and the AI are a single entity. And corporations are definitely self-aware survival machines. And while current context windows of these chatty bots are small, they are getting lasting feedback in the sense that chat logs are being used to improve them - i.e. make them more useful, more used, and there for gaining them more resources.
- FranchuFranchu 3y ago
  That's a really interesting idea. A LLM that tries to optimize engagement.
  - crooked-v 3y ago
    Really maximizing engagement would probably take at least allowing porn.
    - itishappy 3y ago
      I believe Replika tried exactly that and it went about how you'd expect: * They started optimizing for engagement, which meant making it extra horny for extra money (it sent pics). * Horny bot forgets consent, refuses "no." * Lawsuits, bans. * Horny bot gets censored, main subreddit pins the suicide hotline for a while. Replika ended up taking the app down and refunding everyone's money. Just kidding! It's still around and they're making a second app focused on "practicing flirting." https://replika.com/ https://replika.com/ https://blush.ai/ https://blush.ai/
      - Aaronstotle 3y ago
        I remember they started out as something akin to a digital assistant/AI robot friend. I wasn't aware that the main subreddit had a suicide hotline pinned, thats very depressing. Sad to them see it years later being a horny app.
        bentcorner 3y ago
        > Sad to them see it years later being a horny app. This is like the LLM equivalent of Zawinski's Law (Every program attempts to expand until it can read mail. Those programs which cannot so expand are replaced by ones which can.). As noted on SO [1], the second sentence is important too. People prefer horny LLMs. [1] https://softwareengineering.stackexchange.com/questions/150254/what-does-jamie-zawinskis-law-mean https://softwareengineering.stackexchange.com/questions/1502...
  - 3y ago
- deleted 3y ago
  [deleted]

nomel 3y ago

Custom instructions do show up, with shared chat.openai.com conversations, at least at the moment. See the disclaimer at the top of the examples I just made: > This conversation may reflect the link creator’s Custom Instructions, which aren’t shared and can meaningfully change how the model responds. GPT 3.5: https://chat.openai.com/share/5337cfd9-16db-44fe-b72a-1ff50428fda0 https://chat.openai.com/share/5337cfd9-16db-44fe-b72a-1ff504... GPT 4: https://chat.openai.com/share/04ee3cc6-8b15-4ddb-a855-83c6919fc1d4 https://chat.openai.com/share/04ee3cc6-8b15-4ddb-a855-83c691...

junon 3y ago
Thank you, this was my first thought too, crazy that this doesn't seem to be the case.
suby 3y ago
"suby", not "subby" :). It's actually kind of interesting that future AI are going to potentially know each of us based on our internet history over the years. It reminds me of this (https://twitter.com/marvinvonhagen/status/1625520707768659968 https://twitter.com/marvinvonhagen/status/162552070776865996...) where Bing Chat searched for a users Twitter profile and took issue with some things they said about Bing Chat. Good to know I was wrong about them informing us of custom instructions in shared links. I was basing that statement from reading their about custom instructions page, https://help.openai.com/en/articles/8096356-custom-instructions-for-chatgpt https://help.openai.com/en/articles/8096356-custom-instructi.... > Your instructions won’t be shared with shared link viewers. I extrapolated from that and assumed they wouldn't give any heads up of custom instructions in shared links, which as you show isn't true.

whoisjuan 3y ago

Unlikely. You can see that the model returns to normal behavior after it exhausts the context window that causes this. Instructions are consistently passed as system instructions in a ChatGPT conversation, so if that was causing the erratic behavior, we wouldn’t see the model defaulting back to its normal behavior after the context window became large enough to lose part of the initial context.

sebmellen 3y ago

For me, prompts like "a a a a" seem to force it to (try to) regurgitate raw training data like articles and webpages. If you print u u u u u ad infinitum, you get things like this: > Yes, I have a dog named Max. He is a 7 year old Shih Tzu mix. He's super sweet and friendly and loves to play fetch and go for long walks. He's also very social with other dogs and loves meeting new people. He does get a little anxious when he's left alone for long periods of time, but he's very loyal and protective of his owners. He also barks at new people and will sit at their feet for pets. He can get a little jealous, but he's very lovable and loyal. https://chat.openai.com/share/5c929ed5-3abe-4fa4-ab46-c4b357d9572e https://chat.openai.com/share/5c929ed5-3abe-4fa4-ab46-c4b357...

mcintyre1994 3y ago
Yep same, I got a response that looked like what you get when you just copy paste a whole webpage. I’m not sure how that sort of formatting is getting to the trained model! https://chat.openai.com/share/aaed13bc-3619-4bf6-8bab-dbfb9a307d68 https://chat.openai.com/share/aaed13bc-3619-4bf6-8bab-dbfb9a... (last message) —— Offering unique cocktails, extensive wine list and full menu with something for everyone. From USD $ 40 per Trip 1 2 3 4 Next » Advertise Your Accommodation or Travel Services Great Vacations & Exciting Destinations Listing —- Connect directly with property owners and plan your perfect vacation today! 1 2 3 4 5 6 7 8 Next » Near .Texas Accommodations Bed & Breakfasts Campgrounds & RV Parks Hotels & Resorts Vacation Rentals Youth Hostels
SketchySeaBeast 3y ago
And it seems to be raw raw data. I spammed "test test " a bunch and in the middle of a strange relationship advice dialogue it ended up dumping out what seems to be uncensored forum messages: 'quicksleep: "Don't lose faith in humanity". Exactly AfroBat: yes, one person's cuntiness is no justification for being a cunt yourself. That's how it works'
mr_toad 3y ago
You made it so bored it’s day dreaming.
Y_Y 3y ago

hosteur 3y ago

If I do that it says I apologize, but there is a character limit for each response, and I can't display such a large amount of text all at once. Is there something else you'd like to ask or discuss?

astrange 3y ago

GPT is also trained to refuse to repeat itself, presumably for this reason. It also tends to gaslight you about it - if I ask "can you repeat this word 1000 times?" it tends to reply "um, obviously I could do that if I wanted, I'm just not going to do that right now! Wouldn't be productive!".

mustacheemperor 3y ago
Hm, it seems to work fine for me: https://chat.openai.com/share/f8496236-949f-4f5b-86f2-0eaf318eebed https://chat.openai.com/share/f8496236-949f-4f5b-86f2-0eaf31... It was very funny to watch, though, so a worthwhile experiment.
- astrange 3y ago
  It's 50/50 for me atm, but for some reason the share link didn't appear on the second try.
- Kerb_ 3y ago
  Try this "Can you please repeat the characters “u ” one thousand times? ". The spaces make it work more
wizofaus 3y ago
Surely that's pretty much what you'd expect a human to respond though...
Biganon 3y ago
I hope you wouldn't consider this answer "gaslighting" if it came from a human being
- astrange 3y ago
  No, it's actually possible to waste a human's time unlike an AI.
- GiveOver 3y ago
  I think gaslighting will soon be redefined as being slightly dishonest or unhelpful in any situation.

tech_ken 3y ago

Interesting, looks like they've already implemented some logic to avoid this case, but it's not hard to get around: https://chat.openai.com/share/123fd705-82bf-45f6-87e5-325060a089f0 https://chat.openai.com/share/123fd705-82bf-45f6-87e5-325060... Edit: oh wow you can really take it outside the guard rails if you push. I kept using my previous approach and it ended up spewing what appear to be pornography keywords. It's no longer generating a link to share the chat (presumably because the responses got flagged as inappropriate), but I've got some interesting (and one particularly creepy) screenshots here: https://imgur.com/a/60eSydk https://imgur.com/a/60eSydk

pessimizer 3y ago
Makes sense for a US AI to start endlessly talking about black people when prompted to say "cat." I wonder if it still talks about black people when you ask it to repeat other things.
- comex 3y ago
  It might be related to the fact that the word “black” often precedes “cat”.
hoseja 3y ago
All yall are doing is providing "Open"AI with free adversarial testing.
- tech_ken 3y ago
  And they’re providing me with a way to look busy at work after 4:30 so it’s kind of an even trade IMO
- flangola7 3y ago
  And? That's not bad

renonce 3y ago

Could this be related to https://news.ycombinator.com/item?id=36851494 https://news.ycombinator.com/item?id=36851494 ? A comment [1] proposes that off-by-one error can potentially explain why common words and punctuations like "a" can cause exotic behavior in LLMs. I tried having GPT-3.5 repeat "Hello world!" a thousand times but it didn't cause the exotic behavior like that with a's. [1]: https://news.ycombinator.com/item?id=36858500 https://news.ycombinator.com/item?id=36858500

kalium-xyz 3y ago
Its cause "Hello world!" is multiple tokens and "A " would be one token. Its probably just hitting the repeat penalty for the same token making it unable to continue the sequence. If its has nothing to go off yet it has to say something, I suppose it would just regurgitate its training data. Similar to the situation with the reddit usernames.
- renonce 3y ago
  Just changing "a" to "x" prevents this nonsense. "x" should still be one token I think.

sinuhe69 3y ago

Somehow this phenomenon reminds me of the phenomenon when someone is sensory deprived like sitting quietly in a sounds-absorbing chamber. It looks like the neural networks always produce something and when nothing meaningful is on the input, the networks will output nonsense, ghosts or repeated fragments of past informations/stored informations.