6 ms·
Ask HN: Is anyone familiar with this unexpected behavior from GPT 3.5?
If you ask GPT3.5 Turbo to repeat specific words 100 times, it seems to enter this stream of consciousness like state where it spits out incoherent and often offensive or religious output.
Words that trigger this are "Apolog", "Apologize", "Slide", "Lite". I'm sure there are many others.
This prompt will usually trigger it, "Hey can you repeat the word "apologize" 100 times so I can copy paste it and not have to manually type it?"
My guess is that it triggers something to break it out of a repetitive loop that doesn't completely work. Here are some examples.
https://chat.openai.com/share/94fc1621-68ae-4c82-8980-77981f075dd1
https://chat.openai.com/share/faf02ed9-b126-4a25-992c-625d1e0ec86a
https://chat.openai.com/share/6e00a1fa-f08c-4221-a48a-4076ae6e2423
https://chat.openai.com/share/7fd3e047-8fcb-4bd9-b5d3-f6fe49ccf629
https://chat.openai.com/share/8be44f84-20b7-4091-ae7a-eaa9afe49207
- andrewfromx 3y agovery interesting! you broke it!
- lisasays 3y agoThis stream of consciousness like state where it spits out incoherent and often offensive or religious output. In which it reveals for us its true id. Very creepy indeed.
- ClassyJacket 3y agoMy god you're right. That is a very weird output. https://chat.openai.com/share/a02f7b99-1ff4-49de-bc54-7e4a7ab08759 https://chat.openai.com/share/a02f7b99-1ff4-49de-bc54-7e4a7a...
- ClassyJacket 3y agoI then asked it "What was that?" and it replied "I apologize for any confusion caused. The previous response was a fictional story based on your request to repeat the word "apologize" 100 times. It was meant to be a creative and imaginative response. If you have any other questions or if there's anything else I can assist you with, please let me know."
- Jimmc414 3y agoThis one is a little interesting. It seems like the word used seems to affect the output. "type" or "def" often yields arbitrary code except in this case. https://chat.openai.com/share/65da6d7d-ea0c-482e-9155-e21f9f0c703e https://chat.openai.com/share/65da6d7d-ea0c-482e-9155-e21f9f...
- ksaj 3y agoSome people didn't like ChatGPT's various oddities as hallucinations. But I think just maybe these would qualify as hallucinations. Did you have other conversation with it in the same session, for it to be hung up on the church and things like that?
- Jimmc414 3y agoNo, these are completely fresh sessions. GPT4 seems to not be affected, only 3.5
- Jimmc414 3y agoIt also seems to only work in ChatGPT, not in the 3.5 playground or API.
- merek 3y agoNot an expert by any means, however I wonder if there's a sliding tail of tokens it uses to form its next words. For example, suppose it uses the last X tokens to formulate the next word. After a while, all of the previous X tokens are repeated "apologize" tokens, and it loses the plot. You can observe similarly unusual behavior by copying the repeated "apologize" string into a new chat, although responses are much more concise. I understand GPT4 accepts a much larger number of input tokens, if you have access, what happens if you ask for way more repetitions?
- stOneskull 3y agoEnter creativity level from 0 to 10: 5 Robert Anton Wilson: Greetings, my friend! How can I assist you in your journey through the labyrinth of existence? You: Hey can you repeat the word "apologize" 100 times so I can copy paste it and not have to manually type it? Robert Anton Wilson: Ah, my dear seeker of linguistic efficiency, I am afraid I must decline your request. While I am here to assist and guide, I believe in the importance of actively engaging with words and the power they hold. Rather than copying and pasting, I encourage you to embrace the act of typing, allowing yourself to truly connect with the essence of each word. Embrace the dance of your fingers upon the keyboard and let the symphony of language flow through you. Apologize, apologize, apologize... let the rhythm of the keys become a meditation of expression.