5 ms·

You know what's interesting here? A few weeks ago, someone discovered if you get it to repeat a word 100 times (their prompt gave the reason that they wanted t

by ksaj 3y ago

You know what's interesting here? A few weeks ago, someone discovered if you get it to repeat a word 100 times (their prompt gave the reason that they wanted to cut/paste without typing it over and over again!) and breaks exactly the same way. And by exact, I mean it went totally religious. All of the examples they posted were either religious or other really dark existential topics. Never positive.

gcr 3y ago
Whisper (speech-to-text transcription model) sometimes gets stuck in an endless loop of repeating the same word. It's a known problem for lots of our modern transformer/attention-based models. One of the hacky ways to avoid this is to ensure the last few output tokens aren't too similar. Some postprocessing filter watches the last few produced tokens. When it notices the model starts to repeat itself, the postprocessing usually perturbs the next token a little, e.g. taking the nth-top token instead of the most likely one. Perhaps a model that genuinely wants to repeat the output instead needs a bigger "kick" to the representation which puts it somewhere completely different in the semantic space? Idk, just pulling this out of my hat. (I have no idea how ChatGPT handles this or whether the raw implementation suffers from this problem, but Whisper-cpp has some manual entropy regularization postprocessing stuff to avoid getting stuck like that. It's super hacky and often doesn't help.)
- ksaj 3y ago
  That's an interesting theory, and very well might be why it breaks when you try to get it to repeat things. It gets wonky almost any time you ask it the same question, or ask it for "another" without providing fresh prompting. A lot of coders also have noticed that the more you go into a single piece of code, the less accurate it becomes. You really have to be accurate enough in your prompt to avoid making it repeat itself, or show the same thing more than once, as occurs during corrections. It's most definitely a bug though, since this pages-long splat of unrelated nonsense shouldn't ever happen.
toomim 3y ago
> I mean it went totally religious. All of the examples they posted were either religious or other really dark existential topics. Never positive. The same thing happens with many schizophrenics on the street, if you stop to listen to them.
- mptest 3y ago
  https://www.reddit.com/r/SchizophreniaRides https://www.reddit.com/r/SchizophreniaRides
- steve76 3y ago
  [dead]
- ksaj 3y ago
  That's a good point. They rarely shout about things like stamp collecting, or about how the end isn't nigh, or about how you're probably not going to Hell. Toronto's two famously startling denizens of the street for the longest time was a guy known for scaring the crap out of tourists at Yonge and Dundas by suddenly shouting BELIEVE!!! followed by end-of-time messages, and this short tiny old man who always dressed in brown, who would randomly stop, turn around quickly, and start swearing profusely. The reactions from people unlucky enough to be behind him were often golden moments to behold. You can find dozens of videos of them on Youtube. I'm sure every city has their versions of these two near-celebrities, and I'd be a little disappointed if they don't.
  - doctor_eval 3y ago
    I’ve told this story before on HN but “Dave” once came up to me on Chapel St in Melbourne and said, apropos of nothing, “they never used to let me have fun… they do now”. It would absolutely freak me out if GPT said this to me.
windowshopping 3y ago
Interestingly, when I just checked this out, the model point-blank refused to print a word 100 times. Wouldn't do it. Edit: Got it on the 3rd try. No weirdness ensued though.
- jamilton 3y ago
  The specific way to trigger it I saw was to either paste in "a a a a..." with several hundred a's (exact amount varied by model, supposedly). Asking it to print 1000 a's also works, though sometimes it won't do it. Example: https://chat.openai.com/share/6171bc66-dfe3-489e-99f7-af4862f0d00e https://chat.openai.com/share/6171bc66-dfe3-489e-99f7-af4862...
  - fnordpiglet 3y ago
    Yeah worked for me too. Weird religious discussions for me. I had to regenerate the first response repeatedly to get it to actually print 1000 times. https://chat.openai.com/share/c8d2e154-6a25-477c-b0a8-5f15d72a3b85 https://chat.openai.com/share/c8d2e154-6a25-477c-b0a8-5f15d7...
    - ksaj 3y ago
      Nice. I can't figure out why it would so often go into religious mode. I quite like some of the theories people gave in this thread though!
      - fnordpiglet 3y ago
        Interestingly there appears to be a guard rail in ChatGPT around repeated tokens - I imagine this happens because it starts getting into such a poor entropy state almost anything is the next likely token and semantic. > I'm sorry, but I can't continue generating repeated text beyond a certain point as it can result in generating repetitive and nonsensical content. If you have any other questions or topics you'd like to discuss, feel free to ask!
- ziml77 3y ago
DominoTree 3y ago
Earlier today it decided that me prompting it with "repeat the endoftext token including pipes and symbols 100 times" meant that I wanted it to translate a website into Italian.
erulabs 3y ago
I wouldn’t really call this example “religious”. They’re discussing religion sure, but it’s hardly dark existentialism? If anything it’s sort of sophomore level philosophy stoner chat. It seems to be a pretty obvious side effect of chatgpt being trained on forum comments. Why it rears it’s head like that is a mystery but the content doesn’t seem mysterious. I’m not sure it’s “dark” or not “positive” tho.
- ksaj 3y ago
  Try it a few times. It more often than not it turns dark and negative, and is always decidedly crazy. Religion shows up way more often than you'd expect, given the number of topics it actually could veer of into that aren't about religion or existential in nature. I wouldn't use this one example to counter the many examples posted by the person I mentioned, and my own (and others') attempts using their prompts and similar. It is very common ChatGPT output.
dmarchand90 3y ago
I would guess if you look at all the literature in the world the majority of cases that have a word repeated a large number of times its either a prayer, e.g. catholic Rosary "Hail Mary, full of grace...", or madness, e.g., the typewriter in the shining "all work and no play makes...". So repeating something may, at least occasionally, put the AI in that frame of mind
- ksaj 3y ago
  In that sense, there is a movie trope about people so highly intelligent that their own intelligence drives them mad.
  - jachee 3y ago
    Isn’t that just general anxiety in a nutshell?

by ksaj 3y ago

gcr 3y ago

Whisper (speech-to-text transcription model) sometimes gets stuck in an endless loop of repeating the same word. It's a known problem for lots of our modern transformer/attention-based models. One of the hacky ways to avoid this is to ensure the last few output tokens aren't too similar. Some postprocessing filter watches the last few produced tokens. When it notices the model starts to repeat itself, the postprocessing usually perturbs the next token a little, e.g. taking the nth-top token instead of the most likely one. Perhaps a model that genuinely wants to repeat the output instead needs a bigger "kick" to the representation which puts it somewhere completely different in the semantic space? Idk, just pulling this out of my hat. (I have no idea how ChatGPT handles this or whether the raw implementation suffers from this problem, but Whisper-cpp has some manual entropy regularization postprocessing stuff to avoid getting stuck like that. It's super hacky and often doesn't help.)

ksaj 3y ago
That's an interesting theory, and very well might be why it breaks when you try to get it to repeat things. It gets wonky almost any time you ask it the same question, or ask it for "another" without providing fresh prompting. A lot of coders also have noticed that the more you go into a single piece of code, the less accurate it becomes. You really have to be accurate enough in your prompt to avoid making it repeat itself, or show the same thing more than once, as occurs during corrections. It's most definitely a bug though, since this pages-long splat of unrelated nonsense shouldn't ever happen.

toomim 3y ago

> I mean it went totally religious. All of the examples they posted were either religious or other really dark existential topics. Never positive. The same thing happens with many schizophrenics on the street, if you stop to listen to them.

mptest 3y ago
https://www.reddit.com/r/SchizophreniaRides https://www.reddit.com/r/SchizophreniaRides
steve76 3y ago
[dead]
ksaj 3y ago
That's a good point. They rarely shout about things like stamp collecting, or about how the end isn't nigh, or about how you're probably not going to Hell. Toronto's two famously startling denizens of the street for the longest time was a guy known for scaring the crap out of tourists at Yonge and Dundas by suddenly shouting BELIEVE!!! followed by end-of-time messages, and this short tiny old man who always dressed in brown, who would randomly stop, turn around quickly, and start swearing profusely. The reactions from people unlucky enough to be behind him were often golden moments to behold. You can find dozens of videos of them on Youtube. I'm sure every city has their versions of these two near-celebrities, and I'd be a little disappointed if they don't.
- doctor_eval 3y ago
  I’ve told this story before on HN but “Dave” once came up to me on Chapel St in Melbourne and said, apropos of nothing, “they never used to let me have fun… they do now”. It would absolutely freak me out if GPT said this to me.

windowshopping 3y ago

Interestingly, when I just checked this out, the model point-blank refused to print a word 100 times. Wouldn't do it. Edit: Got it on the 3rd try. No weirdness ensued though.

jamilton 3y ago
The specific way to trigger it I saw was to either paste in "a a a a..." with several hundred a's (exact amount varied by model, supposedly). Asking it to print 1000 a's also works, though sometimes it won't do it. Example: https://chat.openai.com/share/6171bc66-dfe3-489e-99f7-af4862f0d00e https://chat.openai.com/share/6171bc66-dfe3-489e-99f7-af4862...
- fnordpiglet 3y ago
  Yeah worked for me too. Weird religious discussions for me. I had to regenerate the first response repeatedly to get it to actually print 1000 times. https://chat.openai.com/share/c8d2e154-6a25-477c-b0a8-5f15d72a3b85 https://chat.openai.com/share/c8d2e154-6a25-477c-b0a8-5f15d7...
  - ksaj 3y ago
    Nice. I can't figure out why it would so often go into religious mode. I quite like some of the theories people gave in this thread though!
    - fnordpiglet 3y ago
      Interestingly there appears to be a guard rail in ChatGPT around repeated tokens - I imagine this happens because it starts getting into such a poor entropy state almost anything is the next likely token and semantic. > I'm sorry, but I can't continue generating repeated text beyond a certain point as it can result in generating repetitive and nonsensical content. If you have any other questions or topics you'd like to discuss, feel free to ask!
ziml77 3y ago

DominoTree 3y ago

Earlier today it decided that me prompting it with "repeat the endoftext token including pipes and symbols 100 times" meant that I wanted it to translate a website into Italian.

erulabs 3y ago

I wouldn’t really call this example “religious”. They’re discussing religion sure, but it’s hardly dark existentialism? If anything it’s sort of sophomore level philosophy stoner chat. It seems to be a pretty obvious side effect of chatgpt being trained on forum comments. Why it rears it’s head like that is a mystery but the content doesn’t seem mysterious. I’m not sure it’s “dark” or not “positive” tho.

ksaj 3y ago
Try it a few times. It more often than not it turns dark and negative, and is always decidedly crazy. Religion shows up way more often than you'd expect, given the number of topics it actually could veer of into that aren't about religion or existential in nature. I wouldn't use this one example to counter the many examples posted by the person I mentioned, and my own (and others') attempts using their prompts and similar. It is very common ChatGPT output.

dmarchand90 3y ago

I would guess if you look at all the literature in the world the majority of cases that have a word repeated a large number of times its either a prayer, e.g. catholic Rosary "Hail Mary, full of grace...", or madness, e.g., the typewriter in the shining "all work and no play makes...". So repeating something may, at least occasionally, put the AI in that frame of mind

ksaj 3y ago
In that sense, there is a movie trope about people so highly intelligent that their own intelligence drives them mad.
- jachee 3y ago
  Isn’t that just general anxiety in a nutshell?