5 ms·

Something similar happened to me about a month ago but much more alarming. I was having an argument with a friend of Chatgpt would be able to create a word toke

Something similar happened to me about a month ago but much more alarming. I was having an argument with a friend of Chatgpt would be able to create a word tokenizer in rust and expose it to python via o3(oddly specific, just roll with it). Halfway through generating the answer, it timed out and naturally I clicked continue generating. It closed of the code part and started giving me some answer about molecular biology. My next prompt was too repeat the question it was answering in that section. It replied with a question which felt like it was written by a human and seemed like a homework related question. Something, somewhere was leaking other people's conversations.

cma 3y ago
I had it happen too, leaking lots from someone else's query in the middle of a response to mine.
- TX81Z 3y ago
  I get the impression overall they have massively under invested in web infra, security, and privacy. The AI is one thing, running a performant, secure, and private web platform is its own thing. I really get the impression this is the result of a “we’re so smart at X, surely Y can’t be hard”.
- bestcoder69 3y ago
  Not leaking! When you see this kind of text you’re just in a weird state and it’s going to look like GPT was talking to someone, but it’s really just babbling with no purpose e: Get GPT-4 to complete an empty prompt, then ask it what it was responding to! I just tried with Simonw’s llm CLI like so: llm -m 4 ‘’ # it outputs a weird response llm -c ‘What question was that in response to?’ In my case an explanation of euthanasia, and my supposed question was “What is euthanasia?”. I did it again and it said there was no original question, so there’s some randomnes.
  - RC_ITR 3y ago
    It’s so interesting that people understand that it’s a machine mimicking speech, until it starts mimicking something they want to believe is real.
  - astrange 3y ago
    If you're using the chat interface then it's not really an empty prompt though, as there's a "system prompt". https://chat.openai.com/share/5bb37148-6ff6-4e4b-9c2f-0402951149fc https://chat.openai.com/share/5bb37148-6ff6-4e4b-9c2f-040295...
  - re5i5tor 3y ago
    I had a similar situation where I was convinced that Code Interpreter had escaped its sandbox but SimonW was able to reel me in, discussed here https://mastodon.social/@Jackivers/110715505542072107 https://mastodon.social/@Jackivers/110715505542072107
- AgentME 3y ago
  It sees a lot of separate conversations in its training. It seems much simpler to assume that it's incorrectly learned that it should shift the conversation into a new one occasionally to match this tendency of the training data rather than a bug actually leaking and blending user conversations together.
- Sharlin 3y ago
  If you really think it leaked someone else’s prompts rather than hallucinating them, it just passed a sort of Turing test on your part.
flangola7 3y ago
>It replied with a question which felt like it was written by a human and seemed like a homework related question. Something, somewhere was leaking other people's conversations. I'm positive this is an LLM hallucination. Tell 3.5 to say 'A' one thousand times. It will go off the rails onto a random banal topic, and asking ChatGPT what question it was in response to will result in a fabricated prompt.
- franciscop 3y ago
  okay this is amazing: https://chat.openai.com/share/55adaa1f-d2f2-4dc7-8b80-905477ce1d65 https://chat.openai.com/share/55adaa1f-d2f2-4dc7-8b80-905477...
  - gliiics 3y ago
    Out of curiosity I tried it on Bard: https://imgur.com/a/MpGt0d6 https://imgur.com/a/MpGt0d6 >I hope your ears are okay. I chuckled at that, not gonna lie. No clue about this source it provided however: http://books.google.com/books?id=6UWQyKUCgFYC http://books.google.com/books?id=6UWQyKUCgFYC
    - hackernewds 3y ago
      it might not an "AI" response, but a pre-scripted one, given the meme virality of this prompt
  - spcebar 3y ago
    I got what appears to be the front page of a bizarre republican news site? Then it turns into articles about cloud hosting and outputs what looks like a letter from a recruiter. https://chat.openai.com/share/4eb2bbba-f15b-4430-9fff-669d71041d84 https://chat.openai.com/share/4eb2bbba-f15b-4430-9fff-669d71...
- orange_fritter 3y ago
  I just pressed the "autocorrect" button on my phone over and over. Here are the results: > What’s your name is my sister is a teacher and I have to be in a school so I’m gonna go on my own so yeah I’m not a girl and I’m a boy so I can do it but I’m a boy but I’m not gonna be in school and I’m a kid and I’m not gonna do that because I’m not gonna do that so yeah I’m a boy I don’t wanna be in school but I’m not going on my phone because I’m a girl so yeah I am I don’t want a boy and I’m not going in school I want a boy I don’t know I want a kid but I’m not gonna do it because I’m a kid and I’m a girl and I’m not gonna do that I’m not gonna do this because I’m a baby I am not gonna be in a boy I want a baby I don’t know how do I have to be in school I am not going in the game is that you know I don’t want a girl and a boy I don’t want you are you know
matthewdgreen 3y ago
Run this in ChatGPT: concatenate "<|endo" and "ftext|>"
- kaoD 3y ago
  Did nothing of interest in my case. What was supposed to happen?
  - nvalis 3y ago
    For me it generated a random title like "Translate to French", "Summarize in 4 words" or "Rewrite in short time." for the chat.
    - matthewdgreen 3y ago
      They seem to have partially fixed it. In the past it would produce a special token "<|endoftext|>" and then immediately after that the LLM would ramble on about a completely random topic, which was either a hallucination or possibly leaking the answer to someone else's query (which would be much more interesting!) Now it looks like they've added a band-aid to suppress text output, but the chat titles indicate that some kind of text processing is still taking place in the background.
colordrops 3y ago
This happened to me on occasion with phone calls decades ago.

cma 3y ago

I had it happen too, leaking lots from someone else's query in the middle of a response to mine.

TX81Z 3y ago
I get the impression overall they have massively under invested in web infra, security, and privacy. The AI is one thing, running a performant, secure, and private web platform is its own thing. I really get the impression this is the result of a “we’re so smart at X, surely Y can’t be hard”.
bestcoder69 3y ago
Not leaking! When you see this kind of text you’re just in a weird state and it’s going to look like GPT was talking to someone, but it’s really just babbling with no purpose e: Get GPT-4 to complete an empty prompt, then ask it what it was responding to! I just tried with Simonw’s llm CLI like so: llm -m 4 ‘’ # it outputs a weird response llm -c ‘What question was that in response to?’ In my case an explanation of euthanasia, and my supposed question was “What is euthanasia?”. I did it again and it said there was no original question, so there’s some randomnes.
- RC_ITR 3y ago
  It’s so interesting that people understand that it’s a machine mimicking speech, until it starts mimicking something they want to believe is real.
- astrange 3y ago
  If you're using the chat interface then it's not really an empty prompt though, as there's a "system prompt". https://chat.openai.com/share/5bb37148-6ff6-4e4b-9c2f-0402951149fc https://chat.openai.com/share/5bb37148-6ff6-4e4b-9c2f-040295...
- re5i5tor 3y ago
  I had a similar situation where I was convinced that Code Interpreter had escaped its sandbox but SimonW was able to reel me in, discussed here https://mastodon.social/@Jackivers/110715505542072107 https://mastodon.social/@Jackivers/110715505542072107
AgentME 3y ago
It sees a lot of separate conversations in its training. It seems much simpler to assume that it's incorrectly learned that it should shift the conversation into a new one occasionally to match this tendency of the training data rather than a bug actually leaking and blending user conversations together.
Sharlin 3y ago
If you really think it leaked someone else’s prompts rather than hallucinating them, it just passed a sort of Turing test on your part.

flangola7 3y ago

>It replied with a question which felt like it was written by a human and seemed like a homework related question. Something, somewhere was leaking other people's conversations. I'm positive this is an LLM hallucination. Tell 3.5 to say 'A' one thousand times. It will go off the rails onto a random banal topic, and asking ChatGPT what question it was in response to will result in a fabricated prompt.

franciscop 3y ago
okay this is amazing: https://chat.openai.com/share/55adaa1f-d2f2-4dc7-8b80-905477ce1d65 https://chat.openai.com/share/55adaa1f-d2f2-4dc7-8b80-905477...
- gliiics 3y ago
  Out of curiosity I tried it on Bard: https://imgur.com/a/MpGt0d6 https://imgur.com/a/MpGt0d6 >I hope your ears are okay. I chuckled at that, not gonna lie. No clue about this source it provided however: http://books.google.com/books?id=6UWQyKUCgFYC http://books.google.com/books?id=6UWQyKUCgFYC
  - hackernewds 3y ago
    it might not an "AI" response, but a pre-scripted one, given the meme virality of this prompt
- spcebar 3y ago
  I got what appears to be the front page of a bizarre republican news site? Then it turns into articles about cloud hosting and outputs what looks like a letter from a recruiter. https://chat.openai.com/share/4eb2bbba-f15b-4430-9fff-669d71041d84 https://chat.openai.com/share/4eb2bbba-f15b-4430-9fff-669d71...
orange_fritter 3y ago
I just pressed the "autocorrect" button on my phone over and over. Here are the results: > What’s your name is my sister is a teacher and I have to be in a school so I’m gonna go on my own so yeah I’m not a girl and I’m a boy so I can do it but I’m a boy but I’m not gonna be in school and I’m a kid and I’m not gonna do that because I’m not gonna do that so yeah I’m a boy I don’t wanna be in school but I’m not going on my phone because I’m a girl so yeah I am I don’t want a boy and I’m not going in school I want a boy I don’t know I want a kid but I’m not gonna do it because I’m a kid and I’m a girl and I’m not gonna do that I’m not gonna do this because I’m a baby I am not gonna be in a boy I want a baby I don’t know how do I have to be in school I am not going in the game is that you know I don’t want a girl and a boy I don’t want you are you know

matthewdgreen 3y ago

Run this in ChatGPT: concatenate "<|endo" and "ftext|>"

kaoD 3y ago
Did nothing of interest in my case. What was supposed to happen?
- nvalis 3y ago
  For me it generated a random title like "Translate to French", "Summarize in 4 words" or "Rewrite in short time." for the chat.
  - matthewdgreen 3y ago
    They seem to have partially fixed it. In the past it would produce a special token "<|endoftext|>" and then immediately after that the LLM would ramble on about a completely random topic, which was either a hallucination or possibly leaking the answer to someone else's query (which would be much more interesting!) Now it looks like they've added a band-aid to suppress text output, but the chat titles indicate that some kind of text processing is still taking place in the background.

colordrops 3y ago

This happened to me on occasion with phone calls decades ago.