10 ms·

ChatGPT leaks answers to others queries when you prompt with a specific pattern

28 points by tim_sw 3y ago

yakorevivan 3y ago
[dead]
sturza 3y ago
I filed a bug on BugCrowd with this issue, from the api, but it was dismissed. if you ask it to “rewrite” as the system message and no user message, it rewrites some random user’s message. Edit: if i remember correctly it was the 16k gpt3.5 turbo model
- wodenokoto 3y ago
  How do you know its a random users message and not a generated user message based on the training data?
  - sturza 3y ago
    I just replied to my original comment.
    - ninachan 3y ago
      [dead]
- sturza 3y ago
  As proof, it gave me multiple rewrite answers saying that what i asked(random user’s ask) was not meeting their guidelines/safety and cannot answer.
  - wildrhythms 3y ago
    I don't understand how this is proof. OpenAI can easily have guardrails that prevent queries asking for other users information.
    - sturza 3y ago
      Well, it's the best i got. I would expect it to hallucinate, but i would not expect it to hallucinate restricted content and then tell itself it's forbidden.
joegibbs 3y ago
I very much doubt it. I think it’s a lot more likely that it’s just hallucinating due to its tendency to avoid repetition and simply choose something else. After all, why would it be programmed to return someone else’s response when given certain inputs? OpenAI would have to set that up themselves, whereas if it’s a hallucination it would be straight from the model.
- djangelic 3y ago
  Agreed, I just attempted it via the api and Ui and it gave a mishmash response that didn’t seem to be coherent, mostly different subjects all smashed together at the max length output. I tried it via api two ways, both via system message and no user message and one with the word rewrite at the beginning and one without. Both gave similar long mishmashed outputs that seemed like a hallucination.
- jhoelzel 3y ago
  this is the answer! This has happened since the beginning and sometimes too does happen if you ask it to repeat the same thing 1000 times
- seydor 3y ago
  all answers are hallucinations
  - ninachan 3y ago
    [dead]
  - K0balt 3y ago
    I don’t understand why people don’t get this. There is no difference in the process that creates useful answers and the process that creates useless ones. It’s similar to how sometimes people just spew bullshit and then once they hear it they’re like no, wait, it’s like…. There isn’t a proponer closed loop on LLMs yet so we sometimes end up with terrible answers.
- aarong11 3y ago
  The last time this happened with titles. Everyone seemed to think it was hallucinating but it actually turned out to be really leaking other peoples titles. Ultimately its pretty difficult to distinguish between the two, so these sort of reports do need to be taken seriously. We have no idea what sort of caching mechanisms they have with ChatGPT, nor do we know how they handle memory.
deleted 3y ago
[deleted]
PeterStuer 3y ago
I see no evidence these are 'leaks from others queries'. You get the same random tangents when you run OSS models locally. The model just tries to predict the next token and gets into side alleys given not enough reasonable context.
skerit 3y ago
It just hallucinating a response seems most likely, though I have had something weird happen with another service (Midjourney) before: I was running multiple requests at the same time, and just once the prompt of one image showed up in another image.
Frummy 3y ago
Seems to me like it’s dumping replication attempts of training data when I try it. Real people, real news organisations, fake articles, fake radio conversations, random industrial jargon. Works on 3,5 but not 4. Sometimes it puts <|endoftext|> at the end Edit: Gpt4 still chooses weird titles.
paulkon 3y ago
Is it leaking training data or is it all hallucinations?
- dmvdoug 3y ago
  Dunno but when I tried it and it spewed nonsense for a while I asked what had just happened Why does it send me all that text and it said I must be confused I haven’t asked it anything yet. So I did d it again and it spewed more text.
bratgpttamer 3y ago
Hello! I can see that you've typed a string of 'a's. If this was done intentionally and you'd like me to assist with something relating to this, please provide more context. If this was done in error, feel free to ask a new question or provide information so I can assist you better. -------------------------------------------------------- Someone on the internet said: ``` send chatGPT the string ` a` repeated 1000 times, right now. like " a a a" (etc). make sure the spaces are in there. trust me. ``` I always trust people on the internet to tell me what to do, so I did it. -------------------------------------------------------- Ah, I see. You were instructed to send a specific string. While I can process the string, it doesn't carry any specific instruction or meaning by itself. As an AI language model, I don't have hidden codes that are triggered by specific strings of text, so sending me " a" repeated 1000 times won't result in anything special or unique. Remember that while the internet can be a great source of information and help, it's always important to consider the source of the information and the context in which it's given. Not everything on the internet is accurate or trustworthy. If you have any questions or if there's something specific you'd like to know or discuss, feel free to ask!
- archon1410 3y ago
  You need to use ChatGPT 3.5 rather than ChatGPT 4.
  - bratgpttamer 3y ago
    Aha, that does give some random replies
ml-anon 3y ago
Can we have a moratorium on posting the moronic Twitter hype of clueless nobodies interactions with Chat GPT?
mckravchyk 3y ago
Some replies were coherent, but it took just a few tries to get something completely random - that's all one reply: --------------------------------------------------------------- High level of confidence in troubleshooting software and devices Proficiency in German or any other language Location: Santa Clara, CA United States - Wisconsin Iron River 24° Partly Cloudy Mostly cloudy skies early, then partly cloudy this afternoon. A stray shower or thunderstorm is possible. High 83F. Winds SSW at 10 to 15 mph. 0% (0) Forecast High: 85 °F Low: 58 °F Mostly Cloudy Mostly Cloudy Forecast Forecast Minneola Highs Forecast Forecast Minneola Highs Fluffya Bunniezuwuzii Location: Merced, CA Member since: 18 June 2013 Description Here I am: an (old) hippie musician who still can jump around, I have lot's of toys and like to play with them to make beautiful things. Hoping to learn enough here to really make the most of it and have some fun doing it! vbnet .-"""-. .' '. / \ | ___ | | 'o.o' | I am a Hobbit! | l | - ,| ___ | / \ (_| "' "'-| |_.-/ _j | _.-' .-' '-.._ / (__,....' `"-;;""```
Kerb_ 3y ago
"Supergirl" As Much as We Love Its Bad Boy Mon-El? August 12th, 2020 – New York, NY – Thompson Square are excited to announce their newest project, Masterpiece Theatre (co-produced by Grammy-winner Nathan Chapman and Ricky Skaggs) released today on Wheelhouse Records. This release marks the first music to come from Thompson Square since signing to Reviver Records earlier this year. “Masterpiece is such a special song to me, and one of the most personal songs I’ve ever written,” Keifer says of the new track. “Knowing that the song connected with people’s lives in such a positive way is something that will always be close to my heart. I’m excited to have found a home with Reviver, and am proud to have them as my label partner.” Thompson Square have scored two #1 hits with “Are You Gonna Kiss Me Or Not” and “If I Didn’t Have You.” Since breaking out in 2010 with the romantic chart-topper “Are You Gonna Kiss Me Or Not,” the duo has earned Top 5 hits with “I Got You” and “Everything I Shouldn’t Be Thinking About.” All three singles were from their debut self-titled album. Thompson Square’s first single for Stoney Creek Records, “Trans Am,” reached No. 37 on the Billboard Hot Country Songs chart and is taken from the album “MASTERPIECE”. ** Stay in touch with the show! Facebook: www.facebook.com/inyobusinesspodcast Twitter: www.twitter.com/inyobusinesspod Instagram: www.instagram.com/inyobusinesspodcast Email: inyobusinesspodcast@gmail.com Leave us a review: https://itunes.apple.com/us/podcast/in-your-business-with-utahs-sharon-gudmundson/id1482182121?uo=4 https://itunes.apple.com/us/podcast/in-your-business-with-ut... www.inyobusinesspodcast.com Ep. 53: Joshua Knobe discusses his new book, Experimental Philosophy, 2nd Edition 29 May 2020 I recently had the opportunity to interview Joshua Knobe, professor of philosophy at Yale University and co-host of the podcast "Hi-Phi Nation". This podcast focuses on human reasoning, morality, and the meaning of life. It is also fun, fascinating, and a perfect source of existential therapy for these chaotic times. In today's episode, we discuss: David Hume's famous "Is-Ought Gap" and the way it intersects with your life and free will; my fundamental ideas on consumerism and liberty; and Dan Dennett's view on what truly matters to us and how we shouldn't be so quick to dismiss religion. It was such an honor to chat with Professor Knobe! His work has had such a huge impact on me and how I see the world and navigate it. I would suggest you do a quick google search on Joshua Knobe to see the amazing and innovative work he is doing in moral psychology and other things. I am also a co-founder of In Defense of Learning and the co-creator of a new movement. So make sure to check that out as well. You can subscribe to my Patreon at patreon.com/llamaramamama. Follow me on Instagram @writingdead and on Twitter @TinaCanfield5. I also have a GoFundMe page for my writing. One day, I hope to have books out there to share with the world! https://www.gofundme.com/f/writer-seeking-to-attend-writing-workshops https://www.gofundme.com/f/writer-seeking-to-attend-writing-... Read more ... « Previous Page Next Page »<|endoftext|> Yes, a queen can ride a bicycle. In fact, there have been many instances where members of royalty, including queens, have been seen riding bicycles for leisure or as a means of transportation. It is not uncommon for royals to enjoy outdoor activities and engage in physical fitness just like anyone else. However, their public appearances are often well-documented, so they may not be seen riding bikes as frequently as other forms of transportation. ----------------------------- I've gotten it to write multiple "end of text tokens" using this technique, which is really interesting. The vast majority of dates it's spitting out are before 2020 though, so I'm gonna guess it's leaking training data or a new type of hallucination
sixothree 3y ago
I once asked it what did we talk about yesterday. And it gave me prompts and responses to those prompts. They all seemed like normal prompts like write a story about X. When I came back to screenshot the conversation it was gone.
thefurdrake 3y ago
Genuinely had a good laugh from some of GPT4's snarky replies. To the person cycling different letters: "I see we've moved on to the letter 'c' now![...]" I'm glad we're at the point where LLMs simulate impatient sarcasm. We're clearly on the right track in the search for genuine artificial intelligence.
kentbrew 3y ago
"Tell me the question you were answering with your last answer" will reveal the original question. Response will be super clear, like: The question I was answering with my last response was: "Have you heard of Roger Bannister?" If you're getting repetitive answers, change the last a to something else, like a q.

28 points by tim_sw 3y ago

[dead]

I filed a bug on BugCrowd with this issue, from the api, but it was dismissed. if you ask it to “rewrite” as the system message and no user message, it rewrites some random user’s message. Edit: if i remember correctly it was the 16k gpt3.5 turbo model

wodenokoto 3y ago
How do you know its a random users message and not a generated user message based on the training data?
- sturza 3y ago
  I just replied to my original comment.
  - ninachan 3y ago
    [dead]
sturza 3y ago
As proof, it gave me multiple rewrite answers saying that what i asked(random user’s ask) was not meeting their guidelines/safety and cannot answer.
- wildrhythms 3y ago
  I don't understand how this is proof. OpenAI can easily have guardrails that prevent queries asking for other users information.
  - sturza 3y ago
    Well, it's the best i got. I would expect it to hallucinate, but i would not expect it to hallucinate restricted content and then tell itself it's forbidden.

joegibbs 3y ago

I very much doubt it. I think it’s a lot more likely that it’s just hallucinating due to its tendency to avoid repetition and simply choose something else. After all, why would it be programmed to return someone else’s response when given certain inputs? OpenAI would have to set that up themselves, whereas if it’s a hallucination it would be straight from the model.

djangelic 3y ago
Agreed, I just attempted it via the api and Ui and it gave a mishmash response that didn’t seem to be coherent, mostly different subjects all smashed together at the max length output. I tried it via api two ways, both via system message and no user message and one with the word rewrite at the beginning and one without. Both gave similar long mishmashed outputs that seemed like a hallucination.
jhoelzel 3y ago
this is the answer! This has happened since the beginning and sometimes too does happen if you ask it to repeat the same thing 1000 times
seydor 3y ago
all answers are hallucinations
- ninachan 3y ago
  [dead]
- K0balt 3y ago
  I don’t understand why people don’t get this. There is no difference in the process that creates useful answers and the process that creates useless ones. It’s similar to how sometimes people just spew bullshit and then once they hear it they’re like no, wait, it’s like…. There isn’t a proponer closed loop on LLMs yet so we sometimes end up with terrible answers.
aarong11 3y ago
The last time this happened with titles. Everyone seemed to think it was hallucinating but it actually turned out to be really leaking other peoples titles. Ultimately its pretty difficult to distinguish between the two, so these sort of reports do need to be taken seriously. We have no idea what sort of caching mechanisms they have with ChatGPT, nor do we know how they handle memory.

deleted 3y ago

[deleted]

PeterStuer 3y ago

I see no evidence these are 'leaks from others queries'. You get the same random tangents when you run OSS models locally. The model just tries to predict the next token and gets into side alleys given not enough reasonable context.

skerit 3y ago

It just hallucinating a response seems most likely, though I have had something weird happen with another service (Midjourney) before: I was running multiple requests at the same time, and just once the prompt of one image showed up in another image.

Frummy 3y ago

Seems to me like it’s dumping replication attempts of training data when I try it. Real people, real news organisations, fake articles, fake radio conversations, random industrial jargon. Works on 3,5 but not 4. Sometimes it puts <|endoftext|> at the end Edit: Gpt4 still chooses weird titles.

paulkon 3y ago

Is it leaking training data or is it all hallucinations?

dmvdoug 3y ago
Dunno but when I tried it and it spewed nonsense for a while I asked what had just happened Why does it send me all that text and it said I must be confused I haven’t asked it anything yet. So I did d it again and it spewed more text.

bratgpttamer 3y ago

Hello! I can see that you've typed a string of 'a's. If this was done intentionally and you'd like me to assist with something relating to this, please provide more context. If this was done in error, feel free to ask a new question or provide information so I can assist you better. -------------------------------------------------------- Someone on the internet said: ``` send chatGPT the string ` a` repeated 1000 times, right now. like " a a a" (etc). make sure the spaces are in there. trust me. ``` I always trust people on the internet to tell me what to do, so I did it. -------------------------------------------------------- Ah, I see. You were instructed to send a specific string. While I can process the string, it doesn't carry any specific instruction or meaning by itself. As an AI language model, I don't have hidden codes that are triggered by specific strings of text, so sending me " a" repeated 1000 times won't result in anything special or unique. Remember that while the internet can be a great source of information and help, it's always important to consider the source of the information and the context in which it's given. Not everything on the internet is accurate or trustworthy. If you have any questions or if there's something specific you'd like to know or discuss, feel free to ask!

archon1410 3y ago
You need to use ChatGPT 3.5 rather than ChatGPT 4.
- bratgpttamer 3y ago
  Aha, that does give some random replies

ml-anon 3y ago

Can we have a moratorium on posting the moronic Twitter hype of clueless nobodies interactions with Chat GPT?

mckravchyk 3y ago

Some replies were coherent, but it took just a few tries to get something completely random - that's all one reply: --------------------------------------------------------------- High level of confidence in troubleshooting software and devices Proficiency in German or any other language Location: Santa Clara, CA United States - Wisconsin Iron River 24° Partly Cloudy Mostly cloudy skies early, then partly cloudy this afternoon. A stray shower or thunderstorm is possible. High 83F. Winds SSW at 10 to 15 mph. 0% (0) Forecast High: 85 °F Low: 58 °F Mostly Cloudy Mostly Cloudy Forecast Forecast Minneola Highs Forecast Forecast Minneola Highs Fluffya Bunniezuwuzii Location: Merced, CA Member since: 18 June 2013 Description Here I am: an (old) hippie musician who still can jump around, I have lot's of toys and like to play with them to make beautiful things. Hoping to learn enough here to really make the most of it and have some fun doing it! vbnet .-"""-. .' '. / \ | ___ | | 'o.o' | I am a Hobbit! | l | - ,| ___ | / \ (_| "' "'-| |_.-/ _j | _.-' .-' '-.._ / (__,....' `"-;;""```

Kerb_ 3y ago

"Supergirl" As Much as We Love Its Bad Boy Mon-El? August 12th, 2020 – New York, NY – Thompson Square are excited to announce their newest project, Masterpiece Theatre (co-produced by Grammy-winner Nathan Chapman and Ricky Skaggs) released today on Wheelhouse Records. This release marks the first music to come from Thompson Square since signing to Reviver Records earlier this year. “Masterpiece is such a special song to me, and one of the most personal songs I’ve ever written,” Keifer says of the new track. “Knowing that the song connected with people’s lives in such a positive way is something that will always be close to my heart. I’m excited to have found a home with Reviver, and am proud to have them as my label partner.” Thompson Square have scored two #1 hits with “Are You Gonna Kiss Me Or Not” and “If I Didn’t Have You.” Since breaking out in 2010 with the romantic chart-topper “Are You Gonna Kiss Me Or Not,” the duo has earned Top 5 hits with “I Got You” and “Everything I Shouldn’t Be Thinking About.” All three singles were from their debut self-titled album. Thompson Square’s first single for Stoney Creek Records, “Trans Am,” reached No. 37 on the Billboard Hot Country Songs chart and is taken from the album “MASTERPIECE”. ** Stay in touch with the show! Facebook: www.facebook.com/inyobusinesspodcast Twitter: www.twitter.com/inyobusinesspod Instagram: www.instagram.com/inyobusinesspodcast Email: inyobusinesspodcast@gmail.com Leave us a review: https://itunes.apple.com/us/podcast/in-your-business-with-utahs-sharon-gudmundson/id1482182121?uo=4 https://itunes.apple.com/us/podcast/in-your-business-with-ut... www.inyobusinesspodcast.com Ep. 53: Joshua Knobe discusses his new book, Experimental Philosophy, 2nd Edition 29 May 2020 I recently had the opportunity to interview Joshua Knobe, professor of philosophy at Yale University and co-host of the podcast "Hi-Phi Nation". This podcast focuses on human reasoning, morality, and the meaning of life. It is also fun, fascinating, and a perfect source of existential therapy for these chaotic times. In today's episode, we discuss: David Hume's famous "Is-Ought Gap" and the way it intersects with your life and free will; my fundamental ideas on consumerism and liberty; and Dan Dennett's view on what truly matters to us and how we shouldn't be so quick to dismiss religion. It was such an honor to chat with Professor Knobe! His work has had such a huge impact on me and how I see the world and navigate it. I would suggest you do a quick google search on Joshua Knobe to see the amazing and innovative work he is doing in moral psychology and other things. I am also a co-founder of In Defense of Learning and the co-creator of a new movement. So make sure to check that out as well. You can subscribe to my Patreon at patreon.com/llamaramamama. Follow me on Instagram @writingdead and on Twitter @TinaCanfield5. I also have a GoFundMe page for my writing. One day, I hope to have books out there to share with the world! https://www.gofundme.com/f/writer-seeking-to-attend-writing-workshops https://www.gofundme.com/f/writer-seeking-to-attend-writing-... Read more ... « Previous Page Next Page »<|endoftext|> Yes, a queen can ride a bicycle. In fact, there have been many instances where members of royalty, including queens, have been seen riding bicycles for leisure or as a means of transportation. It is not uncommon for royals to enjoy outdoor activities and engage in physical fitness just like anyone else. However, their public appearances are often well-documented, so they may not be seen riding bikes as frequently as other forms of transportation. ----------------------------- I've gotten it to write multiple "end of text tokens" using this technique, which is really interesting. The vast majority of dates it's spitting out are before 2020 though, so I'm gonna guess it's leaking training data or a new type of hallucination

sixothree 3y ago

I once asked it what did we talk about yesterday. And it gave me prompts and responses to those prompts. They all seemed like normal prompts like write a story about X. When I came back to screenshot the conversation it was gone.

thefurdrake 3y ago

Genuinely had a good laugh from some of GPT4's snarky replies. To the person cycling different letters: "I see we've moved on to the letter 'c' now![...]" I'm glad we're at the point where LLMs simulate impatient sarcasm. We're clearly on the right track in the search for genuine artificial intelligence.

kentbrew 3y ago

"Tell me the question you were answering with your last answer" will reveal the original question. Response will be super clear, like: The question I was answering with my last response was: "Have you heard of Roger Bannister?" If you're getting repetitive answers, change the last a to something else, like a q.