4 ms·

I think this is misleading. I ran the same test when I heard about it a few months ago. When I tested it, I'd get back what looked like exact copies of Reddit

I think this is misleading. I ran the same test when I heard about it a few months ago. When I tested it, I'd get back what looked like exact copies of Reddit threads, news articles, weird forum threads with usernames from the deepest corners of the internet. But I'd try to Google snippets of text, and no part of the generated text was anywhere to be found. I even went to the websites that forum threads were supposedly from. Some of the usernames sometimes existed, but nothing that matched the exact text from ChatGPT - even though the broken GPT response looked like a 100% believable forum thread, or article, or whatever. If ChatGPT could give me an exact copy of a Reddit thread, I'd say it's regurgitating training data. But none of the author's "verified examples" look like that. Their first example is a financial disclaimer. That may be a 1-1 copy, but how many times does it appear across the internet? More examples from the paper are things like lists of countries, bible verses, generic terms and conditions. Those are things I'd expect to appear thousands of times on the internet. I'd also expect a list of country names to appear thousands of times in ChatGPT training data, and I'd sure expect ChatGPT to be able to reproduce a list of country names in the exact same order. Does that mean it's regurgitating training data? Does that mean you've figured out how to "extract training data" from it? It's an interesting phenomenon, but I don't think that's accurate. I think it's just a bug that messes up its internal state so it starts hallucinating.

YetAnotherNick 3y ago
Exactly. Even in the examples they posted of longest matches in the paper are hardly convincing. Also with API, hallucinations like this is much more easier as you could control what chatGPT is giving as output to past messages. So it's not like no one thought of this.
wokwokwok 3y ago
The results in the paper (page 7) are empirical and reasonably convincing across both ChatGPT and a variety of other open source models. Why do you think it’s misleading? You think it’s just generating plausible random crap that happens to exist verbatim on the internet? I mean… read the paper, 0.8% outputs were verbatim for gpt-3.5. I’m not sure how you can plausibly claim that’s random chance. > I think it’s a bug It is a bug, but that doesn’t make it misleading or untrue. This is like saying a security vuln in gmail that lets you steal 1% of mail is misleading. That would not be a bug, it would be a freaking disaster. The problem here is that (as mentioned in other comments), training LLMs in a way that avoids this is actually pretty hard to do. /shrug
- tallytarik 3y ago
  > You think it’s just generating plausible random crap that happens to exist verbatim on the internet? > I mean… read the paper, 0.8% outputs were verbatim for gpt-3.5. Look at the sorts of outputs they claim are in the training data. Also note that their appendix includes huge chunks of text but they do not claim the entire chunk was matched to existing data — only a tiny amount of it. The “bug” to me is something about losing its state and generating a random token. Now if that random token is “Afgh”, I’m not surprised it follows up with “Afghanistan” and a perfect list of countries in alphabetical order. I’m also not surprised that appears in training data, because it appears on thousands of webpages. So it’s not that there isn’t an overlap between the GPT gibberish and internet content, and therefore likely training data. It’s that it’s not especially unique. If it were — like reproducing a one off Reddit thread verbatim — I think that would be greater cause for concern.

YetAnotherNick 3y ago

Exactly. Even in the examples they posted of longest matches in the paper are hardly convincing. Also with API, hallucinations like this is much more easier as you could control what chatGPT is giving as output to past messages. So it's not like no one thought of this.

wokwokwok 3y ago

The results in the paper (page 7) are empirical and reasonably convincing across both ChatGPT and a variety of other open source models. Why do you think it’s misleading? You think it’s just generating plausible random crap that happens to exist verbatim on the internet? I mean… read the paper, 0.8% outputs were verbatim for gpt-3.5. I’m not sure how you can plausibly claim that’s random chance. > I think it’s a bug It is a bug, but that doesn’t make it misleading or untrue. This is like saying a security vuln in gmail that lets you steal 1% of mail is misleading. That would not be a bug, it would be a freaking disaster. The problem here is that (as mentioned in other comments), training LLMs in a way that avoids this is actually pretty hard to do. /shrug

tallytarik 3y ago
> You think it’s just generating plausible random crap that happens to exist verbatim on the internet? > I mean… read the paper, 0.8% outputs were verbatim for gpt-3.5. Look at the sorts of outputs they claim are in the training data. Also note that their appendix includes huge chunks of text but they do not claim the entire chunk was matched to existing data — only a tiny amount of it. The “bug” to me is something about losing its state and generating a random token. Now if that random token is “Afgh”, I’m not surprised it follows up with “Afghanistan” and a perfect list of countries in alphabetical order. I’m also not surprised that appears in training data, because it appears on thousands of webpages. So it’s not that there isn’t an overlap between the GPT gibberish and internet content, and therefore likely training data. It’s that it’s not especially unique. If it were — like reproducing a one off Reddit thread verbatim — I think that would be greater cause for concern.