5 ms·

What is entrapment via prompt injection?

seanmcdirmid 3y ago
You give pieces of the article during your chat session giving enough context for the LLM to put it together when you ask for it. The other one is just being confused when Bing, who has some sort of deal with the publishers, includes the article in the prompt when you casually use the system to ask for its first ten lines, and get it. It isn’t baked into the model, the attention aspect is just really leaned into. Mostly it’s just people not understanding how LLMs work, and assuming everything is coming from the model when they don’t realize it’s all coming from information in the prompt, and the model is just really good at getting things from the prompt. Again, courts will sort those claims out quickly.
__loam 3y ago
Copyright infringement.
gamblor956 3y ago
There's no such thing as "entrapment via prompt injection" since it still requires the LLM to have the underlying article in its training set. And that is precisely the problem: that the underlying article is in the training set and in the body of reproducible output without permission. This requires an act of copying, and was done without permission, so it's a copyright violation unless a fair use exception applies. (Courts haven't yet ruled on that; it's likely that the NYT case is the case that will ultimately lead to a SCOTUS ruling resolving the issue one way or the other.)
- seanmcdirmid 3y ago
  This is incorrect. The model does not need to be trained on an article for the LLM to regurgitate it if the article is fed into it (even in small pieces) in the prompt. Again, quit misunderstanding the tech or we can’t argue about it in an intelligent way.
  - gamblor956 3y ago
    OpenAI states directly in their response the NYT lawsuit that an article needs to be part of its training set in order to be regurgitated, and indeed they note that "memorization" can be a big problem if an article shows up multiple times in its training set. They further admit that their LLM is (or was) capable of replicating content from its training set wholesale and that specific safeguards were put in place after complaints to limit that. OpenAI accuses the NYT of using "manipulated prompts" to trigger regurgitation of NYT artices. But so what? For legal purposes, and more relevantly for copyright purposes, it's irrelevant that the LLM needed large prompts to "regurgitate" the entire article. What matters is that, given a prompt that was less than the full article the LLM was able to replicate the entire article. In other words, it distributed a copy of an NYT article without the right to do so. And that is literally what they are being sued for. Whether the use of the article in the training set, or the reproduction of the article in the LLM responses, constitute fair use is still to be decided (and those answers will determine who wins this case.) But it's the legal facts that will determine the winner, not the technical implementation, because those details are irrelevant for purposes of copyright law.
    - seanmcdirmid 3y ago
      > OpenAI states directly in their response the NYT lawsuit that an article needs to be part of its training set in order to be regurgitated, and indeed they note that "memorization" can be a big problem if an article shows up multiple times in its training set. They further admit that their LLM is (or was) capable of replicating content from its training set wholesale and that specific safeguards were put in place after complaints to limit that. Nope, that's not what they claim, and you are intentionally misrepresenting them. See https://openai.com/blog/openai-and-journalism https://openai.com/blog/openai-and-journalism: > Memorization is a rare failure of the learning process that we are continually making progress on, but it’s more common when particular content appears more than once in training data, like if pieces of it appear on lots of different public websites. So we have measures in place to limit inadvertent memorization and prevent regurgitation in model outputs. ... > We had explained to The New York Times that, like any single source, their content didn't meaningfully contribute to the training of our existing models and also wouldn't be sufficiently impactful for future training. Their lawsuit on December 27—which we learned about by reading The New York Times—came as a surprise and disappointment to us. > Along the way, they had mentioned seeing some regurgitation of their content but repeatedly refused to share any examples, despite our commitment to investigate and fix any issues. We’ve demonstrated how seriously we treat this as a priority, such as in July when we took down a ChatGPT feature immediately after we learned it could reproduce real-time content in unintended ways. > Interestingly, the regurgitations The New York Times induced appear to be from years-old articles that have proliferated on multiple third-party websites. It seems they intentionally manipulated prompts, often including lengthy excerpts of articles, in order to get our model to regurgitate. Even when using such prompts, our models don’t typically behave the way The New York Times insinuates, which suggests they either instructed the model to regurgitate or cherry-picked their examples from many attempts. End quote. > But so what? For legal purposes, and more relevantly for copyright purposes, it's irrelevant that the LLM needed large prompts to "regurgitate" the entire article. They used multiple queries to build up a fairly large prompt, it is not not irrelevant at all. > What matters is that, given a prompt that was less than the full article the LLM was able to replicate the entire article. In other words, it distributed a copy of an NYT article without the right to do so. And that is literally what they are being sued for. That is incorrect, they only need to provide enough context to reproduce the article, providing excerpts of the article in their own prompting would immediately raise red flags in a legal context. NyTimes could be found at fault here, because if they are found to have purposely primed the prompt, it won't surprise me if NyTimes winds up paying OpenAI over this. > But it's the legal facts that will determine the winner, not the technical implementation, because those details are irrelevant for purposes of copyright law. In this case, there is no difference, this was resolved long ago in lawsuits against xerox (is xerox liable if you use it to copy a copyrighted book? nope). Just because you get ChatGPT to reproduce an article because you yourself provided excerpts of the article to it doesn't mean it infringed on the content. If anything you did. NyTimes will have to provide their methodology as evidence in a civil trial, and the fact they haven't released it so far, they are definitely not confident of their position and are hoping for a settlement.

You give pieces of the article during your chat session giving enough context for the LLM to put it together when you ask for it. The other one is just being confused when Bing, who has some sort of deal with the publishers, includes the article in the prompt when you casually use the system to ask for its first ten lines, and get it. It isn’t baked into the model, the attention aspect is just really leaned into. Mostly it’s just people not understanding how LLMs work, and assuming everything is coming from the model when they don’t realize it’s all coming from information in the prompt, and the model is just really good at getting things from the prompt. Again, courts will sort those claims out quickly.

__loam 3y ago

gamblor956 3y ago

There's no such thing as "entrapment via prompt injection" since it still requires the LLM to have the underlying article in its training set. And that is precisely the problem: that the underlying article is in the training set and in the body of reproducible output without permission. This requires an act of copying, and was done without permission, so it's a copyright violation unless a fair use exception applies. (Courts haven't yet ruled on that; it's likely that the NYT case is the case that will ultimately lead to a SCOTUS ruling resolving the issue one way or the other.)

seanmcdirmid 3y ago
This is incorrect. The model does not need to be trained on an article for the LLM to regurgitate it if the article is fed into it (even in small pieces) in the prompt. Again, quit misunderstanding the tech or we can’t argue about it in an intelligent way.
- gamblor956 3y ago
  OpenAI states directly in their response the NYT lawsuit that an article needs to be part of its training set in order to be regurgitated, and indeed they note that "memorization" can be a big problem if an article shows up multiple times in its training set. They further admit that their LLM is (or was) capable of replicating content from its training set wholesale and that specific safeguards were put in place after complaints to limit that. OpenAI accuses the NYT of using "manipulated prompts" to trigger regurgitation of NYT artices. But so what? For legal purposes, and more relevantly for copyright purposes, it's irrelevant that the LLM needed large prompts to "regurgitate" the entire article. What matters is that, given a prompt that was less than the full article the LLM was able to replicate the entire article. In other words, it distributed a copy of an NYT article without the right to do so. And that is literally what they are being sued for. Whether the use of the article in the training set, or the reproduction of the article in the LLM responses, constitute fair use is still to be decided (and those answers will determine who wins this case.) But it's the legal facts that will determine the winner, not the technical implementation, because those details are irrelevant for purposes of copyright law.
  - seanmcdirmid 3y ago
    > OpenAI states directly in their response the NYT lawsuit that an article needs to be part of its training set in order to be regurgitated, and indeed they note that "memorization" can be a big problem if an article shows up multiple times in its training set. They further admit that their LLM is (or was) capable of replicating content from its training set wholesale and that specific safeguards were put in place after complaints to limit that. Nope, that's not what they claim, and you are intentionally misrepresenting them. See https://openai.com/blog/openai-and-journalism https://openai.com/blog/openai-and-journalism: > Memorization is a rare failure of the learning process that we are continually making progress on, but it’s more common when particular content appears more than once in training data, like if pieces of it appear on lots of different public websites. So we have measures in place to limit inadvertent memorization and prevent regurgitation in model outputs. ... > We had explained to The New York Times that, like any single source, their content didn't meaningfully contribute to the training of our existing models and also wouldn't be sufficiently impactful for future training. Their lawsuit on December 27—which we learned about by reading The New York Times—came as a surprise and disappointment to us. > Along the way, they had mentioned seeing some regurgitation of their content but repeatedly refused to share any examples, despite our commitment to investigate and fix any issues. We’ve demonstrated how seriously we treat this as a priority, such as in July when we took down a ChatGPT feature immediately after we learned it could reproduce real-time content in unintended ways. > Interestingly, the regurgitations The New York Times induced appear to be from years-old articles that have proliferated on multiple third-party websites. It seems they intentionally manipulated prompts, often including lengthy excerpts of articles, in order to get our model to regurgitate. Even when using such prompts, our models don’t typically behave the way The New York Times insinuates, which suggests they either instructed the model to regurgitate or cherry-picked their examples from many attempts. End quote. > But so what? For legal purposes, and more relevantly for copyright purposes, it's irrelevant that the LLM needed large prompts to "regurgitate" the entire article. They used multiple queries to build up a fairly large prompt, it is not not irrelevant at all. > What matters is that, given a prompt that was less than the full article the LLM was able to replicate the entire article. In other words, it distributed a copy of an NYT article without the right to do so. And that is literally what they are being sued for. That is incorrect, they only need to provide enough context to reproduce the article, providing excerpts of the article in their own prompting would immediately raise red flags in a legal context. NyTimes could be found at fault here, because if they are found to have purposely primed the prompt, it won't surprise me if NyTimes winds up paying OpenAI over this. > But it's the legal facts that will determine the winner, not the technical implementation, because those details are irrelevant for purposes of copyright law. In this case, there is no difference, this was resolved long ago in lawsuits against xerox (is xerox liable if you use it to copy a copyrighted book? nope). Just because you get ChatGPT to reproduce an article because you yourself provided excerpts of the article to it doesn't mean it infringed on the content. If anything you did. NyTimes will have to provide their methodology as evidence in a civil trial, and the fact they haven't released it so far, they are definitely not confident of their position and are hoping for a settlement.