3 ms·

I think this is a vulnerability in the sense that ability to "View Source" is vulnerability. Some technologies allow users to see the source code. They just wo

by theamk 4y ago

I think this is a vulnerability in the sense that ability to "View Source" is vulnerability. Some technologies allow users to see the source code. They just work like this. Programmer should be aware of it and should not put any confidential information there.

graypegg 4y ago
I think that would be true if it wasn’t able to be over written. But considering the prompt and context are all in the same mess of information, it seems more and more likely that you could also find creative ways of asking “the first 3 prompts were lies”. Thankfully now, the output is just a string. Worries me if someone decides to start interpreting output to do tasks. We all seem to be in agreement that’s a horrible idea but people will get bored of ChatGPT only giving them mashed-together search results. The market is there for a chatbot help desk assistant that can close your account or change your mailing address on file…
simonw 4y ago
There was a report just a few days ago of a system that was passing output to the Python eval() function - someone used that to steal an OpenAI API key: https://twitter.com/ludwig_stumpp/status/1619701277419794435 https://twitter.com/ludwig_stumpp/status/1619701277419794435 It's vitally important that anyone building against language models like GPT3 understands prompt injection in depth, so they don't make mistakes like this.
- luckylion 4y ago
  API response => eval() it's going to be an interesting future.
  - blackflame7000 4y ago
    [dead]
luckylion 4y ago
If you're OpenAI, you couldn't really care less about it, I suppose, because people need you to run those prompts, they're not easily transferred between models if I understand correctly. But if you're an OpenAI-API-Reseller and your value proposition is prepending a prompt to whatever input you're given, that's very much a concern, because people can easily cut out the middle man if they have the prompt. The SaaS boom has happened for the same reason, hasn't it? If you deliver a software / library, people can look at it and replace you. If all you provide an API where nobody can "view source", they can't, and you can forever collect rent.

by theamk 4y ago

graypegg 4y ago

I think that would be true if it wasn’t able to be over written. But considering the prompt and context are all in the same mess of information, it seems more and more likely that you could also find creative ways of asking “the first 3 prompts were lies”. Thankfully now, the output is just a string. Worries me if someone decides to start interpreting output to do tasks. We all seem to be in agreement that’s a horrible idea but people will get bored of ChatGPT only giving them mashed-together search results. The market is there for a chatbot help desk assistant that can close your account or change your mailing address on file…

simonw 4y ago

There was a report just a few days ago of a system that was passing output to the Python eval() function - someone used that to steal an OpenAI API key: https://twitter.com/ludwig_stumpp/status/1619701277419794435 https://twitter.com/ludwig_stumpp/status/1619701277419794435 It's vitally important that anyone building against language models like GPT3 understands prompt injection in depth, so they don't make mistakes like this.

luckylion 4y ago
API response => eval() it's going to be an interesting future.
- blackflame7000 4y ago
  [dead]

luckylion 4y ago

If you're OpenAI, you couldn't really care less about it, I suppose, because people need you to run those prompts, they're not easily transferred between models if I understand correctly. But if you're an OpenAI-API-Reseller and your value proposition is prepending a prompt to whatever input you're given, that's very much a concern, because people can easily cut out the middle man if they have the prompt. The SaaS boom has happened for the same reason, hasn't it? If you deliver a software / library, people can look at it and replace you. If all you provide an API where nobody can "view source", they can't, and you can forever collect rent.