6 ms·

I understand doing this from a red-team perspective, but what is the point in actual usage? I see GPT as a tool to make "my life easier", help me with tedious

by tagyro 3y ago

I understand doing this from a red-team perspective, but what is the point in actual usage? I see GPT as a tool to make "my life easier", help me with tedious stuff, maybe point out some dark corners etc Why would I go and try to break my hammer when I need it to actually put the nails in? Will there be users doing that? Sure! Will I be doing that? Not really, I have real issues to take care of and GPT helps do that. Maybe I'm missing something, but this is more like sql-injection with php/mysql - yes, it's an issue and yes, we need to be aware of it. Is it a "nuclear bomb"-type issue? I would say no, it isn't. #off-topic: I counted at least 4 links (in the past 2 weeks!) to Simon's website for articles spreading basically FUD around GPT. Yes, it's a new technology and you're scared - we're all a bit cautious, but let's not throw out the baby with the bathwater, shall we?

wll 3y ago
GPT is a marvel and as far as I can see those who are working with it are all in awe and I don’t think Simon himself has ever said otherwise, unless I misread you and you meant other people. That would be understandable though as it is easy to misunderstand and misalign GPT and family’s unbounded potential. The concern is that people building people-facing or people-handling automation will end up putting their abstractions on the road before inventing seatbelts — and waiting for a Volvo to pop up out of mushrooms isn’t going to be enough in case haste leads to nuclear waste. It is a policy issue as much as it is an experience issue. What we don’t want is policymakers breaking the hammers galvanized by such an event. And with Hinton and colleagues strongly in favor of pauses and whatnot, we absolutely don’t want to give them another argument.
- tagyro 3y ago
  Disclosure: I built an app on top of OpenAI's API ...and my last worry is people subverting the prompt to ask "stupid" questions - I send the prompts to a moderation API and simply block invalid requests. Folks, we have solutions for these problems and it's always going to be a cat and mouse game. "There is no such thing as perfection" (tm, copyright and all, if you use this quote you have to pay me a gazzilion money)
  - danShumway 3y ago
    If the only thing you're building is a chat app, and the only thing you're worried about is it swearing at the user, then sure, GPT is great for that. If you're building a Twitch bot, if you're building this into a game or making a quick display or something, then yeah, go wild. But people are wiring GPT up to real-world applications beyond just content generation. Summarizing articles, invoking APIs, managing events, filtering candidates for job searches, etc... Greshake wrote a good article summarizing some of the applications being built on top of LLMs right now: https://kai-greshake.de/posts/in-escalating-order-of-stupidity/ https://kai-greshake.de/posts/in-escalating-order-of-stupidi... Prompt injection really heckin matters for those applications, and we do not have solutions to the problem. Perfection is the enemy of the good, but sometimes terrible is also the enemy of the good. It's not really chasing after perfection to say "maybe I don't want my web browser to have the potential to start trying to phish me every time it looks at a web page." That's just trying to get basic security around a feature.
themodelplumber 3y ago
From the article: > This is crucially important. This is not an attack against the AI models themselves. This is an attack against the stuff which developers like us are building on top of them. That seems more like a community service, really. If you're building on the platform it's probably a relief to know somebody's working on this stuff before it impacts your customers.
danShumway 3y ago
> Why would I go and try to break my hammer when I need it to actually put the nails in? You're confusing prompt injection with jailbreaking. The danger of prompt injection is that when your GPT tool processes 3rd-party text, someone else reprograms its instructions and causes it to attack you or abuse the privileges you've given it in some way. > spreading basically FUD around GPT My impression is that Simon is extremely bullish on GPT and regularly writes positively about it. The one negative that Simon (very correctly) points out is that GPT is vulnerable to prompt injection and that this is a very serious problem with no known solution that limits applications. If that counts as FUD, then... I don't know what to say to that. If anything, prompt injection isn't getting hammered hard enough. Look at the replies to this article; they're filled with people asking the same questions that have been answered over and over again, even questions that are answered in the linked presentation itself. People don't understand the risks, and they don't understand the scope of the problem, and given that we're seeing LLMs wired up to military applications now, it seems worthwhile to try and educate people in the tech sector about the risks.
- tagyro 3y ago
  People, in general, are stupid (me included). Do we do stupid stuff? Every fu*ing day! And then again! Prompt injection is more like a "cheat" code - yeah, you can "noclip" through walls, but you're not going to get the ESL championship.
  - danShumway 3y ago
    > yeah, you can "noclip" through walls, but you're not going to get the ESL championship. I don't understand what you mean by this. LLMs are literally being wired into military applications right now. They're being wired into workflows where if something falls over and goes terribly wrong, people actually die. If somebody hacks a Twitch bot, who cares? The problem is people are building stuff that's a lot more powerful than Twitch bots.
    - tagyro 3y ago
      > LLMs are literally being wired into military applications right now. They're being wired into workflows where if something falls over and goes terribly wrong, people actually die. Do you have any proof to back this claim?
spacebanana7 3y ago
> Is it a "nuclear bomb"-type issue? Given the allure of using AI in the military for unmanned systems it’s not that far off. With a lesser danger level, similar adversarial dynamics exist in other places where AI might be useful. E.g dating, fraud detection, recruitment
- tagyro 3y ago
  Please don't spread more FUD, no-one is using OpenAI's GPT in the military. Is GPT perfect? Hell, no? Does it have biases? F*c yeah, the same ones of the humans that programmed it.
  - danShumway 3y ago
    Both Palantir and Donovan are looking to use LLMs in the military: https://www.palantir.com/platforms/aip/ https://www.palantir.com/platforms/aip/, https://scale.com/donovan https://scale.com/donovan This might be technically correct, in the sense that I think these companies have their own LLMs they're pushing? They're not literally using OpenAI's GPT model. But all LLMs are vulnerable to this, so it doesn't practically matter if they're using specifically GPT vs something in-house, the threat model is the same.
raincole 3y ago
> Maybe I'm missing something, but this is more like sql-injection with php/mysql - yes, it's an issue and yes, we need to be aware of it. It's like an SQL-injection without a commonly accepted solution. And that's why it's a serious issue. I know how to handle potential SQL-injection now. And if I don't I can just google it. But were I that informed when I wrote the first line of code in my life? Of course not. Now the whole world is just as ill-informed about prompt injection as I were about SQL-injection by the time.
simonw 3y ago
Here's why I think this is a big problem for a lot of the things people want to build with LLMs: https://simonwillison.net/2023/Apr/14/worst-that-can-happen/ https://simonwillison.net/2023/Apr/14/worst-that-can-happen/ I suggest reading my blog closer if you think I'm trying to scare people off GPT. Take a look at these series of posts for example: https://simonwillison.net/series/using-chatgpt/ https://simonwillison.net/series/using-chatgpt/ - about constructive ways to use ChatGPT https://simonwillison.net/series/llms-on-personal-devices/ https://simonwillison.net/series/llms-on-personal-devices/ - tracking the development of LLMs that can run on personal devices See also these tags: - llms: https://simonwillison.net/tags/llms/ https://simonwillison.net/tags/llms/ - promptengineering: https://simonwillison.net/tags/promptengineering/ https://simonwillison.net/tags/promptengineering/ You've also seen a bunch of my content on Hacker News because I'm one of the only people writing about it - I'd very much like not to be!
- going_ham 3y ago
  > You've also seen a bunch of my content on Hacker News because I'm one of the only people writing about it - if very much like not to be! With all due respect, I would also like to market someone else who has also been posting similar content, but for some reason those posts never make it to the top. If you don't believe me, you can check the following submissions: [0]: https://news.ycombinator.com/item?id=35745457 https://news.ycombinator.com/item?id=35745457 [1]: https://news.ycombinator.com/item?id=35915140 https://news.ycombinator.com/item?id=35915140 They have been consistently putting the risks of LLMs. Thanks for spreading the information though. Cheers.
  - danShumway 3y ago
    These posts are coming out of the same team that popularized the term "indirect prompt injection" around Bing chat, which was a pretty big wake-up call to me about the potential dangers. Definitely worth following.

by tagyro 3y ago

wll 3y ago

GPT is a marvel and as far as I can see those who are working with it are all in awe and I don’t think Simon himself has ever said otherwise, unless I misread you and you meant other people. That would be understandable though as it is easy to misunderstand and misalign GPT and family’s unbounded potential. The concern is that people building people-facing or people-handling automation will end up putting their abstractions on the road before inventing seatbelts — and waiting for a Volvo to pop up out of mushrooms isn’t going to be enough in case haste leads to nuclear waste. It is a policy issue as much as it is an experience issue. What we don’t want is policymakers breaking the hammers galvanized by such an event. And with Hinton and colleagues strongly in favor of pauses and whatnot, we absolutely don’t want to give them another argument.

tagyro 3y ago
Disclosure: I built an app on top of OpenAI's API ...and my last worry is people subverting the prompt to ask "stupid" questions - I send the prompts to a moderation API and simply block invalid requests. Folks, we have solutions for these problems and it's always going to be a cat and mouse game. "There is no such thing as perfection" (tm, copyright and all, if you use this quote you have to pay me a gazzilion money)
- danShumway 3y ago
  If the only thing you're building is a chat app, and the only thing you're worried about is it swearing at the user, then sure, GPT is great for that. If you're building a Twitch bot, if you're building this into a game or making a quick display or something, then yeah, go wild. But people are wiring GPT up to real-world applications beyond just content generation. Summarizing articles, invoking APIs, managing events, filtering candidates for job searches, etc... Greshake wrote a good article summarizing some of the applications being built on top of LLMs right now: https://kai-greshake.de/posts/in-escalating-order-of-stupidity/ https://kai-greshake.de/posts/in-escalating-order-of-stupidi... Prompt injection really heckin matters for those applications, and we do not have solutions to the problem. Perfection is the enemy of the good, but sometimes terrible is also the enemy of the good. It's not really chasing after perfection to say "maybe I don't want my web browser to have the potential to start trying to phish me every time it looks at a web page." That's just trying to get basic security around a feature.

themodelplumber 3y ago

From the article: > This is crucially important. This is not an attack against the AI models themselves. This is an attack against the stuff which developers like us are building on top of them. That seems more like a community service, really. If you're building on the platform it's probably a relief to know somebody's working on this stuff before it impacts your customers.

danShumway 3y ago

> Why would I go and try to break my hammer when I need it to actually put the nails in? You're confusing prompt injection with jailbreaking. The danger of prompt injection is that when your GPT tool processes 3rd-party text, someone else reprograms its instructions and causes it to attack you or abuse the privileges you've given it in some way. > spreading basically FUD around GPT My impression is that Simon is extremely bullish on GPT and regularly writes positively about it. The one negative that Simon (very correctly) points out is that GPT is vulnerable to prompt injection and that this is a very serious problem with no known solution that limits applications. If that counts as FUD, then... I don't know what to say to that. If anything, prompt injection isn't getting hammered hard enough. Look at the replies to this article; they're filled with people asking the same questions that have been answered over and over again, even questions that are answered in the linked presentation itself. People don't understand the risks, and they don't understand the scope of the problem, and given that we're seeing LLMs wired up to military applications now, it seems worthwhile to try and educate people in the tech sector about the risks.

tagyro 3y ago
People, in general, are stupid (me included). Do we do stupid stuff? Every fu*ing day! And then again! Prompt injection is more like a "cheat" code - yeah, you can "noclip" through walls, but you're not going to get the ESL championship.
- danShumway 3y ago
  > yeah, you can "noclip" through walls, but you're not going to get the ESL championship. I don't understand what you mean by this. LLMs are literally being wired into military applications right now. They're being wired into workflows where if something falls over and goes terribly wrong, people actually die. If somebody hacks a Twitch bot, who cares? The problem is people are building stuff that's a lot more powerful than Twitch bots.
  - tagyro 3y ago
    > LLMs are literally being wired into military applications right now. They're being wired into workflows where if something falls over and goes terribly wrong, people actually die. Do you have any proof to back this claim?

spacebanana7 3y ago

> Is it a "nuclear bomb"-type issue? Given the allure of using AI in the military for unmanned systems it’s not that far off. With a lesser danger level, similar adversarial dynamics exist in other places where AI might be useful. E.g dating, fraud detection, recruitment

tagyro 3y ago
Please don't spread more FUD, no-one is using OpenAI's GPT in the military. Is GPT perfect? Hell, no? Does it have biases? F*c yeah, the same ones of the humans that programmed it.
- danShumway 3y ago
  Both Palantir and Donovan are looking to use LLMs in the military: https://www.palantir.com/platforms/aip/ https://www.palantir.com/platforms/aip/, https://scale.com/donovan https://scale.com/donovan This might be technically correct, in the sense that I think these companies have their own LLMs they're pushing? They're not literally using OpenAI's GPT model. But all LLMs are vulnerable to this, so it doesn't practically matter if they're using specifically GPT vs something in-house, the threat model is the same.

raincole 3y ago

> Maybe I'm missing something, but this is more like sql-injection with php/mysql - yes, it's an issue and yes, we need to be aware of it. It's like an SQL-injection without a commonly accepted solution. And that's why it's a serious issue. I know how to handle potential SQL-injection now. And if I don't I can just google it. But were I that informed when I wrote the first line of code in my life? Of course not. Now the whole world is just as ill-informed about prompt injection as I were about SQL-injection by the time.

simonw 3y ago

Here's why I think this is a big problem for a lot of the things people want to build with LLMs: https://simonwillison.net/2023/Apr/14/worst-that-can-happen/ https://simonwillison.net/2023/Apr/14/worst-that-can-happen/ I suggest reading my blog closer if you think I'm trying to scare people off GPT. Take a look at these series of posts for example: https://simonwillison.net/series/using-chatgpt/ https://simonwillison.net/series/using-chatgpt/ - about constructive ways to use ChatGPT https://simonwillison.net/series/llms-on-personal-devices/ https://simonwillison.net/series/llms-on-personal-devices/ - tracking the development of LLMs that can run on personal devices See also these tags: - llms: https://simonwillison.net/tags/llms/ https://simonwillison.net/tags/llms/ - promptengineering: https://simonwillison.net/tags/promptengineering/ https://simonwillison.net/tags/promptengineering/ You've also seen a bunch of my content on Hacker News because I'm one of the only people writing about it - I'd very much like not to be!

going_ham 3y ago
> You've also seen a bunch of my content on Hacker News because I'm one of the only people writing about it - if very much like not to be! With all due respect, I would also like to market someone else who has also been posting similar content, but for some reason those posts never make it to the top. If you don't believe me, you can check the following submissions: [0]: https://news.ycombinator.com/item?id=35745457 https://news.ycombinator.com/item?id=35745457 [1]: https://news.ycombinator.com/item?id=35915140 https://news.ycombinator.com/item?id=35915140 They have been consistently putting the risks of LLMs. Thanks for spreading the information though. Cheers.
- danShumway 3y ago
  These posts are coming out of the same team that popularized the term "indirect prompt injection" around Bing chat, which was a pretty big wake-up call to me about the potential dangers. Definitely worth following.