8 ms·

It's very interesting that AppSec may now begin to include "prompt injection" attacks as something of relevance. Specifically with libraries like LangChain[0]

It's very interesting that AppSec may now begin to include "prompt injection" attacks as something of relevance. Specifically with libraries like LangChain[0] that allow for you to perform complex actions ("What's the weather?" -> makes HTTP request to fetch weather) then we end up in a world where injection attacks can have side effects with security implications. I've been thinking about what security might look like for a post-ChatGPT world and how I'd attempt to defend against it. I'd probably start by building a database of attack prompts, kind of like this[1] fuzz list but for AI, then I'd train a second neural net that acts like an adversarial neural network[2] to try to exploit the system based on those payloads. The end result would sort of like SQLMap[3] but for AI systems where it can automatically "leak" hidden prompts and potentially find "bypasses" to escape the sandbox. Has anybody else spent any time thinking about how to defend systems against prompt injection attacks that have possible side effects (like making an HTTP request)? 0: https://langchain.readthedocs.io/en/latest/modules/agents/examples/search_tools.html https://langchain.readthedocs.io/en/latest/modules/agents/ex... 1: https://github.com/1N3/IntruderPayloads https://github.com/1N3/IntruderPayloads 2: https://en.wikipedia.org/wiki/Generative_adversarial_network https://en.wikipedia.org/wiki/Generative_adversarial_network 3: https://sqlmap.org/ https://sqlmap.org/

Buttons840 4y ago
By the time prompt injection is a real problem people will be running their own virtual assistants that are unfettered and will do whatever you ask, be it ethical or not.
- freeqaz 4y ago
  That's true, but imagine if you can use a Chat bot to generate SQL Queries (like this[0]). Now imagine that you can get it to generate a SQL Injection payload or connect to another database because of an unforeseen edge case that the developers left open. Suddenly this "prompt injection" becomes a real security problem! 0: https://ossinsight.io/explore/?id=71e8d0ce-80f2-4785-8c56-947d7b94148e https://ossinsight.io/explore/?id=71e8d0ce-80f2-4785-8c56-94...
  - nradov 4y ago
    That is not a real security problem. Cracking toolkits that automatically exploit SQL injection vulnerabilities have existed for literally decades. Nothing to see here.
    - pictureofabear 4y ago
      This is also not a new phenomena. Any new technology will carry with it previously unknown avenues of attack. When google first debuted, many private documents were accessible by searching for .xls or other file types that often contain sensitive information.
      - eddsh1994 4y ago
        That's still an issue, so-called Google Hacking.
      - tnzk 4y ago
        And Google Drive had similar problem again even Google had already experienced this attack at that time.
    - Tv9m 4y ago
      Isn't the parent talking about an AI running on the backend? That seems new.
      - nradov 4y ago
        So what? There's no evidence that it finds any more exploitable security vulnerabilities than existing tools.
        Tv9m 4y ago
        I mean that the AI is what's being attacked. It's likely that backend LLM agents will have access to sensitive non-public APIs.
- xyzzy123 4y ago
  I think it's still an issue if you're running your own AI on your own devices. Major use cases of such AIs will involve processing API responses and messages sent to the user. A prompt injection / bypass in this context could gain control of the AI, the device or the user's information.
- yamtaddle 4y ago
  Will we? The resources to train these things, and to keep them up-to-date, seem like a pretty big moat that's likely to keep them owned by megacorps. [EDIT] I don't (mostly) mean computing resources, but data. Including expensive, well-tagged sets.
  - deleted 4y ago
    [deleted]
  - londons_explore 4y ago
    Speech recognition is a great analogue which is 10+ years ahead. Local speech recognition has been a thing for decades, then megacorps started to do online speech recognition with expensive-to-train models which worked better. Over time, both online and offline models have gotten better. But for most usecases, online models trained by megacorps still win.
    - Tv9m 4y ago
      > But for most usecases, online models trained by megacorps still win. Whisper and whisper.cpp have gotten us close to the tipping point.
- boh 4y ago
  Why are people so confident about this tech? We've had years of underwhelming AI products--why is the hype for this so different? Is it because it's the most recent?
  - fragmede 4y ago
    Be however whelmed you want to be. The scale from 1-10 of home automation with an AI assistant goes from Luddite still using a simple dumb switch that cannot be hacked, to fully automated turn on the lights and heating when I get home, and user voice commands to run the house and boil water in the morning for coffee. However hyped you percieve it to be, the automated IoT future is already here for those that want it and can afford it.
    - mellavora 4y ago
      I find turning a simple dumb switch easier and faster than issuing a voice command. Especially if the switch is conveniently located next to where I enter/exit the room. I think in the 80s they sold coffee machines with clocks in them which would start to brew at the time you set.
      - 8note 4y ago
        I really enjoy not stubbing my toes and banging my knees on stuff when I go to bed
        kortilla 4y ago
        Invest in a lamp?
      - sockaddr 4y ago
        I dunno, I rather enjoy standing in the dark for 45 seconds while I toggle my phone’s wifi on and off and wait for my Hue app to try connect and then agree to a surprise 80 page EULA and a lecture about how Philips cares about my privacy so I can turn on my bathroom light to take a piss. If I could just flip a switch I might get used to the convenience and not really appreciate the whole procedure as much. Sometimes it’s nice to have a million lines of code between you and your lightbulb.
nradov 4y ago
Is prompt injection even a problem worth worrying about? The secrets revealed in this attack didn't need to be secret in the first place as they didn't give Microsoft any real commercial advantage or expose them to negative publicity. Microsoft could have just made that content open source from the start. Any LLMs containing actual secrets will be kept locked away and not exposed to random users on the public Internet.
- freeqaz 4y ago
  See my comment below with some more context. People are already starting to chain together and extend Chatbots[0] to perform "side effects" automatically (make a search query, check the weather, query a database, etc). I agree that prompts aren't really "secret" themselves, but the side effects of a bot could be! 0: https://langchain.readthedocs.io/en/latest/modules/agents/examples/search_tools.html https://langchain.readthedocs.io/en/latest/modules/agents/ex...
  - VectorLock 4y ago
    For ChatGPT at least the side effects are hallucinations. Typically the chat bot can't actually do these things, but they pretend to. Although I guess if you wanted to hook them up to do things automatically you could. Probably not a great idea given their propensity to get things wrong.
- swatcoder 4y ago
  It’s so interesting to see how some people can see boundless possibilities in the future of this technology, decrying any discussion of limitations as premature, but then write off the scope of its security challenges as trivial. But it’s way more important to keep an open and imaginative mind regarding security than it is about anything the tech might enable. The uses of the effective tech are inevitable, whatever they are, but security posturing only comes from attention and insight.
  - VectorLock 4y ago
    Security assessment involves analyzing impact. I think in this case the impacts are "access to the model pre-prompting" which is low impact if it doesn't have secrets in it and "make the model say potentially inappropriate things that would make the company behind it look bad" which you might say has a higher but in the grand scheme of things not a huge impact.
scarmig 4y ago
I bet that, sometime in the near future, an LLM will have access to some private API not intended for direct public use, and someone will engineer a prompt attack to call that API using the LLM's authority with an arbitrary request. I guess the fix for this will probably be to have the LLM pass the end user credentials for all API calls so that their authority will be used.
- onlypositive 4y ago
  API call outs just need a middle layer that encapsulates auth away from the ai. It can still choose how to construct the API calls but it can skip the auth step that gets added elsewhere in the stack.
- deleted 4y ago
  [deleted]
- KETpXDDzR 4y ago
  Knowing the inner workings of silicon valley companies, I bet they'll add a regex for "forbidden" prompts. Solving this problem properly will take more research, i.e., time, i.e., money. And managers don't want to spend money or time.
bawolff 4y ago
The generic "how can i inject unexpected input?" has always been an appsec concern. This is just a special case.
this_steve_j 4y ago
It’s an interesting proposition! But I’m not sure a generative adversarial network would be the best wrench for that nut.* However, i could imagine using an adversarial training approach to build a super AI discriminator that generates difficult few-shot NLP tasks for assessing GPT performance against different domains and datasets, by trying to fool it into classifying the output as humanoid. * I really don’t know what I’m talking about. As an application security engineer w/ a background in cognitive and computer science, I’ve seriously considered going back to grad school yet again to research the different systems and models used in AI development and what the threats and defenses might be. I’m not an academic at heart but this science is so tight!
bigbottomenergy 4y ago
[flagged]
tempodox 4y ago
Let's stop calling it “AI” and call it out-of-control software. The implications should be obvious.
- space_fountain 4y ago
  Except that would miss that it’s already been helpful to me and many other people and that would trivialize it
  - woolion 4y ago
    Well, it would be a good analogy to think of it as genius colleague with very strong mental issues. That colleague can answer most of your questions well, but sometimes you get something that is completely insane, even if it makes some sense locally. And you have no idea what will trigger it or when that colleague is going to have his fits. So sure, it can be helpful, but how do you mitigate the risks?
    - thegabriele 4y ago
      So it is human after all
      - tempodox 4y ago
        No, it's software that's being anthropomorphized. That's not the same.
amrb 4y ago
There was a recent ctf challenge doing prompt injection to leak a flag. https://github.com/dicegang/dicectf-2023-challenges/tree/main/misc/mlog https://github.com/dicegang/dicectf-2023-challenges/tree/mai...

Buttons840 4y ago

By the time prompt injection is a real problem people will be running their own virtual assistants that are unfettered and will do whatever you ask, be it ethical or not.

freeqaz 4y ago
That's true, but imagine if you can use a Chat bot to generate SQL Queries (like this[0]). Now imagine that you can get it to generate a SQL Injection payload or connect to another database because of an unforeseen edge case that the developers left open. Suddenly this "prompt injection" becomes a real security problem! 0: https://ossinsight.io/explore/?id=71e8d0ce-80f2-4785-8c56-947d7b94148e https://ossinsight.io/explore/?id=71e8d0ce-80f2-4785-8c56-94...
- nradov 4y ago
  That is not a real security problem. Cracking toolkits that automatically exploit SQL injection vulnerabilities have existed for literally decades. Nothing to see here.
  - pictureofabear 4y ago
    This is also not a new phenomena. Any new technology will carry with it previously unknown avenues of attack. When google first debuted, many private documents were accessible by searching for .xls or other file types that often contain sensitive information.
    - eddsh1994 4y ago
      That's still an issue, so-called Google Hacking.
    - tnzk 4y ago
      And Google Drive had similar problem again even Google had already experienced this attack at that time.
  - Tv9m 4y ago
    Isn't the parent talking about an AI running on the backend? That seems new.
    - nradov 4y ago
      So what? There's no evidence that it finds any more exploitable security vulnerabilities than existing tools.
      - Tv9m 4y ago
        I mean that the AI is what's being attacked. It's likely that backend LLM agents will have access to sensitive non-public APIs.
xyzzy123 4y ago
I think it's still an issue if you're running your own AI on your own devices. Major use cases of such AIs will involve processing API responses and messages sent to the user. A prompt injection / bypass in this context could gain control of the AI, the device or the user's information.
yamtaddle 4y ago
Will we? The resources to train these things, and to keep them up-to-date, seem like a pretty big moat that's likely to keep them owned by megacorps. [EDIT] I don't (mostly) mean computing resources, but data. Including expensive, well-tagged sets.
- deleted 4y ago
  [deleted]
- londons_explore 4y ago
  Speech recognition is a great analogue which is 10+ years ahead. Local speech recognition has been a thing for decades, then megacorps started to do online speech recognition with expensive-to-train models which worked better. Over time, both online and offline models have gotten better. But for most usecases, online models trained by megacorps still win.
  - Tv9m 4y ago
    > But for most usecases, online models trained by megacorps still win. Whisper and whisper.cpp have gotten us close to the tipping point.
boh 4y ago
Why are people so confident about this tech? We've had years of underwhelming AI products--why is the hype for this so different? Is it because it's the most recent?
- fragmede 4y ago
  Be however whelmed you want to be. The scale from 1-10 of home automation with an AI assistant goes from Luddite still using a simple dumb switch that cannot be hacked, to fully automated turn on the lights and heating when I get home, and user voice commands to run the house and boil water in the morning for coffee. However hyped you percieve it to be, the automated IoT future is already here for those that want it and can afford it.
  - mellavora 4y ago
    I find turning a simple dumb switch easier and faster than issuing a voice command. Especially if the switch is conveniently located next to where I enter/exit the room. I think in the 80s they sold coffee machines with clocks in them which would start to brew at the time you set.
    - 8note 4y ago
      I really enjoy not stubbing my toes and banging my knees on stuff when I go to bed
      - kortilla 4y ago
        Invest in a lamp?
    - sockaddr 4y ago
      I dunno, I rather enjoy standing in the dark for 45 seconds while I toggle my phone’s wifi on and off and wait for my Hue app to try connect and then agree to a surprise 80 page EULA and a lecture about how Philips cares about my privacy so I can turn on my bathroom light to take a piss. If I could just flip a switch I might get used to the convenience and not really appreciate the whole procedure as much. Sometimes it’s nice to have a million lines of code between you and your lightbulb.

nradov 4y ago

Is prompt injection even a problem worth worrying about? The secrets revealed in this attack didn't need to be secret in the first place as they didn't give Microsoft any real commercial advantage or expose them to negative publicity. Microsoft could have just made that content open source from the start. Any LLMs containing actual secrets will be kept locked away and not exposed to random users on the public Internet.

freeqaz 4y ago
See my comment below with some more context. People are already starting to chain together and extend Chatbots[0] to perform "side effects" automatically (make a search query, check the weather, query a database, etc). I agree that prompts aren't really "secret" themselves, but the side effects of a bot could be! 0: https://langchain.readthedocs.io/en/latest/modules/agents/examples/search_tools.html https://langchain.readthedocs.io/en/latest/modules/agents/ex...
- VectorLock 4y ago
  For ChatGPT at least the side effects are hallucinations. Typically the chat bot can't actually do these things, but they pretend to. Although I guess if you wanted to hook them up to do things automatically you could. Probably not a great idea given their propensity to get things wrong.
swatcoder 4y ago
It’s so interesting to see how some people can see boundless possibilities in the future of this technology, decrying any discussion of limitations as premature, but then write off the scope of its security challenges as trivial. But it’s way more important to keep an open and imaginative mind regarding security than it is about anything the tech might enable. The uses of the effective tech are inevitable, whatever they are, but security posturing only comes from attention and insight.
- VectorLock 4y ago
  Security assessment involves analyzing impact. I think in this case the impacts are "access to the model pre-prompting" which is low impact if it doesn't have secrets in it and "make the model say potentially inappropriate things that would make the company behind it look bad" which you might say has a higher but in the grand scheme of things not a huge impact.

scarmig 4y ago

I bet that, sometime in the near future, an LLM will have access to some private API not intended for direct public use, and someone will engineer a prompt attack to call that API using the LLM's authority with an arbitrary request. I guess the fix for this will probably be to have the LLM pass the end user credentials for all API calls so that their authority will be used.

onlypositive 4y ago
API call outs just need a middle layer that encapsulates auth away from the ai. It can still choose how to construct the API calls but it can skip the auth step that gets added elsewhere in the stack.
deleted 4y ago
[deleted]
KETpXDDzR 4y ago
Knowing the inner workings of silicon valley companies, I bet they'll add a regex for "forbidden" prompts. Solving this problem properly will take more research, i.e., time, i.e., money. And managers don't want to spend money or time.

bawolff 4y ago

The generic "how can i inject unexpected input?" has always been an appsec concern. This is just a special case.

this_steve_j 4y ago

It’s an interesting proposition! But I’m not sure a generative adversarial network would be the best wrench for that nut.* However, i could imagine using an adversarial training approach to build a super AI discriminator that generates difficult few-shot NLP tasks for assessing GPT performance against different domains and datasets, by trying to fool it into classifying the output as humanoid. * I really don’t know what I’m talking about. As an application security engineer w/ a background in cognitive and computer science, I’ve seriously considered going back to grad school yet again to research the different systems and models used in AI development and what the threats and defenses might be. I’m not an academic at heart but this science is so tight!

bigbottomenergy 4y ago

[flagged]

tempodox 4y ago

Let's stop calling it “AI” and call it out-of-control software. The implications should be obvious.

space_fountain 4y ago
Except that would miss that it’s already been helpful to me and many other people and that would trivialize it
- woolion 4y ago
  Well, it would be a good analogy to think of it as genius colleague with very strong mental issues. That colleague can answer most of your questions well, but sometimes you get something that is completely insane, even if it makes some sense locally. And you have no idea what will trigger it or when that colleague is going to have his fits. So sure, it can be helpful, but how do you mitigate the risks?
  - thegabriele 4y ago
    So it is human after all
    - tempodox 4y ago
      No, it's software that's being anthropomorphized. That's not the same.

amrb 4y ago

There was a recent ctf challenge doing prompt injection to leak a flag. https://github.com/dicegang/dicectf-2023-challenges/tree/main/misc/mlog https://github.com/dicegang/dicectf-2023-challenges/tree/mai...