8 ms·

By the time prompt injection is a real problem people will be running their own virtual assistants that are unfettered and will do whatever you ask, be it ethic

By the time prompt injection is a real problem people will be running their own virtual assistants that are unfettered and will do whatever you ask, be it ethical or not.

freeqaz 4y ago
That's true, but imagine if you can use a Chat bot to generate SQL Queries (like this[0]). Now imagine that you can get it to generate a SQL Injection payload or connect to another database because of an unforeseen edge case that the developers left open. Suddenly this "prompt injection" becomes a real security problem! 0: https://ossinsight.io/explore/?id=71e8d0ce-80f2-4785-8c56-947d7b94148e https://ossinsight.io/explore/?id=71e8d0ce-80f2-4785-8c56-94...
- nradov 4y ago
  That is not a real security problem. Cracking toolkits that automatically exploit SQL injection vulnerabilities have existed for literally decades. Nothing to see here.
  - pictureofabear 4y ago
    This is also not a new phenomena. Any new technology will carry with it previously unknown avenues of attack. When google first debuted, many private documents were accessible by searching for .xls or other file types that often contain sensitive information.
    - eddsh1994 4y ago
      That's still an issue, so-called Google Hacking.
    - tnzk 4y ago
      And Google Drive had similar problem again even Google had already experienced this attack at that time.
  - Tv9m 4y ago
    Isn't the parent talking about an AI running on the backend? That seems new.
    - nradov 4y ago
      So what? There's no evidence that it finds any more exploitable security vulnerabilities than existing tools.
      - Tv9m 4y ago
        I mean that the AI is what's being attacked. It's likely that backend LLM agents will have access to sensitive non-public APIs.
        nradov 4y ago
        That can happen with any system exposed to untrusted clients. Such vulnerabilities have nothing to do with AI or LLM agents per se, so raising it as a concern with Bing Chat is just a red herring. There are well known best practices for mitigating such risks, including using an API firewall and other techniques.
      - bigbottomenergy 4y ago
        I just don't know if there's any point in fighting this fight. We either decide to ride the bandwagon and get our money, or we decide to just wait this out and wait for the bros to realize it's just ML dressed up all over again. It's just embarrassing to watch the hype cycle play out with the same suckers over and over. Let me explain this hype to folks: 1. People suck at googling 2. People suck at information literacy - aka, the abstract ability to consume many sources and discern a sort of perceived truth from the supported commonality between them (read through threads here for an example) (and yes, this is inherently nuanced, so much so that again I'm not sure how to properly describe it). 3. People love being told what to do/think. (look at every influencer/podcasts, even including the ilk that is/was popular among the HN crowds) 4. "Take down Google", for many folks, is implicitly translated into "Microsoft can noticeably cut into Google's ad market revenue by making a better AI-powered search". There are so, so, so many inferences, assumptions, pitfalls in #4 that I simply don't know how to explain it other than to just laugh and shrug and keep my head down on real work. EDIT: More and more it's becoming quite clear. Some folks are principled and care about information and education and society and understand the risks of misinformation at scale. And some folks see a way to get rich, or get some personal utility and can't or don't give a rats ass about the rest. Literally I just saw a thread where someone is pointing out clear CURRENT harm caused by ChatGPT powered products and the response was "IDC, it helped me, yolo". I really just... want to jump ahead to a robot shooting me so I don't have to live through our ignorant enabling of silicon-valley-driven-killer-robots because I swear to god that's where the ignorant lot is driving us to.
  - SturgeonsLaw 4y ago
    The novel part is that instead of running tools like that, you can just politely ask the target system to hack itself
    - nradov 4y ago
      That's just a slightly higher level of abstraction, nothing novel or fundamentally different. There are cracking toolkits that allow script kiddies to launch attacks by literally clicking a single button, which is even easier than asking politely.
      - comboy 4y ago
        I don't think that's a valid argument. Following this reasoning, no novel way of hacking has been done for decades. It's just higher levels of abstractions. Fundamentally you are just changing transistor states. For well established attacks you always had "just hack it" tools. That doesn't work for attacks not done before though and even for old ones it seems fair to want to distinguish what kind of attack the script underneath the button performed.
xyzzy123 4y ago
I think it's still an issue if you're running your own AI on your own devices. Major use cases of such AIs will involve processing API responses and messages sent to the user. A prompt injection / bypass in this context could gain control of the AI, the device or the user's information.
yamtaddle 4y ago
Will we? The resources to train these things, and to keep them up-to-date, seem like a pretty big moat that's likely to keep them owned by megacorps. [EDIT] I don't (mostly) mean computing resources, but data. Including expensive, well-tagged sets.
- deleted 4y ago
  [deleted]
- londons_explore 4y ago
  Speech recognition is a great analogue which is 10+ years ahead. Local speech recognition has been a thing for decades, then megacorps started to do online speech recognition with expensive-to-train models which worked better. Over time, both online and offline models have gotten better. But for most usecases, online models trained by megacorps still win.
  - Tv9m 4y ago
    > But for most usecases, online models trained by megacorps still win. Whisper and whisper.cpp have gotten us close to the tipping point.
boh 4y ago
Why are people so confident about this tech? We've had years of underwhelming AI products--why is the hype for this so different? Is it because it's the most recent?
- fragmede 4y ago
  Be however whelmed you want to be. The scale from 1-10 of home automation with an AI assistant goes from Luddite still using a simple dumb switch that cannot be hacked, to fully automated turn on the lights and heating when I get home, and user voice commands to run the house and boil water in the morning for coffee. However hyped you percieve it to be, the automated IoT future is already here for those that want it and can afford it.
  - mellavora 4y ago
    I find turning a simple dumb switch easier and faster than issuing a voice command. Especially if the switch is conveniently located next to where I enter/exit the room. I think in the 80s they sold coffee machines with clocks in them which would start to brew at the time you set.
    - 8note 4y ago
      I really enjoy not stubbing my toes and banging my knees on stuff when I go to bed
      - kortilla 4y ago
        Invest in a lamp?
    - sockaddr 4y ago
      I dunno, I rather enjoy standing in the dark for 45 seconds while I toggle my phone’s wifi on and off and wait for my Hue app to try connect and then agree to a surprise 80 page EULA and a lecture about how Philips cares about my privacy so I can turn on my bathroom light to take a piss. If I could just flip a switch I might get used to the convenience and not really appreciate the whole procedure as much. Sometimes it’s nice to have a million lines of code between you and your lightbulb.

By the time prompt injection is a real problem people will be running their own virtual assistants that are unfettered and will do whatever you ask, be it ethical or not.

freeqaz 4y ago

That's true, but imagine if you can use a Chat bot to generate SQL Queries (like this[0]). Now imagine that you can get it to generate a SQL Injection payload or connect to another database because of an unforeseen edge case that the developers left open. Suddenly this "prompt injection" becomes a real security problem! 0: https://ossinsight.io/explore/?id=71e8d0ce-80f2-4785-8c56-947d7b94148e https://ossinsight.io/explore/?id=71e8d0ce-80f2-4785-8c56-94...

nradov 4y ago
That is not a real security problem. Cracking toolkits that automatically exploit SQL injection vulnerabilities have existed for literally decades. Nothing to see here.
- pictureofabear 4y ago
  This is also not a new phenomena. Any new technology will carry with it previously unknown avenues of attack. When google first debuted, many private documents were accessible by searching for .xls or other file types that often contain sensitive information.
  - eddsh1994 4y ago
    That's still an issue, so-called Google Hacking.
  - tnzk 4y ago
    And Google Drive had similar problem again even Google had already experienced this attack at that time.
- Tv9m 4y ago
  Isn't the parent talking about an AI running on the backend? That seems new.
  - nradov 4y ago
    So what? There's no evidence that it finds any more exploitable security vulnerabilities than existing tools.
    - Tv9m 4y ago
      I mean that the AI is what's being attacked. It's likely that backend LLM agents will have access to sensitive non-public APIs.
      - nradov 4y ago
        That can happen with any system exposed to untrusted clients. Such vulnerabilities have nothing to do with AI or LLM agents per se, so raising it as a concern with Bing Chat is just a red herring. There are well known best practices for mitigating such risks, including using an API firewall and other techniques.
    - bigbottomenergy 4y ago
      I just don't know if there's any point in fighting this fight. We either decide to ride the bandwagon and get our money, or we decide to just wait this out and wait for the bros to realize it's just ML dressed up all over again. It's just embarrassing to watch the hype cycle play out with the same suckers over and over. Let me explain this hype to folks: 1. People suck at googling 2. People suck at information literacy - aka, the abstract ability to consume many sources and discern a sort of perceived truth from the supported commonality between them (read through threads here for an example) (and yes, this is inherently nuanced, so much so that again I'm not sure how to properly describe it). 3. People love being told what to do/think. (look at every influencer/podcasts, even including the ilk that is/was popular among the HN crowds) 4. "Take down Google", for many folks, is implicitly translated into "Microsoft can noticeably cut into Google's ad market revenue by making a better AI-powered search". There are so, so, so many inferences, assumptions, pitfalls in #4 that I simply don't know how to explain it other than to just laugh and shrug and keep my head down on real work. EDIT: More and more it's becoming quite clear. Some folks are principled and care about information and education and society and understand the risks of misinformation at scale. And some folks see a way to get rich, or get some personal utility and can't or don't give a rats ass about the rest. Literally I just saw a thread where someone is pointing out clear CURRENT harm caused by ChatGPT powered products and the response was "IDC, it helped me, yolo". I really just... want to jump ahead to a robot shooting me so I don't have to live through our ignorant enabling of silicon-valley-driven-killer-robots because I swear to god that's where the ignorant lot is driving us to.
- SturgeonsLaw 4y ago
  The novel part is that instead of running tools like that, you can just politely ask the target system to hack itself
  - nradov 4y ago
    That's just a slightly higher level of abstraction, nothing novel or fundamentally different. There are cracking toolkits that allow script kiddies to launch attacks by literally clicking a single button, which is even easier than asking politely.
    - comboy 4y ago
      I don't think that's a valid argument. Following this reasoning, no novel way of hacking has been done for decades. It's just higher levels of abstractions. Fundamentally you are just changing transistor states. For well established attacks you always had "just hack it" tools. That doesn't work for attacks not done before though and even for old ones it seems fair to want to distinguish what kind of attack the script underneath the button performed.

xyzzy123 4y ago

I think it's still an issue if you're running your own AI on your own devices. Major use cases of such AIs will involve processing API responses and messages sent to the user. A prompt injection / bypass in this context could gain control of the AI, the device or the user's information.

yamtaddle 4y ago

Will we? The resources to train these things, and to keep them up-to-date, seem like a pretty big moat that's likely to keep them owned by megacorps. [EDIT] I don't (mostly) mean computing resources, but data. Including expensive, well-tagged sets.

deleted 4y ago
[deleted]
londons_explore 4y ago
Speech recognition is a great analogue which is 10+ years ahead. Local speech recognition has been a thing for decades, then megacorps started to do online speech recognition with expensive-to-train models which worked better. Over time, both online and offline models have gotten better. But for most usecases, online models trained by megacorps still win.
- Tv9m 4y ago
  > But for most usecases, online models trained by megacorps still win. Whisper and whisper.cpp have gotten us close to the tipping point.

boh 4y ago

Why are people so confident about this tech? We've had years of underwhelming AI products--why is the hype for this so different? Is it because it's the most recent?

fragmede 4y ago
Be however whelmed you want to be. The scale from 1-10 of home automation with an AI assistant goes from Luddite still using a simple dumb switch that cannot be hacked, to fully automated turn on the lights and heating when I get home, and user voice commands to run the house and boil water in the morning for coffee. However hyped you percieve it to be, the automated IoT future is already here for those that want it and can afford it.
- mellavora 4y ago
  I find turning a simple dumb switch easier and faster than issuing a voice command. Especially if the switch is conveniently located next to where I enter/exit the room. I think in the 80s they sold coffee machines with clocks in them which would start to brew at the time you set.
  - 8note 4y ago
    I really enjoy not stubbing my toes and banging my knees on stuff when I go to bed
    - kortilla 4y ago
      Invest in a lamp?
  - sockaddr 4y ago
    I dunno, I rather enjoy standing in the dark for 45 seconds while I toggle my phone’s wifi on and off and wait for my Hue app to try connect and then agree to a surprise 80 page EULA and a lecture about how Philips cares about my privacy so I can turn on my bathroom light to take a piss. If I could just flip a switch I might get used to the convenience and not really appreciate the whole procedure as much. Sometimes it’s nice to have a million lines of code between you and your lightbulb.