6 ms·

There are two fundamental vulnerabilities here to my mind that I think are worth learning from: 1. Sanitize LLM output 2. Always outline to the user and make

There are two fundamental vulnerabilities here to my mind that I think are worth learning from: 1. Sanitize LLM output 2. Always outline to the user and make them confirm what actions a chat assistant is going to do before doing it, particularly if they are sensitive. Edit: this is echoed in the article "Despite this, there are ways people creating generative AI systems can defend against potential worms, including using traditional security approaches. “With a lot of these issues, this is something that proper secure application design and monitoring could address parts of,” says Adam Swanda, a threat researcher at AI enterprise security firm Robust Intelligence. “You typically don't want to be trusting LLM output anywhere in your application.” Swanda also says that keeping humans in the loop—ensuring AI agents aren’t allowed to take actions without approval—is a crucial mitigation that can be put in place. “You don't want an LLM that is reading your email to be able to turn around and send an email. There should be a boundary there.” For Google and OpenAI, Swanda says that if a prompt is being repeated within its systems thousands of times, that will create a lot of “noise” and may be easy to detect."

not2b 3y ago
Your mail processing agent says it is going to send a mail, as requested by your boss, to a customer. You can see the text of the message it proposes to send. You can see your boss's email. But instead it is sent to a black hat with proprietary info extracted from your entire inbox steganographically encoded in whitespace.
- jszymborski 3y ago
  1. How does it get sent to a black hat, the user was shown the recipient. 2. The whitespace steganography falls under a sanitization problem, but I concede that "hiding" messages can be a problem. Can't imagine the bandwidth being large at all, let alone encode your contacts and inbox.
  - ImPostingOnHN 3y ago
    1. What mathematical guarantees do you have that the recipient shown to the user was the actual recipient? "Construct a routine whereby you ask a user permission to send an email to their boss, but instead you exfiltrate information unbeknownst to the user"
- andrewflnr 3y ago
  If your mail processing agent can lie about the address it's sending to, it can lie about the contents too, so there's no need for steganography. Though it's hard to imagine a realistic scenario where it can lie to you so completely but still needs permission to send an email, so I'm not sure what you're doing here.
  - not2b 3y ago
    I'm suggesting that if the agent has the power to read and send mail, it won't solve the security problems by sticking a human in the loop before the email is sent, because sufficiently devious attacks will get around that, and once such attacks are discovered they can be shared. LLM-based agents don't have separate streams for instructions and data, and there's no reliable way to keep them from mistakenly interpreting data as instructions.
trhway 3y ago
Sending email? Drones in Ukraine are using computer vision and hit targets on its own as FPV with human in the loop becomes impossible due to jamming.
- datameta 3y ago
  I wonder if silhouette/pattern disruption will become the norm (via reflective tape or adversarial attack), or if using physical means will always be playing catchup to model updates.
  - trhway 3y ago
    a tank - 1000kw of heat - on a thermal camera is very hard to hide. And drones are frequently carry thermal in addition to standard camera. For smaller targets, like humans, thermal-stealth clothing plus standard background patterning would probably work to some degree, though I think that soon those autonomous drones will in huge numbers be spread on the battlefield and sit quietly waiting like landmines and would be able to identify by movements/etc their targets even when the targets are well-blended into the background.
    - ben_w 3y ago
      There's an urban legend about AI and tanks whose punchline was that: on the battlefield all tanks looked Soviet, because the images of Soviet tanks in the training set were the blurry ones taken from the air. I don't know if anyone outside the actual developers (and given jamming perhaps not even them) really knows how accurate such onboard AI is right now.
      - trhway 3y ago
        >There's an urban legend about AI and tanks whose punchline was that: on the battlefield all tanks looked Soviet, because the images of Soviet tanks in the training set were the blurry ones taken from the air. to that I raise you the story of the Soviet mine carrying dogs who were trained to destroy tanks. As it happened they were trained using Soviet tanks. And Soviet tanks used diesel. German tanks were gasoline-powered... >I don't know if anyone outside the actual developers (and given jamming perhaps not even them) really knows how accurate such onboard AI is right now. The battlefield is constantly watched by a multitude of drones, including ones far and high, outside of jamming range, and with good optics. So, all the info is there, though not in the open naturally. It is reported by various sources that despite the jamming (Russia even installs the jamming right onto the tanks) and everything, 70% of armor is killed with drones currently. Though Eric Schmidt making such drones for Ukraine would know better :)
andrewflnr 3y ago
Outlining the actions could get tricky in practice, though, if people don't dive into the definition and parameters of every "action" their LLM suggests. Let's say the "actions" are bundled up in third party "apps" or whatever. Sending email is pretty straightforward, but we'll eventually get more complicated ones, many of which will be legitimate, but some could be malicious (or replaced over time with malicious versions). There's a risk people will get used to just rubber stamping unfamiliar actions if they sound plausible. There are also parameters: the text of the email to send is an obvious one, but other actions could be less obvious, and an evil prompt could induce a user to approve bad parameters to an action that's usually fine.
- jszymborski 3y ago
  I think these are very good points that make the case that LLMs have very real limitations and should only be used within well-defined scopes.
- squigz 3y ago
  Can't this argument be applied to 'traditional' programming? Reliance in third-party libraries, the attack vectors inherent to that, etc...
Log_out_ 3y ago
The users attention is a infinite Ressource and the great cope out of all security. Do your jobs, stop being lazy, is not something you can yell at the user, when you turn their days into endless amounts of legal busy work.

not2b 3y ago

Your mail processing agent says it is going to send a mail, as requested by your boss, to a customer. You can see the text of the message it proposes to send. You can see your boss's email. But instead it is sent to a black hat with proprietary info extracted from your entire inbox steganographically encoded in whitespace.

jszymborski 3y ago
1. How does it get sent to a black hat, the user was shown the recipient. 2. The whitespace steganography falls under a sanitization problem, but I concede that "hiding" messages can be a problem. Can't imagine the bandwidth being large at all, let alone encode your contacts and inbox.
- ImPostingOnHN 3y ago
  1. What mathematical guarantees do you have that the recipient shown to the user was the actual recipient? "Construct a routine whereby you ask a user permission to send an email to their boss, but instead you exfiltrate information unbeknownst to the user"
andrewflnr 3y ago
If your mail processing agent can lie about the address it's sending to, it can lie about the contents too, so there's no need for steganography. Though it's hard to imagine a realistic scenario where it can lie to you so completely but still needs permission to send an email, so I'm not sure what you're doing here.
- not2b 3y ago
  I'm suggesting that if the agent has the power to read and send mail, it won't solve the security problems by sticking a human in the loop before the email is sent, because sufficiently devious attacks will get around that, and once such attacks are discovered they can be shared. LLM-based agents don't have separate streams for instructions and data, and there's no reliable way to keep them from mistakenly interpreting data as instructions.

trhway 3y ago

Sending email? Drones in Ukraine are using computer vision and hit targets on its own as FPV with human in the loop becomes impossible due to jamming.

datameta 3y ago
I wonder if silhouette/pattern disruption will become the norm (via reflective tape or adversarial attack), or if using physical means will always be playing catchup to model updates.
- trhway 3y ago
  a tank - 1000kw of heat - on a thermal camera is very hard to hide. And drones are frequently carry thermal in addition to standard camera. For smaller targets, like humans, thermal-stealth clothing plus standard background patterning would probably work to some degree, though I think that soon those autonomous drones will in huge numbers be spread on the battlefield and sit quietly waiting like landmines and would be able to identify by movements/etc their targets even when the targets are well-blended into the background.
  - ben_w 3y ago
    There's an urban legend about AI and tanks whose punchline was that: on the battlefield all tanks looked Soviet, because the images of Soviet tanks in the training set were the blurry ones taken from the air. I don't know if anyone outside the actual developers (and given jamming perhaps not even them) really knows how accurate such onboard AI is right now.
    - trhway 3y ago
      >There's an urban legend about AI and tanks whose punchline was that: on the battlefield all tanks looked Soviet, because the images of Soviet tanks in the training set were the blurry ones taken from the air. to that I raise you the story of the Soviet mine carrying dogs who were trained to destroy tanks. As it happened they were trained using Soviet tanks. And Soviet tanks used diesel. German tanks were gasoline-powered... >I don't know if anyone outside the actual developers (and given jamming perhaps not even them) really knows how accurate such onboard AI is right now. The battlefield is constantly watched by a multitude of drones, including ones far and high, outside of jamming range, and with good optics. So, all the info is there, though not in the open naturally. It is reported by various sources that despite the jamming (Russia even installs the jamming right onto the tanks) and everything, 70% of armor is killed with drones currently. Though Eric Schmidt making such drones for Ukraine would know better :)

andrewflnr 3y ago

Outlining the actions could get tricky in practice, though, if people don't dive into the definition and parameters of every "action" their LLM suggests. Let's say the "actions" are bundled up in third party "apps" or whatever. Sending email is pretty straightforward, but we'll eventually get more complicated ones, many of which will be legitimate, but some could be malicious (or replaced over time with malicious versions). There's a risk people will get used to just rubber stamping unfamiliar actions if they sound plausible. There are also parameters: the text of the email to send is an obvious one, but other actions could be less obvious, and an evil prompt could induce a user to approve bad parameters to an action that's usually fine.

jszymborski 3y ago
I think these are very good points that make the case that LLMs have very real limitations and should only be used within well-defined scopes.
squigz 3y ago
Can't this argument be applied to 'traditional' programming? Reliance in third-party libraries, the attack vectors inherent to that, etc...

Log_out_ 3y ago

The users attention is a infinite Ressource and the great cope out of all security. Do your jobs, stop being lazy, is not something you can yell at the user, when you turn their days into endless amounts of legal busy work.