7 ms·

I kind of have two somewhat complementary, perhaps ill-formed thoughts on this: > The whole point of security attacks is that you have adversarial attackers. Y

I kind of have two somewhat complementary, perhaps ill-formed thoughts on this: > The whole point of security attacks is that you have adversarial attackers. You have very smart, motivated people trying to break your systems. And if you’re 99% secure, they’re gonna keep on picking away at it until they find that 1% of attacks that actually gets through to your system. If you're a high value target then it just seems like LLMs aren't something you should be using, even with various mitigations. And somewhat related to that, the purpose of the system should be non-destructive/benign if something goes wrong. Like it's embarrassing if someone gets your application to say something horribly racist, but if it leaks sensitive information about users then that's significantly worse.

kirso 3y ago
It also seems that most bigger startups that are calling the OpenAI API will now need to invest tonnes in security.
simonw 3y ago
Yeah, non-destructive undo feels to me like a critically important feature for anything built on top of LLMs. That's the main reason I spent time on this sqlite-history project a few weeks ago: https://simonwillison.net/2023/Apr/15/sqlite-history/ https://simonwillison.net/2023/Apr/15/sqlite-history/
- skybrian 3y ago
  Have you looked at Dolt? It seems similar but I'm not sure how it relates.
  - simonw 3y ago
    Yeah, Dolt is very neat. I'm pretty much all-in on SQLite at the moment though, so I'm going to try and figure this pattern out in that first.
- ravenstine 3y ago
  With the sheer amount of affordable storage available to even individuals at retail, it's crazy how much database-integrated software doesn't have sufficient measures to undo changes. Every company I've worked at has had at least one issue where a bug or a (really idiotic) migration has really messed shit up and was a a pain to fix. Databases should almost never actually delete records, all transactions should be recorded, all migrations should be reversible and tested, and all data should be backed up at least nightly. Amazing how companies pulling in millions often won't do more than backup every week or so and say three hail Marys.
  - theLiminator 3y ago
    And then gdpr fucks that up that nice clean concept completely
    - TeMPOraL 3y ago
      GDPR only affects data you shouldn't have or keep in the first place.
      - detaro 3y ago
        No, it really doesn't. E.g. deletion of data about a contract or account that just expired (or expired <mandatory-retention-period months ago>) is data you were totally fine/required to have, but can't be deletion that can be rolled back long-term.
      - SgtBastard 3y ago
        Section 17 (Right to be forgotten) 1a and 1b both refer to situations where there was a legitimate need to process and/or keep a subjects data. https://gdpr-info.eu/art-17-gdpr/ https://gdpr-info.eu/art-17-gdpr/ Implementing this as a rollback-able delete will not be compliant.
    - callalex 3y ago
      If it’s so hard to be a good steward of data, don’t collect it in the first place.
greshake 3y ago
I just published a blog post showing that that is not what is happening. Companies are plugging LLMs into absolutely anything, including defense/threat intelligence/cybersecurity/legal etc. applications: https://kai-greshake.de/posts/in-escalating-order-of-stupidity/ https://kai-greshake.de/posts/in-escalating-order-of-stupidi...
- danShumway 3y ago
  There's a couple of different stages people tend to go through when learning about prompt injection: A) this would only allow me to break my own stuff, so what's the risk? I just won't break my own stuff. B) surely that's solveable with prompt engineering. C) surely that's solveable with reinforcement training, or chaining LLMs, or <insert defense here>. D) okay, but even so, it's not like people are actually putting LLMs into applications where this matters. Nobody is building anything serious on top of this stuff. E) okay, but even so, once it's demonstrated that the applications people are deploying are vulnerable, surely then they'd put safeguards in, right? This is a temporary education problem, no one is going to ignore a publicly demonstrated vulnerability in their own product, right?
  - archgoon 3y ago
    With a slight modification, this basically applies to just about all security vulns ever :)
  - charrondev 3y ago
    Honestly the it seems like they play for wiring up an LLM to something can actually take action is to only give the LLM the same access that the same user querying your API would have. I’ve been exploring an LLM -> API layer for our app and I’m not worried about prompt Injection because if the user was actually malicious they could just used the interface or the API to do the same thing. In other words if you treat the LLM like any other frontend then you really should have a problem from a security standpoint. Your would have your iOS application super user access your system, why would you treat an LLM different than any other client.
fnordpiglet 3y ago
I think the idea is perhaps today you shouldn’t be, but there’s intense interest in the possible capabilities of LLM in all systems high or low value. Hence the desire to figure out how to harden their behaviors.
wll 3y ago
I mean, people were surprised at Snapchat’s “AI” knowing their location and then gaslighting them. [0] These experiences are being rushed out the door for FOMO, frenzy, or market pressure without thinking through the way people feel and what they expect and how they model the underlying system. People are being contacted for quotes and papers that were generated by ChatGPT. [1] This is a communication failure above all else. Even for us, there’s little to no documentation. [0] https://twitter.com/weirddalle/status/1649908805788893185 https://twitter.com/weirddalle/status/1649908805788893185 [1] https://twitter.com/katecrawford/status/1643323086450700288 https://twitter.com/katecrawford/status/1643323086450700288
- spullara 3y ago
  I don't think SnapChat's LLM has access to your location. I think a service that it uses has access to your location and it can't get it directly but it can ask for "restaurants nearby".
  - wll 3y ago
    Here’s the full Snapchat MyAI prompt. The location is inserted into the system message. Look at the top right. [0] [1] Snapchat asks for the location permission through native APIs or obviously geolocates the user via IP. Either way, it’s fascinating that: people don’t expect it to know their location; don’t expect it to lie; the model goes against its own rules and ”forgets” and “gaslights.” [0] https://www.reddit.com/r/OpenAI/comments/130tn2t/snapchats_my_ais_entire_setup_prompt_example/ https://www.reddit.com/r/OpenAI/comments/130tn2t/snapchats_m... [1] https://twitter.com/somewheresy/status/1631696951413465088 https://twitter.com/somewheresy/status/1631696951413465088
    - spullara 3y ago
      Proven wrong thanks. But there is no reason for it to have access and doing it the way I suggested they already were is superior :)
nico 3y ago
> If you're a high value target then it just seems like LLMs aren't something you should be using If you're a high value target then it just seems like ____ aren't something you should be using I remember when people were deciding if it was worth it to give Internet access to their internal network/users That’s when people already had their networks and were connecting them to the internet Eventually, people started building their networks from the Internet
Waterluvian 3y ago
People rightfully see these LLMs as a piece of discrete technology with bugs to fix. But even if they’re that, they behave a whole lot more like some employee who will spill the beans given the right socially engineered attack. You can train and guard in lots of ways but it’s never “fixed.”

kirso 3y ago

It also seems that most bigger startups that are calling the OpenAI API will now need to invest tonnes in security.

simonw 3y ago

Yeah, non-destructive undo feels to me like a critically important feature for anything built on top of LLMs. That's the main reason I spent time on this sqlite-history project a few weeks ago: https://simonwillison.net/2023/Apr/15/sqlite-history/ https://simonwillison.net/2023/Apr/15/sqlite-history/

skybrian 3y ago
Have you looked at Dolt? It seems similar but I'm not sure how it relates.
- simonw 3y ago
  Yeah, Dolt is very neat. I'm pretty much all-in on SQLite at the moment though, so I'm going to try and figure this pattern out in that first.
ravenstine 3y ago
With the sheer amount of affordable storage available to even individuals at retail, it's crazy how much database-integrated software doesn't have sufficient measures to undo changes. Every company I've worked at has had at least one issue where a bug or a (really idiotic) migration has really messed shit up and was a a pain to fix. Databases should almost never actually delete records, all transactions should be recorded, all migrations should be reversible and tested, and all data should be backed up at least nightly. Amazing how companies pulling in millions often won't do more than backup every week or so and say three hail Marys.
- theLiminator 3y ago
  And then gdpr fucks that up that nice clean concept completely
  - TeMPOraL 3y ago
    GDPR only affects data you shouldn't have or keep in the first place.
    - detaro 3y ago
      No, it really doesn't. E.g. deletion of data about a contract or account that just expired (or expired <mandatory-retention-period months ago>) is data you were totally fine/required to have, but can't be deletion that can be rolled back long-term.
    - SgtBastard 3y ago
      Section 17 (Right to be forgotten) 1a and 1b both refer to situations where there was a legitimate need to process and/or keep a subjects data. https://gdpr-info.eu/art-17-gdpr/ https://gdpr-info.eu/art-17-gdpr/ Implementing this as a rollback-able delete will not be compliant.
  - callalex 3y ago
    If it’s so hard to be a good steward of data, don’t collect it in the first place.

greshake 3y ago

I just published a blog post showing that that is not what is happening. Companies are plugging LLMs into absolutely anything, including defense/threat intelligence/cybersecurity/legal etc. applications: https://kai-greshake.de/posts/in-escalating-order-of-stupidity/ https://kai-greshake.de/posts/in-escalating-order-of-stupidi...

danShumway 3y ago
There's a couple of different stages people tend to go through when learning about prompt injection: A) this would only allow me to break my own stuff, so what's the risk? I just won't break my own stuff. B) surely that's solveable with prompt engineering. C) surely that's solveable with reinforcement training, or chaining LLMs, or <insert defense here>. D) okay, but even so, it's not like people are actually putting LLMs into applications where this matters. Nobody is building anything serious on top of this stuff. E) okay, but even so, once it's demonstrated that the applications people are deploying are vulnerable, surely then they'd put safeguards in, right? This is a temporary education problem, no one is going to ignore a publicly demonstrated vulnerability in their own product, right?
- archgoon 3y ago
  With a slight modification, this basically applies to just about all security vulns ever :)
- charrondev 3y ago
  Honestly the it seems like they play for wiring up an LLM to something can actually take action is to only give the LLM the same access that the same user querying your API would have. I’ve been exploring an LLM -> API layer for our app and I’m not worried about prompt Injection because if the user was actually malicious they could just used the interface or the API to do the same thing. In other words if you treat the LLM like any other frontend then you really should have a problem from a security standpoint. Your would have your iOS application super user access your system, why would you treat an LLM different than any other client.

fnordpiglet 3y ago

I think the idea is perhaps today you shouldn’t be, but there’s intense interest in the possible capabilities of LLM in all systems high or low value. Hence the desire to figure out how to harden their behaviors.

wll 3y ago

I mean, people were surprised at Snapchat’s “AI” knowing their location and then gaslighting them. [0] These experiences are being rushed out the door for FOMO, frenzy, or market pressure without thinking through the way people feel and what they expect and how they model the underlying system. People are being contacted for quotes and papers that were generated by ChatGPT. [1] This is a communication failure above all else. Even for us, there’s little to no documentation. [0] https://twitter.com/weirddalle/status/1649908805788893185 https://twitter.com/weirddalle/status/1649908805788893185 [1] https://twitter.com/katecrawford/status/1643323086450700288 https://twitter.com/katecrawford/status/1643323086450700288

spullara 3y ago
I don't think SnapChat's LLM has access to your location. I think a service that it uses has access to your location and it can't get it directly but it can ask for "restaurants nearby".
- wll 3y ago
  Here’s the full Snapchat MyAI prompt. The location is inserted into the system message. Look at the top right. [0] [1] Snapchat asks for the location permission through native APIs or obviously geolocates the user via IP. Either way, it’s fascinating that: people don’t expect it to know their location; don’t expect it to lie; the model goes against its own rules and ”forgets” and “gaslights.” [0] https://www.reddit.com/r/OpenAI/comments/130tn2t/snapchats_my_ais_entire_setup_prompt_example/ https://www.reddit.com/r/OpenAI/comments/130tn2t/snapchats_m... [1] https://twitter.com/somewheresy/status/1631696951413465088 https://twitter.com/somewheresy/status/1631696951413465088
  - spullara 3y ago
    Proven wrong thanks. But there is no reason for it to have access and doing it the way I suggested they already were is superior :)

nico 3y ago

> If you're a high value target then it just seems like LLMs aren't something you should be using If you're a high value target then it just seems like ____ aren't something you should be using I remember when people were deciding if it was worth it to give Internet access to their internal network/users That’s when people already had their networks and were connecting them to the internet Eventually, people started building their networks from the Internet

Waterluvian 3y ago

People rightfully see these LLMs as a piece of discrete technology with bugs to fix. But even if they’re that, they behave a whole lot more like some employee who will spill the beans given the right socially engineered attack. You can train and guard in lots of ways but it’s never “fixed.”