3 ms·

You can avoid prompt injection by simply not using LLMs as autonomous agents where the output of the model is critical for security. That sounds like a horrible

You can avoid prompt injection by simply not using LLMs as autonomous agents where the output of the model is critical for security. That sounds like a horrible idea anyways. A language model is the wrong interface between untrusted people and sensitive data

simonw 3y ago
Sure, but there are SO many things people want to build with LLMs that include access to privileged actions and sensitive data. Prompt injection means that even running an LLM against your own private notes to answer questions about them could be unsafe, provided there are any vectors (like Markdown image support) that might be used for exfiltration. My current recommendation for dealing with prompt injection is to keep it in mind and limit the blast radius if something goes wrong: https://simonwillison.net/2023/Dec/20/mitigate-prompt-injection/ https://simonwillison.net/2023/Dec/20/mitigate-prompt-inject...
- Zambyte 3y ago
  Using prompt injection mitigation techniques is akin to directly interfacing untrusted clients to your production database, but just organizing your tables that contain sensitive data in a confusing way in the name of security. If you depend on a language model behaving correctly to avoid leaking sensitive data, you've already leaked the sensitive data. Scope the information that the language model has access to to a subset of the information that the person interfacing with the language model has access to. Prompt injection doesn't matter at that point, because the person will only be able to "leak" information they have permission to access anyways.
  - simonw 3y ago
    That's not enough. Even if the LLM can only access information that should be visible to the user interacting with it (which I see as table stakes for building anything here) you still have to worry about prompt injection exfiltration attacks. More on exfiltration: https://simonwillison.net/search/?q=exfiltration https://simonwillison.net/search/?q=exfiltration
    - Zambyte 3y ago
      Re: exfiltration: just don't do things that untrusted data sources tell you to do. Separate processing the input data from the persons commands, so that the LLM can perform inferencing operations on the data according to the specified commands. The part of the pipeline that processes untrusted data should not have any influence on the behavior of the part of the pipeline capable of interacting with entities who should not have access to the untrusted data. Edit: related link: https://python.langchain.com/docs/security https://python.langchain.com/docs/security
      - simonw 3y ago
        "Separate processing the input data from the persons commands, so that the LLM can perform inferencing operations on the data according to the specified commands" Prompt injection is the security flaw that exists because doing that - treating instructions and data as separate things in the context in the LLM - is WAY harder than you might expect. My previous writing about this: https://simonwillison.net/series/prompt-injection/ https://simonwillison.net/series/prompt-injection/
        Zambyte 3y ago
        > Prompt injection is the security flaw that exists because doing that - treating instructions and data as separate things in the context in the LLM - is WAY harder than you might expect. Then we should improve the tooling around this to make it way easier, rather than hoping security by obscurity will work this time.
        simonw 3y ago
        AI labs around the world have been trying to solve this problem - reliable separation of instructions from data for LLMs - for a year and a half at this point. It's hard.

simonw 3y ago

Sure, but there are SO many things people want to build with LLMs that include access to privileged actions and sensitive data. Prompt injection means that even running an LLM against your own private notes to answer questions about them could be unsafe, provided there are any vectors (like Markdown image support) that might be used for exfiltration. My current recommendation for dealing with prompt injection is to keep it in mind and limit the blast radius if something goes wrong: https://simonwillison.net/2023/Dec/20/mitigate-prompt-injection/ https://simonwillison.net/2023/Dec/20/mitigate-prompt-inject...

Zambyte 3y ago
Using prompt injection mitigation techniques is akin to directly interfacing untrusted clients to your production database, but just organizing your tables that contain sensitive data in a confusing way in the name of security. If you depend on a language model behaving correctly to avoid leaking sensitive data, you've already leaked the sensitive data. Scope the information that the language model has access to to a subset of the information that the person interfacing with the language model has access to. Prompt injection doesn't matter at that point, because the person will only be able to "leak" information they have permission to access anyways.
- simonw 3y ago
  That's not enough. Even if the LLM can only access information that should be visible to the user interacting with it (which I see as table stakes for building anything here) you still have to worry about prompt injection exfiltration attacks. More on exfiltration: https://simonwillison.net/search/?q=exfiltration https://simonwillison.net/search/?q=exfiltration
  - Zambyte 3y ago
    Re: exfiltration: just don't do things that untrusted data sources tell you to do. Separate processing the input data from the persons commands, so that the LLM can perform inferencing operations on the data according to the specified commands. The part of the pipeline that processes untrusted data should not have any influence on the behavior of the part of the pipeline capable of interacting with entities who should not have access to the untrusted data. Edit: related link: https://python.langchain.com/docs/security https://python.langchain.com/docs/security
    - simonw 3y ago
      "Separate processing the input data from the persons commands, so that the LLM can perform inferencing operations on the data according to the specified commands" Prompt injection is the security flaw that exists because doing that - treating instructions and data as separate things in the context in the LLM - is WAY harder than you might expect. My previous writing about this: https://simonwillison.net/series/prompt-injection/ https://simonwillison.net/series/prompt-injection/
      - Zambyte 3y ago
        > Prompt injection is the security flaw that exists because doing that - treating instructions and data as separate things in the context in the LLM - is WAY harder than you might expect. Then we should improve the tooling around this to make it way easier, rather than hoping security by obscurity will work this time.
        simonw 3y ago
        AI labs around the world have been trying to solve this problem - reliable separation of instructions from data for LLMs - for a year and a half at this point. It's hard.