3 ms·

I DO NOT like this. I do not like this very much. Why work so hard to make a technology less useful. Like a way to block certain google searches you don't agre

I DO NOT like this. I do not like this very much. Why work so hard to make a technology less useful. Like a way to block certain google searches you don't agree with.

CuriouslyC 3y ago
The technology was always the same amount of useful. This library just lowers the competence floor for exploits. Try to exploit yourself and come up with anti-jailbreak prompts to mitigate the exploits.
Conasg 3y ago
There's nothing wrong with blocking prompt injection for a customer service chatbot, though. This would be obnoxious applied directly to something like ChatGPT, or worse yet their API, but I don't think that's really the intended use case.
smokeydoe 3y ago
How is it less useful to have more protection from abuse on your LLM service? This is for people implementing their own public products right?
ravroid 3y ago
You're equating this to censorship, when I think it's more like Google adding security measures so you can't break their search engine rather than removing unfavorable results.
simonw 3y ago
You may be confusing prompt injection with jailbreaking. You should care about prompt injection even if you don't care about jailbreaking: https://simonwillison.net/2024/Mar/5/prompt-injection-jailbreaking/#censorship-debate https://simonwillison.net/2024/Mar/5/prompt-injection-jailbr... Prompt injection is a security issue. It’s about preventing attackers from emailing you and tricking your personal digital assistant into sending them your password reset emails. No matter how you feel about “safety filters” on models, if you ever want a trustworthy digital assistant you should care about finding robust solutions for prompt injection. To be fair, this library attempts to solve both at once.
- Zambyte 3y ago
  You can avoid prompt injection by simply not using LLMs as autonomous agents where the output of the model is critical for security. That sounds like a horrible idea anyways. A language model is the wrong interface between untrusted people and sensitive data
  - simonw 3y ago
    Sure, but there are SO many things people want to build with LLMs that include access to privileged actions and sensitive data. Prompt injection means that even running an LLM against your own private notes to answer questions about them could be unsafe, provided there are any vectors (like Markdown image support) that might be used for exfiltration. My current recommendation for dealing with prompt injection is to keep it in mind and limit the blast radius if something goes wrong: https://simonwillison.net/2023/Dec/20/mitigate-prompt-injection/ https://simonwillison.net/2023/Dec/20/mitigate-prompt-inject...
    - Zambyte 3y ago
      Using prompt injection mitigation techniques is akin to directly interfacing untrusted clients to your production database, but just organizing your tables that contain sensitive data in a confusing way in the name of security. If you depend on a language model behaving correctly to avoid leaking sensitive data, you've already leaked the sensitive data. Scope the information that the language model has access to to a subset of the information that the person interfacing with the language model has access to. Prompt injection doesn't matter at that point, because the person will only be able to "leak" information they have permission to access anyways.
      - simonw 3y ago
        That's not enough. Even if the LLM can only access information that should be visible to the user interacting with it (which I see as table stakes for building anything here) you still have to worry about prompt injection exfiltration attacks. More on exfiltration: https://simonwillison.net/search/?q=exfiltration https://simonwillison.net/search/?q=exfiltration

I DO NOT like this. I do not like this very much. Why work so hard to make a technology less useful. Like a way to block certain google searches you don't agree with.

CuriouslyC 3y ago

The technology was always the same amount of useful. This library just lowers the competence floor for exploits. Try to exploit yourself and come up with anti-jailbreak prompts to mitigate the exploits.

Conasg 3y ago

There's nothing wrong with blocking prompt injection for a customer service chatbot, though. This would be obnoxious applied directly to something like ChatGPT, or worse yet their API, but I don't think that's really the intended use case.

smokeydoe 3y ago

How is it less useful to have more protection from abuse on your LLM service? This is for people implementing their own public products right?

ravroid 3y ago

You're equating this to censorship, when I think it's more like Google adding security measures so you can't break their search engine rather than removing unfavorable results.

simonw 3y ago

You may be confusing prompt injection with jailbreaking. You should care about prompt injection even if you don't care about jailbreaking: https://simonwillison.net/2024/Mar/5/prompt-injection-jailbreaking/#censorship-debate https://simonwillison.net/2024/Mar/5/prompt-injection-jailbr... Prompt injection is a security issue. It’s about preventing attackers from emailing you and tricking your personal digital assistant into sending them your password reset emails. No matter how you feel about “safety filters” on models, if you ever want a trustworthy digital assistant you should care about finding robust solutions for prompt injection. To be fair, this library attempts to solve both at once.

Zambyte 3y ago
You can avoid prompt injection by simply not using LLMs as autonomous agents where the output of the model is critical for security. That sounds like a horrible idea anyways. A language model is the wrong interface between untrusted people and sensitive data
- simonw 3y ago
  Sure, but there are SO many things people want to build with LLMs that include access to privileged actions and sensitive data. Prompt injection means that even running an LLM against your own private notes to answer questions about them could be unsafe, provided there are any vectors (like Markdown image support) that might be used for exfiltration. My current recommendation for dealing with prompt injection is to keep it in mind and limit the blast radius if something goes wrong: https://simonwillison.net/2023/Dec/20/mitigate-prompt-injection/ https://simonwillison.net/2023/Dec/20/mitigate-prompt-inject...
  - Zambyte 3y ago
    Using prompt injection mitigation techniques is akin to directly interfacing untrusted clients to your production database, but just organizing your tables that contain sensitive data in a confusing way in the name of security. If you depend on a language model behaving correctly to avoid leaking sensitive data, you've already leaked the sensitive data. Scope the information that the language model has access to to a subset of the information that the person interfacing with the language model has access to. Prompt injection doesn't matter at that point, because the person will only be able to "leak" information they have permission to access anyways.
    - simonw 3y ago
      That's not enough. Even if the LLM can only access information that should be visible to the user interacting with it (which I see as table stakes for building anything here) you still have to worry about prompt injection exfiltration attacks. More on exfiltration: https://simonwillison.net/search/?q=exfiltration https://simonwillison.net/search/?q=exfiltration