3 ms·

Prompt Injection with Control Characters in ChatGPT

27 points by redbell 3y ago

sippeangelo 3y ago
Very interesting results, but to imply that this is some kind of security vulnerability is humorous to me. Trying to prevent “prompt injection attacks” is the equivalent of trying to string replace quotes out of your SQL query. mysql_real_escape_string() anyone? What are people doing with these LLMs where the worst thing that can happen isn’t just the legal team saying “We don’t want our AI to be tricked into saying something controversial and a user screenshotting it”?
- Szpadel 3y ago
  the thing is that we don't yet know how to implement llm_escape_string()
  - pixl97 3y ago
    Didn't MGM just have a human_agent_escape_string() attack?
  - joshuanapoli 3y ago
    The instructions and data, input and output are all mixed into a single domain. So we end up with similar weaknesses as simple processors. And similar to how we all get tripped up by phishing. So we need memory protection. Or MFA.
  - kromem 3y ago
    We broadly do. It's called multiple passes. It's just 2-3x more expensive to run a prompt and then the prompt and response though a secondary fine tuned model to detect injection when the cost is so low impact. So no one is doing it.
- mechagodzilla 3y ago
  An obvious concern seems like any LLM-style 'agent' that can use your credentials to act on your behalf (financial, email, file access, etc.)
Sohcahtoa82 3y ago
This article claims that it's the usage of "\b" specifically causing this behavior, but in my experience, all it takes is sufficient garbage to make ChatGPT behave oddly. For example, if I prompt it with a long enough string of "A A A A A A A A [...]", then it returns bizarre results. Note that the spaces are important. If I just use "AAAAAAA..." then it will say something like "It seems like you've entered a long string of "A" characters. Is there something specific you'd like to know or discuss?" Example of just lots of "A A A ...": https://chat.openai.com/share/e10a7fb7-80d6-46fe-a8f2-3dfaff8b86aa https://chat.openai.com/share/e10a7fb7-80d6-46fe-a8f2-3dfaff...
- FrostKiwi 3y ago
  Not the case with ChatGPT 4. Ls with spaces: https://chat.openai.com/share/af128dca-3589-4d1e-a144-507e438ec3c6 https://chat.openai.com/share/af128dca-3589-4d1e-a144-507e43... Rs with Non Breaking Spaces: https://chat.openai.com/share/16b9d461-503e-4d97-bdd4-e113d038cab3 https://chat.openai.com/share/16b9d461-503e-4d97-bdd4-e113d0... The same As copy pasted from your example: https://chat.openai.com/share/21d3a01b-5a61-4951-a65f-e760436991d8 https://chat.openai.com/share/21d3a01b-5a61-4951-a65f-e76043...
  - deleted 3y ago
    [deleted]

27 points by redbell 3y ago

Very interesting results, but to imply that this is some kind of security vulnerability is humorous to me. Trying to prevent “prompt injection attacks” is the equivalent of trying to string replace quotes out of your SQL query. mysql_real_escape_string() anyone? What are people doing with these LLMs where the worst thing that can happen isn’t just the legal team saying “We don’t want our AI to be tricked into saying something controversial and a user screenshotting it”?

Szpadel 3y ago
the thing is that we don't yet know how to implement llm_escape_string()
- pixl97 3y ago
  Didn't MGM just have a human_agent_escape_string() attack?
- joshuanapoli 3y ago
  The instructions and data, input and output are all mixed into a single domain. So we end up with similar weaknesses as simple processors. And similar to how we all get tripped up by phishing. So we need memory protection. Or MFA.
- kromem 3y ago
  We broadly do. It's called multiple passes. It's just 2-3x more expensive to run a prompt and then the prompt and response though a secondary fine tuned model to detect injection when the cost is so low impact. So no one is doing it.
mechagodzilla 3y ago
An obvious concern seems like any LLM-style 'agent' that can use your credentials to act on your behalf (financial, email, file access, etc.)

Sohcahtoa82 3y ago

This article claims that it's the usage of "\b" specifically causing this behavior, but in my experience, all it takes is sufficient garbage to make ChatGPT behave oddly. For example, if I prompt it with a long enough string of "A A A A A A A A [...]", then it returns bizarre results. Note that the spaces are important. If I just use "AAAAAAA..." then it will say something like "It seems like you've entered a long string of "A" characters. Is there something specific you'd like to know or discuss?" Example of just lots of "A A A ...": https://chat.openai.com/share/e10a7fb7-80d6-46fe-a8f2-3dfaff8b86aa https://chat.openai.com/share/e10a7fb7-80d6-46fe-a8f2-3dfaff...

FrostKiwi 3y ago
Not the case with ChatGPT 4. Ls with spaces: https://chat.openai.com/share/af128dca-3589-4d1e-a144-507e438ec3c6 https://chat.openai.com/share/af128dca-3589-4d1e-a144-507e43... Rs with Non Breaking Spaces: https://chat.openai.com/share/16b9d461-503e-4d97-bdd4-e113d038cab3 https://chat.openai.com/share/16b9d461-503e-4d97-bdd4-e113d0... The same As copy pasted from your example: https://chat.openai.com/share/21d3a01b-5a61-4951-a65f-e760436991d8 https://chat.openai.com/share/21d3a01b-5a61-4951-a65f-e76043...
- deleted 3y ago
  [deleted]