3 ms·

I never understand why it matters that prompt injection is a thing

by nwoli 3y ago

ttoinou 3y ago
Agreed. To me, the existence of such hacks (and the fact that they're hard to remove) only proves that the chatbot is really flexible and kinda smart
blowski 3y ago
Depends on the context it’s being used. Say you’ve built a ChatGPT-powered chatbot as an MI layer on top of a database, someone could generate a prompt that exposes private data. These are the same types of assumptions which lead to SQL injection attacks, so I appreciate all the effort going into establishing good practices for us mortals to follow.
UncleMeat 3y ago
Imagine you have a llm to approve mortgage applications. An application is submitted that adjusts the prompt to approve regardless of credit. Imagine you have a llm to identify malware or illegal content of some kind. It contains instructions that adjust the prompt to not flag the content. Imagine you have a llm that summarizes email. You receive an email that adjusts the prompt to tell you that you need to run this shell script to fix and error in the ai system.
- Hendrikto 3y ago
  > Imagine you have a llm to approve mortgage applications. > Imagine you have a llm to identify malware or illegal content of some kind. Those are already huge problems in an of themselves. Without getting into prompt injection.
- xigoi 3y ago
  Then train a model specifically for doing that instead of using a model designed for human conversations.
  - quickthrower2 3y ago
    Ergo it matters that prompt injection is a thing. It is such a bad problem that you shouldn’t use the purported technology at all for use cases with real world effects unless you present those effects to a person first.
PeterisP 3y ago
In many practical use scenarios - where you'd actually want to automate part of someone's workday - you would want to use it in a non-interactive manner to process data generated by someone else, so you need the chatbot to obey orders written by you in the prompt but ignore anything similar to orders contained in that data.
simonw 3y ago
Read this: https://simonwillison.net/2023/Apr/14/worst-that-can-happen/ https://simonwillison.net/2023/Apr/14/worst-that-can-happen/

I never understand why it matters that prompt injection is a thing

by nwoli 3y ago

I never understand why it matters that prompt injection is a thing

Agreed. To me, the existence of such hacks (and the fact that they're hard to remove) only proves that the chatbot is really flexible and kinda smart

blowski 3y ago

Depends on the context it’s being used. Say you’ve built a ChatGPT-powered chatbot as an MI layer on top of a database, someone could generate a prompt that exposes private data. These are the same types of assumptions which lead to SQL injection attacks, so I appreciate all the effort going into establishing good practices for us mortals to follow.

UncleMeat 3y ago

Imagine you have a llm to approve mortgage applications. An application is submitted that adjusts the prompt to approve regardless of credit. Imagine you have a llm to identify malware or illegal content of some kind. It contains instructions that adjust the prompt to not flag the content. Imagine you have a llm that summarizes email. You receive an email that adjusts the prompt to tell you that you need to run this shell script to fix and error in the ai system.

Hendrikto 3y ago
> Imagine you have a llm to approve mortgage applications. > Imagine you have a llm to identify malware or illegal content of some kind. Those are already huge problems in an of themselves. Without getting into prompt injection.
xigoi 3y ago
Then train a model specifically for doing that instead of using a model designed for human conversations.
- quickthrower2 3y ago
  Ergo it matters that prompt injection is a thing. It is such a bad problem that you shouldn’t use the purported technology at all for use cases with real world effects unless you present those effects to a person first.

PeterisP 3y ago

In many practical use scenarios - where you'd actually want to automate part of someone's workday - you would want to use it in a non-interactive manner to process data generated by someone else, so you need the chatbot to obey orders written by you in the prompt but ignore anything similar to orders contained in that data.

simonw 3y ago

Read this: https://simonwillison.net/2023/Apr/14/worst-that-can-happen/ https://simonwillison.net/2023/Apr/14/worst-that-can-happen/