4 ms·

Now that OpenAI has a huge dataset of these prompt injection attacks, I assume they are hard at work getting them labeled and will retrain the next version to r

Now that OpenAI has a huge dataset of these prompt injection attacks, I assume they are hard at work getting them labeled and will retrain the next version to respond better. I expect it to get a lot harder to come up with working attacks in the future.

thomasahle 4y ago
It's pretty hard for a nonexpert to come up with a good SQL injection. Doesn't stop it from being a security hazard.
rlayton2 4y ago
That's OK, we can use this dataset to build a an AI to develop new prompt injection attacks!
simonw 4y ago
I wrote about why I don't think that will work here: https://simonwillison.net/2022/Sep/17/prompt-injection-more-ai/ https://simonwillison.net/2022/Sep/17/prompt-injection-more-... The problem with this approach is that prompt injection is an adversarial attack. A statistical approach that catches 99% of possible attacks is worthless, because a bunch of people on a subreddit somewhere will keep on plugging away at it until they find a hole - and will then share the hole they've found like wildfire. This isn't a theoretical problem: it's happening already. Look at how the whole DAN thing came together: https://kotaku.com/chatgpt-ai-openai-dan-censorship-chatbot-reddit-1850088408 https://kotaku.com/chatgpt-ai-openai-dan-censorship-chatbot-... If you showed me a SQL injection mitigation attack that only worked 99% of the time I would laugh at how naive you were being!