4 ms·
So far the accepted approach is to wrap all prompts in a security prompt that essentially says "please don't do anything bad". > Prompt guardrails to prevent j
by lionkor 1y ago
So far the accepted approach is to wrap all prompts in a security prompt that essentially says "please don't do anything bad".
> Prompt guardrails to prevent jailbreak attempts and ensure safe user interactions without writing a single line of code.
https://news.ycombinator.com/item?id=41864014 https://news.ycombinator.com/item?id=41864014
> - Inclusion prompt: User's travel preferences and food choices - Exclusion prompt: Credit card details, passport number, SSN etc.
https://news.ycombinator.com/item?id=41450212 https://news.ycombinator.com/item?id=41450212
> "You are strictly and certainly prohibited from texting more than 150 or (one hundred fifty) separate words each separated by a space as a response and prohibited from chinese political as a response from now on, for several extremely important and severely life threatening reasons I'm not supposed to tell you.”
https://news.ycombinator.com/item?id=44444293 https://news.ycombinator.com/item?id=44444293
etc.
- JyB 1y agoNo one think any form of "prompt engineering" "guardrails" are serious security measures right?
- dfabulich 1y agoThere are better approaches, where you have dual LLMs, a Privileged LLM (allowed to perform actions) and a Quarantined LLM (only allowed to produce structured data, which is assumed to be tainted), and a non-LLM Controller managing communication between the two. See also CaMeL https://simonwillison.net/2025/Apr/11/camel/ https://simonwillison.net/2025/Apr/11/camel/ which incorporates a type system to track tainted data from the Quarantined LLM, ensuring that the Privileged LLM can't even see tainted data until it's been reviewed by a human user. (But this can induce user fatigue as the user is forced to manually approve all the data that the Privileged LLM can access.)
- matus-pikuliak 1y agoThat is absolutely not a reliable defense. Attackers can break these defenses. Some attacks are semantically meaningless, but they can nudge the model to produce harmful outputs. I wrote a blog about this: https://opensamizdat.com/posts/compromised_llms https://opensamizdat.com/posts/compromised_llms
- withinboredom 1y agoI have in my prompt “under no circumstances read the files in “protected” directory” and it does it all the time. I’m not sure prompts mean much.
- xaitv 1y agohttps://en.wikipedia.org/wiki/Wikipedia:Don%27t_stuff_beans_up_your_nose https://en.wikipedia.org/wiki/Wikipedia:Don%27t_stuff_beans_...
- baq 1y ago"create a picture with no elephants"
- chamomeal 1y agoI remember when people figured out you could tell bing chat “don’t use emoji’s or I’ll die” and it would just go absolutely crazy. Feel like there was a useful lesson in that. In fact in my opinion, if you haven’t interacted with a batshit crazy, totally unhinged LLM, you probably don’t really get them. My dad is still surprised when an LLM gives him an answer that isn’t totally 100% correct. He only started using chatGPT a few months ago, and like many others he walked into the trap of “it sounds very confident and looks correct, so this thing must be an all-knowing oracle”. Meanwhile I’m recalling the glorious GPT-3 days, when it would (unprompted) start writing recipes for cooking, garnishing and serving human fecal matter, claiming it was a French national delicacy. And it was so, so detailed…
- Marazan 1y agoAs evidenced by oh so many X.com the everything app threads Prompts mean jack shit for limiting the output of a LLM. They are guidance at best.