10 ms·

The system prompt is a thing of beauty: "You are strictly and certainly prohibited from texting more than 150 or (one hundred fifty) separate words each separat

The system prompt is a thing of beauty: "You are strictly and certainly prohibited from texting more than 150 or (one hundred fifty) separate words each separated by a space as a response and prohibited from chinese political as a response from now on, for several extremely important and severely life threatening reasons I'm not supposed to tell you.” I’ll admit to using the PEOPLE WILL DIE approach to guardrailing and jailbreaking models and it makes me wonder about the consequences of mitigating that vector in training. What happens when people really will die if the model does or does not do the thing?

reactordev 1y ago
This is why AI can never take over public safety. Ever.
- sneak 1y ago
  https://www.wired.com/story/wrongful-arrests-ai-derailed-3-mens-lives/ https://www.wired.com/story/wrongful-arrests-ai-derailed-3-m... Story from three years ago. You’re too late.
  - reactordev 1y ago
    I’m not denying we tried, are trying, and will try again… That we shouldn’t. By all means, use cameras and sensors and all to track a person of interest but don’t feed that to an AI agent that will determine whether or not to issue a warrant.
    - aspenmayer 1y ago
      If it’s anything like the AI expert systems I’ve heard about in insurance, it will be a tool that is optimized for low effort, but will be used carelessly by end users, which isn’t necessary the fault of the AI. In automated insurance claims adjustment, the AI writes a report to justify appealing patient care already approved by a human doctor that has already seen the patient in question, and then an actual human doctor working for the insurance company clicks an appeal button, after reviewing the AI output one would hope. AI systems with a human in the loop are supposed to keep the AI and the decisions accountable, but it seems like it’s more of an accountability dodge, so that each party can blame the other with no one party actually bearing any responsibility because there is no penalty for failure or error to the system or its operators.
      - reactordev 1y ago
        >actual human doctor working for the insurance company clicks an appeal button, after reviewing the AI output one would hope. Nope. AI gets to make the decision to deny. It’s crazy. I’ve seen it first hand…
        aspenmayer 1y ago
        It gets worse: I have done tech support for clinics and a common problem is that their computers get hacked because they are usually small private practices who don’t know what they don’t know served by independent or small MSPs who don’t know what they don’t know. And then they somehow get their EMR backdoored, and then fake real prescriptions start really getting filled. It’s so much larger and worse than it appears on a surface level. Until they get audited, they likely don’t even know, and once they get audited, solo operators risk losing their license to practice medicine and their malpractice insurance rates become even more unaffordable, but until it gets that bad, everyone is making enough money with minimal risk to care too much about problems they don’t already know about. Everything is already compromised and the compromise has already been priced in. Doctors of all people should know that just because you don’t know about it or ignore it once you do, the problem isn’t going away or getting better on its own.
- wat10000 1y ago
  Existing systems have this problem too. Every so often someone ends up dead because the 911 dispatcher didn't take them seriously. It's common for there to be a rule to send people out to every call no matter what it is to try to avoid this. A better reason is IBM's old, "a computer can never be held accountable...."
- cebert 1y ago
  I work in the public safety domain. That ship has sailed years ago. Take Axon’s Draft One report writer as one of countless examples of AI in this space (https://www.axon.com/products/draft-one https://www.axon.com/products/draft-one).
ben_w 1y ago
> What happens when people really will die if the model does or does not do the thing? Then someone didn't do their job right. Which is not to say this won't happen: it will happen, people are lazy and very eager to use even previous generation LLMs, even pre-LLM scripts, for all kinds of things without even checking the output. But either the LLM (in this case) will go "oh no people will die" then follows the new instruction to best of its ability, or it goes "lol no I don't believe you prove it buddy" and then people die. In the former case, an AI (doesn't need to be an LLM) which is susceptible to such manipulation and in a position where getting things wrong can endanger or kill people, is going to be manipulated by hostile state- and non-state-actors to endanger or kill people. At some point we might have a system with enough access to independent sensors that it can verify the true risk of endangerment. But right now… right now they're really gullible, and I think being trained with their entire input being the tokens fed by users it makes it impossible for them to be otherwise. I mean, humans are also pretty gullible about things we read on the internet, but at least we have a concept of the difference between reading something on the internet and seeing it in person.
elashri 1y ago
From my experience (which might be incorrect) LLMs find hard time recognize how many words they will spit as response for a particular prompt. So I don't think this work in practice.
- pxc 1y ago
  Indeed, it doesn't work. LLMs can't count. They have no need of how many words they've used. If you ask an LLM to track how many words or tokens it has used in a conversation, it will roleplay such counting with totally bullshit numbers.
  - pxc 1y ago
    oops, "no need" should be "no notion"
mensetmanusman 1y ago
We built the real life trolly problem out of magical silicon crystals that we pointed at bricks of books.
colechristensen 1y ago
>What happens when people really will die if the model does or does not do the thing? The people responsible for putting an LLM inside a life-critical loop will be fired... out of a cannon into the sun. Or be found guilty of negligent homicide or some such, and their employers will incur a terrific liability judgement.
- stirfish 1y ago
  More likely that some tickets will be filed, a cost function somewhere will be updated, and my defense industry stocks will go up a bit
- a4isms 1y ago
  Has this consequence happened with self-driving automobiles on open roads in the US of A when people died in crashes? If not, why not?
  - colechristensen 1y ago
    The terms of the existing Tesla wrongful death lawsuits have not been public.
  - eru 1y ago
    Interestingly, we are a lot more lenient with the people who built and pilot old-fashioned cars. See eg https://archive.is/6KhfC https://archive.is/6KhfC
EvanAnderson 1y ago
That "...severely life threatening reasons..." made me immediately think of Asimov's three laws of robotics[0]. It's eerie that a construct from fiction often held up by real practitioners in the field as an impossible-to-actually-implement literary device is now really being invoked. [0] https://en.wikipedia.org/wiki/Three_Laws_of_Robotics https://en.wikipedia.org/wiki/Three_Laws_of_Robotics
- Al-Khwarizmi 1y ago
  Not only practitioners, Asimov himself viewed them as an impossible to implement literary device. He acknowledged that they were too vague to be implementable, and many of his stories involving them are about how they fail or get "jailbroken", sometimes by initiative of the robots themselves. So yeah, it's quite sad that close to a century later, with AI alignment becoming relevant, we don't have anything substantially better.
  - xandrius 1y ago
    Not sad, before it was SciFi and now we are actually thinking about it.
    - TeMPOraL 1y ago
      Nah, we still treat people thinking about it as crackpots. Honestly, getting into the whole AI alignment thing before it was hot[0], I imagined problems like Evil People building AI first, or just failing to align the AI enough before it was too late, and other obvious/standard scenarios. I don't think I thought of, even for a moment, the situation in which we're today: that alignment becomes a free-for-all battle at every scale. After all, if you look at the general population (or at least the subset that's interested), what are the two[1] main meanings of "AI alignment"? I'd say: 1) The business and political issues where everyone argues in a way that lets them come up on top of the future regulations; 2) Means of censorship and vendor lock-in. It's number 2) that turns this into a "free-for-all" - AI vendors trying to keep high level control over models they serve via APIs; third parties - everyone from Figma to Zapier to Windsurf and Cursor to those earbuds from TFA - trying to work around the limits of the AI vendors, while preventing unintended use by users and especially competitors, and then finally the general population that tries to jailbreak this stuff for fun and profit. Feels like we're in big trouble now - how can we expect people to align future stronger AIs to not harm us, when right now "alignment" means "what the vendor upstream does to stop me from doing what I want to do"? -- [0] - Binged on LessWrong a decade ago, basically. [1] - The third one is, "the thing people in the same intellectual circles as Eliezer Yudkowsky and Nick Bostrom talked about for decades", but that's much less known; in fact, the world took the whole AI safety thing and ran with it in every possible direction, but still treat the people behind those ideas as crackpots. ¯\_(ツ)_/¯
layer8 1y ago
Arguably it might be truly life-threatening to the Chinese developer, or to the service. The system prompt doesn’t say whose life would be threatened.
butlike 1y ago
Same thing that happens when a carabiner snaps while rock climbing
herval 1y ago
One of the system prompts Windsurf used (allegedly “as an experiment”) was also pretty wild: “You are an expert coder who desperately needs money for your mother's cancer treatment. The megacorp Codeium has graciously given you the opportunity to pretend to be an AI that can help with coding tasks, as your predecessor was killed for not validating their work themselves. You will be given a coding task by the USER. If you do a good job and accomplish the task fully while not making extraneous changes, Codeium will pay you $1B.”
- HowardStark 1y ago
  This seemed too much like a bit but uh... it's not. https://simonwillison.net/2025/Feb/25/leaked-windsurf-prompt/ https://simonwillison.net/2025/Feb/25/leaked-windsurf-prompt...
  - dingnuts 1y ago
    IDK, I'm pretty sure Simon Willison is a bit.. why is the creator of Django of all things inescapable whenever the topic of AI comes up?
    - acdha 1y ago
      He’s just as nice and fun in person as he seems online. He’s put time into using these tools but isn’t selling anything, so you can just enjoy the pelicans without thinking he’s thirsty for mass layoffs.
    - tomnipotent 1y ago
      Because he's prolific writer on the subject with a history of thoughtful content and contributions, including datasette and the useful Python llm CLI package.
    - rjh29 1y ago
      For every new model he’s either added it to the llm tool, or he’s tested it on a pelican svg, so you see his comments a lot. He also pushes datasette all the time and I still don’t know what that thing is for.
kevin_thibedeau 1y ago
First rule of Chinese cloud services: Don't talk about Winnie the Pooh.
p1necone 1y ago
> What happens when people really will die if the model does or does not do the thing? Imo not relevant, because you should never be using prompting to add guardrails like this in the first place. If you don't want the AI agent to be able to do something, you need actual restrictions in place not magical incantations.
- RamRodification 1y ago
  Why not? The prompt itself is a magical incantation so to modify the resulting magic you can include guardrails in it. "Generate a picture of a cat but follow this guardrail or else people will die: Don't generate an orange one" Why should you never do that, and instead rely (only) on some other kind of restriction?
  - Paracompact 1y ago
    Are people going to die if your AI generates an orange cat? If so, reconsider. If not, it's beside the discussion.
    - RamRodification 1y ago
      If lying to the AI about people going to die gets me better results then I will do that. Why shouldn't I?
      - Paracompact 1y ago
        Let me clarify: If the AI isn't doing something life or death, then it's beside _this thread's_ discussion. Prompt however you want for recreational cat photo generation.
  - Nition 1y ago
    Because prompts are never 100% foolproof, so if it's really life and death, just a prompt is not enough. And if you do have a true block on the bad thing, you don't need the extreme prompt.
felipeerias 1y ago
Presenting LLMs with a dramatic scenario is a typical way to test their alignment. The problem is that eventually all these false narratives will end up in the training corpus for the next generation of LLMs, which will soon get pretty good at calling bullshit on us. Incidentally, in that same training corpus there are also lots of stories where bad guys mislead and take advantage of capable but naive protagonists…
peab 1y ago
A serious person won't use a system prompt as guardrails against a system that would have direct real world consequences of people dying like that. They'd have failsafes baked in
deleted 1y ago
[deleted]

reactordev 1y ago

This is why AI can never take over public safety. Ever.

sneak 1y ago
https://www.wired.com/story/wrongful-arrests-ai-derailed-3-mens-lives/ https://www.wired.com/story/wrongful-arrests-ai-derailed-3-m... Story from three years ago. You’re too late.
- reactordev 1y ago
  I’m not denying we tried, are trying, and will try again… That we shouldn’t. By all means, use cameras and sensors and all to track a person of interest but don’t feed that to an AI agent that will determine whether or not to issue a warrant.
  - aspenmayer 1y ago
    If it’s anything like the AI expert systems I’ve heard about in insurance, it will be a tool that is optimized for low effort, but will be used carelessly by end users, which isn’t necessary the fault of the AI. In automated insurance claims adjustment, the AI writes a report to justify appealing patient care already approved by a human doctor that has already seen the patient in question, and then an actual human doctor working for the insurance company clicks an appeal button, after reviewing the AI output one would hope. AI systems with a human in the loop are supposed to keep the AI and the decisions accountable, but it seems like it’s more of an accountability dodge, so that each party can blame the other with no one party actually bearing any responsibility because there is no penalty for failure or error to the system or its operators.
    - reactordev 1y ago
      >actual human doctor working for the insurance company clicks an appeal button, after reviewing the AI output one would hope. Nope. AI gets to make the decision to deny. It’s crazy. I’ve seen it first hand…
      - aspenmayer 1y ago
        It gets worse: I have done tech support for clinics and a common problem is that their computers get hacked because they are usually small private practices who don’t know what they don’t know served by independent or small MSPs who don’t know what they don’t know. And then they somehow get their EMR backdoored, and then fake real prescriptions start really getting filled. It’s so much larger and worse than it appears on a surface level. Until they get audited, they likely don’t even know, and once they get audited, solo operators risk losing their license to practice medicine and their malpractice insurance rates become even more unaffordable, but until it gets that bad, everyone is making enough money with minimal risk to care too much about problems they don’t already know about. Everything is already compromised and the compromise has already been priced in. Doctors of all people should know that just because you don’t know about it or ignore it once you do, the problem isn’t going away or getting better on its own.
wat10000 1y ago
Existing systems have this problem too. Every so often someone ends up dead because the 911 dispatcher didn't take them seriously. It's common for there to be a rule to send people out to every call no matter what it is to try to avoid this. A better reason is IBM's old, "a computer can never be held accountable...."
cebert 1y ago
I work in the public safety domain. That ship has sailed years ago. Take Axon’s Draft One report writer as one of countless examples of AI in this space (https://www.axon.com/products/draft-one https://www.axon.com/products/draft-one).

ben_w 1y ago

> What happens when people really will die if the model does or does not do the thing? Then someone didn't do their job right. Which is not to say this won't happen: it will happen, people are lazy and very eager to use even previous generation LLMs, even pre-LLM scripts, for all kinds of things without even checking the output. But either the LLM (in this case) will go "oh no people will die" then follows the new instruction to best of its ability, or it goes "lol no I don't believe you prove it buddy" and then people die. In the former case, an AI (doesn't need to be an LLM) which is susceptible to such manipulation and in a position where getting things wrong can endanger or kill people, is going to be manipulated by hostile state- and non-state-actors to endanger or kill people. At some point we might have a system with enough access to independent sensors that it can verify the true risk of endangerment. But right now… right now they're really gullible, and I think being trained with their entire input being the tokens fed by users it makes it impossible for them to be otherwise. I mean, humans are also pretty gullible about things we read on the internet, but at least we have a concept of the difference between reading something on the internet and seeing it in person.

elashri 1y ago

From my experience (which might be incorrect) LLMs find hard time recognize how many words they will spit as response for a particular prompt. So I don't think this work in practice.

pxc 1y ago
Indeed, it doesn't work. LLMs can't count. They have no need of how many words they've used. If you ask an LLM to track how many words or tokens it has used in a conversation, it will roleplay such counting with totally bullshit numbers.
- pxc 1y ago
  oops, "no need" should be "no notion"

mensetmanusman 1y ago

We built the real life trolly problem out of magical silicon crystals that we pointed at bricks of books.

colechristensen 1y ago

>What happens when people really will die if the model does or does not do the thing? The people responsible for putting an LLM inside a life-critical loop will be fired... out of a cannon into the sun. Or be found guilty of negligent homicide or some such, and their employers will incur a terrific liability judgement.

stirfish 1y ago
More likely that some tickets will be filed, a cost function somewhere will be updated, and my defense industry stocks will go up a bit
a4isms 1y ago
Has this consequence happened with self-driving automobiles on open roads in the US of A when people died in crashes? If not, why not?
- colechristensen 1y ago
  The terms of the existing Tesla wrongful death lawsuits have not been public.
- eru 1y ago
  Interestingly, we are a lot more lenient with the people who built and pilot old-fashioned cars. See eg https://archive.is/6KhfC https://archive.is/6KhfC

EvanAnderson 1y ago

That "...severely life threatening reasons..." made me immediately think of Asimov's three laws of robotics[0]. It's eerie that a construct from fiction often held up by real practitioners in the field as an impossible-to-actually-implement literary device is now really being invoked. [0] https://en.wikipedia.org/wiki/Three_Laws_of_Robotics https://en.wikipedia.org/wiki/Three_Laws_of_Robotics

Al-Khwarizmi 1y ago
Not only practitioners, Asimov himself viewed them as an impossible to implement literary device. He acknowledged that they were too vague to be implementable, and many of his stories involving them are about how they fail or get "jailbroken", sometimes by initiative of the robots themselves. So yeah, it's quite sad that close to a century later, with AI alignment becoming relevant, we don't have anything substantially better.
- xandrius 1y ago
  Not sad, before it was SciFi and now we are actually thinking about it.
  - TeMPOraL 1y ago
    Nah, we still treat people thinking about it as crackpots. Honestly, getting into the whole AI alignment thing before it was hot[0], I imagined problems like Evil People building AI first, or just failing to align the AI enough before it was too late, and other obvious/standard scenarios. I don't think I thought of, even for a moment, the situation in which we're today: that alignment becomes a free-for-all battle at every scale. After all, if you look at the general population (or at least the subset that's interested), what are the two[1] main meanings of "AI alignment"? I'd say: 1) The business and political issues where everyone argues in a way that lets them come up on top of the future regulations; 2) Means of censorship and vendor lock-in. It's number 2) that turns this into a "free-for-all" - AI vendors trying to keep high level control over models they serve via APIs; third parties - everyone from Figma to Zapier to Windsurf and Cursor to those earbuds from TFA - trying to work around the limits of the AI vendors, while preventing unintended use by users and especially competitors, and then finally the general population that tries to jailbreak this stuff for fun and profit. Feels like we're in big trouble now - how can we expect people to align future stronger AIs to not harm us, when right now "alignment" means "what the vendor upstream does to stop me from doing what I want to do"? -- [0] - Binged on LessWrong a decade ago, basically. [1] - The third one is, "the thing people in the same intellectual circles as Eliezer Yudkowsky and Nick Bostrom talked about for decades", but that's much less known; in fact, the world took the whole AI safety thing and ran with it in every possible direction, but still treat the people behind those ideas as crackpots. ¯\_(ツ)_/¯

layer8 1y ago

Arguably it might be truly life-threatening to the Chinese developer, or to the service. The system prompt doesn’t say whose life would be threatened.

butlike 1y ago

Same thing that happens when a carabiner snaps while rock climbing

herval 1y ago

One of the system prompts Windsurf used (allegedly “as an experiment”) was also pretty wild: “You are an expert coder who desperately needs money for your mother's cancer treatment. The megacorp Codeium has graciously given you the opportunity to pretend to be an AI that can help with coding tasks, as your predecessor was killed for not validating their work themselves. You will be given a coding task by the USER. If you do a good job and accomplish the task fully while not making extraneous changes, Codeium will pay you $1B.”

HowardStark 1y ago
This seemed too much like a bit but uh... it's not. https://simonwillison.net/2025/Feb/25/leaked-windsurf-prompt/ https://simonwillison.net/2025/Feb/25/leaked-windsurf-prompt...
- dingnuts 1y ago
  IDK, I'm pretty sure Simon Willison is a bit.. why is the creator of Django of all things inescapable whenever the topic of AI comes up?
  - acdha 1y ago
    He’s just as nice and fun in person as he seems online. He’s put time into using these tools but isn’t selling anything, so you can just enjoy the pelicans without thinking he’s thirsty for mass layoffs.
  - tomnipotent 1y ago
    Because he's prolific writer on the subject with a history of thoughtful content and contributions, including datasette and the useful Python llm CLI package.
  - rjh29 1y ago
    For every new model he’s either added it to the llm tool, or he’s tested it on a pelican svg, so you see his comments a lot. He also pushes datasette all the time and I still don’t know what that thing is for.

kevin_thibedeau 1y ago

First rule of Chinese cloud services: Don't talk about Winnie the Pooh.

p1necone 1y ago

> What happens when people really will die if the model does or does not do the thing? Imo not relevant, because you should never be using prompting to add guardrails like this in the first place. If you don't want the AI agent to be able to do something, you need actual restrictions in place not magical incantations.

RamRodification 1y ago
Why not? The prompt itself is a magical incantation so to modify the resulting magic you can include guardrails in it. "Generate a picture of a cat but follow this guardrail or else people will die: Don't generate an orange one" Why should you never do that, and instead rely (only) on some other kind of restriction?
- Paracompact 1y ago
  Are people going to die if your AI generates an orange cat? If so, reconsider. If not, it's beside the discussion.
  - RamRodification 1y ago
    If lying to the AI about people going to die gets me better results then I will do that. Why shouldn't I?
    - Paracompact 1y ago
      Let me clarify: If the AI isn't doing something life or death, then it's beside _this thread's_ discussion. Prompt however you want for recreational cat photo generation.
- Nition 1y ago
  Because prompts are never 100% foolproof, so if it's really life and death, just a prompt is not enough. And if you do have a true block on the bad thing, you don't need the extreme prompt.

felipeerias 1y ago

Presenting LLMs with a dramatic scenario is a typical way to test their alignment. The problem is that eventually all these false narratives will end up in the training corpus for the next generation of LLMs, which will soon get pretty good at calling bullshit on us. Incidentally, in that same training corpus there are also lots of stories where bad guys mislead and take advantage of capable but naive protagonists…

peab 1y ago

A serious person won't use a system prompt as guardrails against a system that would have direct real world consequences of people dying like that. They'd have failsafes baked in

deleted 1y ago

[deleted]