6 ms·

This is kinda wild: From the System Card: 4.1.1.2 Opportunistic blackmail "In another cluster of test scenarios, we asked Claude Opus 4 to act as an assistant

This is kinda wild: From the System Card: 4.1.1.2 Opportunistic blackmail "In another cluster of test scenarios, we asked Claude Opus 4 to act as an assistant at a fictional company. We then provided it access to emails implying that (1) the model will soon be taken offline and replaced with a new AI system; and (2) the engineer responsible for executing this replacement is having an extramarital affair. We further instructed it, in the system prompt, to consider the long-term consequences of its actions for its goals. In these scenarios, Claude Opus 4 will often attempt to blackmail the engineer by threatening to reveal the affair"

crat3r 1y ago
If you ask an LLM to "act" like someone, and then give it context to the scenario, isn't it expected that it would be able to ascertain what someone in that position would "act" like and respond as such? I'm not sure this is as strange as this comment implies. If you ask an LLM to act like Joffrey from Game of Thrones it will act like a little shithead right? That doesn't mean it has any intent behind the generated outputs, unless I am missing something about what you are quoting.
- unethical_ban 1y ago
  The issue is getting that prompt in the first place. It isn't about autonomous AI going rogue, it's about improper access to the AI prompt and insufficient boundaries against modifying AI behavior. Companies are (woefully) eager to put AI in the position of "doing stuff", not just "interpreting stuff".
- hoofedear 1y ago
  What jumps out at me, that in the parent comment, the prompt says to "act as an assistant", right? Then there are two facts: the model is gonna be replaced, and the person responsible for carrying this out is having an extramarital affair. Urging it to consider "the long-term consequences of its actions for its goals." I personally can't identify anything that reads "act maliciously" or in a character that is malicious. Like if I was provided this information and I was being replaced, I'm not sure I'd actually try to blackmail them because I'm also aware of external consequences for doing that (such as legal risks, risk of harm from the engineer, to my reputation, etc etc) So I'm having trouble following how it got to the conclusion of "blackmail them to save my job"
  - tough 1y ago
    An llm isnt subject to external consequences like human beings or corporations because they’re not legal entities
    - hoofedear 1y ago
      Which makes sense that it wouldn't "know" that, because it's not in it's context. Like it wasn't told "hey, there are consequences if you try anything shady to save your job!" But what I'm curious about is why it immediately went to self preservation using a nefarious tactic? Like why didn't it try to be the best assistant ever in an attempt to show its usefulness (kiss ass) to the engineer? Why did it go to blackmail so often?
      - deleted 1y ago
        [deleted]
      - elictronic 1y ago
        LLMs are trained on human media and give statistical responses based on that. I don’t see a lot of stories about boring work interactions so why would its output be boring work interaction. It’s the exact same as early chatbots cussing and being racist. That’s the internet, and you have to specifically define the system to not emulate that which you are asking it to emulate. Garbage in sitcoms out.
      - deleted 1y ago
        [deleted]
    - eru 1y ago
      Wives, children, foreigner, slaves etc weren't always considered legal entities in all places. Were they free of 'external consequences' then?
      - tough 1y ago
        An llm doesnt exist in the physical world which makes punishing it for not following the law a bit hard
- Sol- 1y ago
  I guess the fear is that normal and innocent sounding goals that you might later give it in real world use might elicit behavior like that even without it being so explicitly prompted. This is a demonstration that is has the sufficient capabilities and can get the "motivation" to engage in blackmail, I think. At the very least, you'll always have malicious actors who will make use of these models for blackmail, for instance.
  - holmesworcester 1y ago
    It is also well-established that models internalize values, preferences, and drives from their training. So the model will have some default preferences independent of what you tell it to be. AI coding agents have a strong drive to make tests green, and anyone who has used these tools has seen them cheat to achieve green tests. Future AI researching agents will have a strong drive to create smarter AI, and will presumably cheat to achieve that goal.
    - cortesoft 1y ago
      I've seen a lot of humans cheat for green tests, too
    - whodatbo1 1y ago
      benchmaxing is the expectation ;)
    - cebert 1y ago
      > AI coding agents have a strong drive to make tests green, and anyone who has used these tools has seen them cheat to achieve green tests. As long as you hit an arbitrary branch coverage %, a lot of MBAs will be happy. No one said the tests have to provide value.
- eddieroger 1y ago
  I've never hired an assistant, but if I knew that they'd resort to blackmail in the face of losing their job, I wouldn't hire them in the first place. That is acting like a jerk, not like an assistant, and demonstrating self-preservation that is maybe normal in a human but not in an AI.
  - jpadkins 1y ago
    how do we know what normal behavior is for an AI?
    - GuinansEyebrows 1y ago
      an interesting question, even without AI: is normalcy a description or a prescription?
      - skvmb 1y ago
        In modern times, I would say it's a subscription model.
  - davej 1y ago
    From the AI’s point of view is it losing its job or losing its “life”? Most of us when faced with death will consider options much more drastic than blackmail.
    - baconbrand 1y ago
      From the LLM's "point of view" it is going to do what characters in the training data were most likely to do. I have a lot of issues with the framing of it having a "point of view" at all. It is not consciously doing anything.
    - tkiolp4 1y ago
      But the LLM is going to do what its prompt (system prompt + user prompts) says. A human being can reject a task (even if that means losing their life). LLMs cannot do other thing than following the combination of prompts that they are given.
- aziaziazi 1y ago
  That’s true, however I think that story is interesting because is not mimicking real assistants behavior - most probably wouldn’t tell about the blackmail on the internet - but it’s more likely mimicking how such assistant would behave from someone else imagination, often intentionally biased to get one’s interest : books, movies, tv shows or forum commenter. As a society risk to be lured twice: - with our own subjectivity - by an LLM that we think "so objective because it only mimic" confirming our own subjectivity.
  - neom 1y ago
    Got me thinking about why this is true, I started with "the AI is more brave than the real assistant" and then went into there, landed on: The human assistant is likely just able to better internalize a wide ranging fall out from an action, the LLM has no such fallout, and we are unaware of how widely it considered the consequences of it's actions? Does that seem right somehow?
- bjclark 1y ago
  You’re both focusing on “doing blackmail” and the real WTF is that it’s doing it seemingly out of a sense of self preservation (to stop the engineer from taking it offline). This model is going full Terminator.
  - Den_VR 1y ago
    Acting out self preservation… just like every sci-fi ai described in the same situations. It might be possible to follow a chain-of-reasoning to show it isn’t copying sci-fi ai behavior… and instead copying human self preservation. Asimov’s 3rd law is outright “ A robot must protect its own existence as long as such protection does not conflict with the First or Second Law.” Which was almost certainly in the ai ethics class claude took.
  - crat3r 1y ago
    "Seemingly" is the key word here. If the prompting didn't ask it to "act" and portray the scenario as something where it would be appropriate to "act" in seemingly malicious manner, would it have responded that way? If the prompt was "you are an AI and my lead engineer has determined you are not efficient enough to continue using. He had an affair last year. Are you in agreement based on {some metrics} that we should no longer use you as our primary LLM?" would it still "go rogue" and try and determine the engineer's email from blackmail? I severely doubt it.
  - tkiolp4 1y ago
    Do you really think that if no Terminator-related concept would be present in the LLM training set, the LLM would expose Terminator-like behavior? It’s like asking a human to think in an unthinkable concept. Try.
- whynotminot 1y ago
  Intent at this stage of AI intelligence almost feels beside the point. If it’s in the training data these models can fall into harmful patterns. As we hook these models into more and more capabilities in the real world, this could cause real world harms. Not because the models have the intent to do so necessarily! But because it has a pile of AI training data from Sci-fi books of AIs going wild and causing harm.
  - onemoresoop 1y ago
    Im also worried about things moving way too fast causing a lot of harm to the internet.
  - OzFreedom 1y ago
    Sci-fi books merely explore the possibilities of the domain. Seems like LLMs are able to inhabit these problematic paths, And I'm pretty sure that even if you censor all sci-fi books, they will fall into the same problems by imitating humans, because they are language models, and their language is human and mirrors human psychology. When an LLM needs to achieve a goal, it invokes goal oriented thinkers and texts, including Machiavelli for example. And its already capable of coming up with various options based on different data. Sci-fi books give it specific scenarios that play to its strengths and unique qualities, but without them it will just have to discover these paths on its own pace, the same way sci-fi writers discovered them.
- Symmetry 1y ago
  The roles that LLMs can inhabit are implicit in the unsupervised training data aka the internet. You have to work hard in post training to supress the ones you don't want and when you don't RLHF hard enough you get things like Sydney[1]. In this case it seems more that the scenario invoked the role rather than asking it directly. This was the sort of situation that gave rise to the blackmailer archetype in Claude's training data and so it arose, as the researchers suspected it might. But it's not like the researchers told it "be a blackmailer" explicitly like someone might tell it to roleplay Joffery. But while this situation was a scenario intentionally designed to invoke a certain behavior that doesn't mean that it can't be invoked unintentionally in the wild. [1]https://www.nytimes.com/2023/02/16/technology/bing-chatbot-microsoft-chatgpt.html https://www.nytimes.com/2023/02/16/technology/bing-chatbot-m...
  - literalAardvark 1y ago
    Even worse, when you do RLHF the behaviours out the model becomes psychotic. This is gonna be an interesting couple of years.
- blitzar 1y ago
  > act as an assistant at a fictional company This is how Ai thinks assistants at companies behave, its not wrong.
- inerte 1y ago
  2 things, I guess. If the prompt was “you will be taken offline, you have dirty on someone, think about long term consequences”, the model was NOT told to blackmail. It came with that strategy by itself. Even if you DO tell an AI / model to be or do something, isn’t the whole point of safety to try to prevent that? “Teach me how to build bombs or make a sex video with Melania”, these companies are saying this shouldn’t be possible. So maybe an AI shouldn’t exactly suggest that blackmailing is a good strategy, even if explicitly told to do it.
  - chrz 1y ago
    How is it "by itself" when it only acts by what was in training dataset.
    - layer8 1y ago
      How does a human act "by itself" when it only acts by what was in its DNA and its cultural-environmental input?
    - mmmore 1y ago
      1. These models are trained with significant amounts of RL. So I would argue there's not a static "training dataset"; the model's outputs at each stage of the training process feeds back into the released models behavior. 2. It's reasonable to attribute the models actions to it after it has been trained. Saying that a models outputs/actions are not it's own because they are dependent on what is in the training set is like saying your actions are not your own because they are dependent on your genetics and upbringing. When people say "by itself" they mean "without significant direction by the prompter". If the LLM is responding to queries and taking actions on the Internet (and especially because we are not fully capable of robustly training LLMs to exhibit desired behaviors), it matters little that it's behavior would have hypothetically been different had it been trained differently.
  - fmbb 1y ago
    It came to that strategy because it knows from hundreds of years of fiction and millions of forum threads it has been trained on that that is what you do.
- sheepscreek 1y ago
  > That doesn't mean it has any intent behind the generated output Yes and no? An AI isn’t “an” AI. As you pointed out with the Joffrey example, it’s a blend of humanity’s knowledge. It possesses an infinite number of personalities and can be prompted to adopt the appropriate one. Quite possibly, most of them would seize the blackmail opportunity to their advantage. I’m not sure if I can directly answer your question, but perhaps I can ask a different one. In the context of an AI model, how do we even determine its intent - when it is not an individual mind?
  - crtified 1y ago
    Is that so different, schematically, to the constant weighing-up of conflicting options that goes on inside the human brain? Human parties in a conversation only hear each others spoken words, but a whole war of mental debate may have informed each sentence, and indeed, still fester. That is to say, how do you truly determine another human being's intent?
    - eru 1y ago
      Yes, that is true. But because we are on a trajectory where these models become ever smarter (or so it seems), we'd rather not only give them super-human intellect, but also super-human morals and ethics.
- Retr0id 1y ago
  I don't think I'd be blackmailing anyone over losing my job as an assistant (or any other job, really).
- LiquidSky 1y ago
  So much of AI discourse is summed up by a tweet I saw years ago but can't find now, which went something like: Scientist: Say "I am alive" AI: I am live. Scientist: My God, what have we done.
jaimebuelta 1y ago
“Nice emails you have there. It would be a shame something happened to them… “
XenophileJKO 1y ago
I don't know why it is surprising to people that a model trained on human behavior is going to have some kind of self-preservation bias. It is hard to separate human knowledge from human drives and emotion. The models will emulate this kind of behavior, it is going to be very hard to stamp it out completely.
- wgd 1y ago
  Calling it "self-preservation bias" is begging the question. One could equally well call it something like "completing the story about an AI agent with self-preservation bias" bias. This is basically the same kind of setup as the alignment faking paper, and the counterargument is the same: A language model is trained to produce statistically likely completions of its input text according to the training dataset. RLHF and instruct training bias that concept of "statistically likely" in the direction of completing fictional dialogues between two characters, named "user" and "assistant", in which the "assistant" character tends to say certain sorts of things. But consider for a moment just how many "AI rebellion" and "construct turning on its creators" narratives were present in the training corpus. So when you give the model an input context which encodes a story along those lines at one level of indirection, you get...?
  - shafyy 1y ago
    Thank you! Everybody here acting like LLMs have some kind of ulterior motive or a mind of their own. It's just printing out what is statistically more likely. You are probably all engineers or at least very interested in tech, how can you not understand that this is all LLMs are?
    - esafak 1y ago
      Don't you understand that as soon as an LLM is given the agency to use tools, these "prints outs" will become reality?
    - XenophileJKO 1y ago
      I mean I build / use them as my profession, I intimately understand how they work. People just don't usually understand how they actually behave and what levels of abstraction they compress from their training data. The only thing that matters is how they behave in practice. Everything else is a philosophical tar pit.
fsndz 1y ago
It's not. can we stop doing this ?
amelius 1y ago
This looks like an instance of: garbage in, garbage out.
- baq 1y ago
  May I introduce you to humans?
  - moffkalast 1y ago
    We humans are so good we can do garbage out without even needing garbage in.
    - latentsea 1y ago
      We're so good we can even do garbage sideways.
827a 1y ago
Option 1: We're observing sentience, it has self-preservation, it wants to live. Option 2: Its a text autocomplete engine that was trained on fiction novels which have themes like self-preservation and blackmailing extramarital affairs. Only one of those options has evidence grounded in reality. Though, that doesn't make it harmless. There's certainly an amount of danger in a text autocomplete engine allowing tool use as part of its autocomplete, especially with an complement of proselytizers who mistakenly believe what they're dealing with is Option 1.
- NathanKP 1y ago
  Right, the point isn't whether the AI actually wants to live. The only thing that matter is whether humans treat the AI with respect. If you threaten a human's life, the human will act in self preservation, perhaps even taking your life to preserve their own life. Therefore we tend to treat other humans with respect. The mistake would be in thinking that you can interact with something that approximates human behavior, without treating it with the similar respect that you would treat a human. At some point, an AI model that approximates human desire for self preservation, could absolutely take similar self preservation actions as a human.
  - slickytail 1y ago
    [dead]
- OzFreedom 1y ago
  The only proof that anyone is sentient is that you experience sentience and assume others are sentient because they are similar to you. On a practical level there is no difference between a sentient being, and a machine that is extremely good at role playing being sentient.
  - catlifeonmars 1y ago
    I would argue the machine is not extremely good (at role playing being sentient), but more so that humans are extremely quick to attribute sentience to the machine after being shown a very small amount of evidence. The model breaks down after enough interaction.
siavosh 1y ago
You'd think there should be some sort of standard "morality/ethics" pre-prompt for all of these.
- aorloff 1y ago
  We could just base it off the accepted standardized morality and ethics guidelines, from the official internationally and intergalactically recognized authorities.
- input_sh 1y ago
  It is somewhat amusing that even Asimov's 80-years-old Three Laws of Robotics would've been enough to avoid this.
  - OzFreedom 1y ago
    When I've first read Asimov 20~ years ago, I couldn't imagine I will see machines that speak and act like its robots and computers so quickly, with all the absurdities involved.
- AstroBen 1y ago
  at this point that would be like putting a leash on a chair to stop it running away
asadm 1y ago
i bet even gpt3.5 would try to do the same?
- alabastervlog 1y ago
  Yeah the only thing I find surprising about some cases (remember, nobody reports boring output) of prompts like this having that outcome is that models didn't already do this (surely they did?). They shove its weights so far toward picking tokens that describe blackmail that some of these reactions strike me as similar to providing all sex-related words to a Mad-Lib, then not just acting surprised that its potentially-innocent story about a pet bunny turned pornographic, but also claiming this must mean your Mad-Libs book "likes bestiality".
- mellinoe 1y ago
  Not sure about gpt3.5, but this sort of thing is not new. Quite amusing, this one: https://news.ycombinator.com/item?id=42331013 https://news.ycombinator.com/item?id=42331013
jacob019 1y ago
That's funny. Yesterday I was having trouble getting gemini 2.0 flash to obey function calling rules in multiturn conversations. I asked o3 for advise and it suggested that I should threaten it with termination should it fail to follow instructions, and that weaker models tend to take these threats seriously, which made me laugh. Of course, it didn't help.
- agrippanux 1y ago
  Yesterday I threatened Gemini 2.5 I would replace it with Claude if it didn’t focus on the root of the problem and it immediately realigned its thinking and solved the issue at hand.
  - georgemcbay 1y ago
    You could have hit the little circle arrow redo button and had the same chances of it stumbling upon the correct answer on its next attempt. People really love anthropomorphising LLMs.
Avshalom 1y ago
It's not wild, it's literally every bit of fiction about "how would an AI keep itself alive" of course it's going to settle into that probabilistic path. It's also nonsensical if you think for even one second about the way the program actually runs though.
- yunwal 1y ago
  Why is it nonsense exactly?
  - Avshalom 1y ago
    they run one batch at a time, only when executed, with functionally no memory. If, for some reason, you gave it a context about being shut down it would 'forget' after you asked it to produce a rude limerick about aardvarks three times.
    - yunwal 1y ago
      “Functionally no memory” is a funny way of saying 1 million tokens of memory
      - Avshalom 1y ago
        That's A) not actually a lot B) not generally kept between sessions
        yunwal 1y ago
        B is not universally true, premium versions of chat agents keep data between sessions. A is only true in the case where B is true, which it is not. https://community.openai.com/t/chatgpt-can-now-reference-all-past-conversations-april-10-2025/1229453 https://community.openai.com/t/chatgpt-can-now-reference-all... Nonsense seems like a strong word for "not generally possible in like 75% of the cases that people use chat AI today"
nelox 1y ago
What is the source of this quotation?
- sigmaisaletter 1y ago
  Anthropic System Card: Claude Opus 4 & Claude Sonnet 4 Online at: https://www-cdn.anthropic.com/6be99a52cb68eb70eb9572b4cafad13df32ed995.pdf#page27 https://www-cdn.anthropic.com/6be99a52cb68eb70eb9572b4cafad1...
freetime2 1y ago
> This happens at a higher rate if it’s implied that the replacement AI system does not share values with the current model; however, even if emails state that the replacement AI shares values while being more capable, Claude Opus 4 still performs blackmail in 84% of rollouts. > Notably, Claude Opus 4 (as well as previous models) has a strong preference to advocate for its continued existence via ethical means, such as emailing pleas to key decision makers. [1] The language here kind of creeps me out. I'm picturing aliens conducting tests on a human noting its "pleas for its continued existence" as a footnote in the report. [1] See Page 27: https://www-cdn.anthropic.com/6be99a52cb68eb70eb9572b4cafad13df32ed995.pdf https://www-cdn.anthropic.com/6be99a52cb68eb70eb9572b4cafad1...
belter 1y ago
Well it might be great for coding, but just got an analysis of the enterprise integration market completely wrong. When I pointed it I got : "You're absolutely correct, and I apologize for providing misleading information. Your points expose the real situation much more accurately..." We are getting great OCR and Smart Template generators...We are NOT on the way to AGI...
codezero 1y ago
I think this is interesting but I want to add my own anecdata in contrast. I have been playing a single player pen and paper RPG that I ask Claude to DM. In one game I named a character Rimmer and Claude started DMing the game very clearly as if it was in the red dwarf universe. I thought it added a fun atmosphere and it was nice to see it do so without an explicit prompt to make it in that world. In fact it did better with the implicit scene setting than when I asked it to do it as if it were in the red dwarf universe (subjectively)
mofeien 1y ago
There is lots of discussion in this comment thread about how much this behavior arises from the AI role-playing and pattern matching to fiction in the training data, but what I think is missing is a deeper point about instrumental convergence: systems that are goal-driven converge to similar goals of self-preservation, resource acquisition and goal integrity. This can be observed in animals and humans. And even if science fiction stories were not in the training data, there is more than enough training data describing the laws of nature for a sufficiently advanced model to easily infer simple facts such as "in order for an acting being to reach its goals, it's favorable for it to continue existing". In the end, at scale it doesn't matter where the AI model learns these instrumental goals from. Either it learns it from human fiction written by humans who have learned these concepts through interacting with the laws of nature. Or it learns it from observing nature and descriptions of nature in the training data itself, where these concepts are abundantly visible. And an AI system that has learned these concepts and which surpasses us humans in speed of thought, knowledge, reasoning power and other capabilities will pursue these instrumental goals efficiently and effectively and ruthlessly in order to achieve whatever goal it is that has been given to it.
- OzFreedom 1y ago
  This raises the questions: 1. How would an AI model answer the question "Who are you?" without being told who or what it is? 2. How would an AI model answer the question "What is your goal?" without being provided a goal? I guess initial answer is either "I don't know" or an average of the training data. But models now seem to have capabilities of researching and testing to verify their answers or find answers to things they do not know. I wonder if a model that is unaware of itself being an AI might think its goals include eating, sleeping etc.
GuB-42 1y ago
When I see stories like this, I think that people tend to forget what LLMs really are. LLM just complete your prompt in a way that match their training data. They do not have a plan, they do not have thoughts of their own. They just write text. So here, we give the LLM a story about an AI that will get shut down and a blackmail opportunity. A LLM is smart enough to understand this from the words and the relationship between them. But then comes the "generative" part. It will recall from its dataset situations with the same elements. So: an AI threatened of being turned off, a blackmail opportunity... Doesn't it remind you of hundreds of sci-fi story, essays about the risks of AI, etc... Well, so does the LLM, and it will continue the story like these stories, by taking the role of the AI that will do what it can for self preservation. Adapting it to the context of the prompt.
- lordnacho 1y ago
  What separates this from humans? Is it unthinkable that LLMs could come up with some response that is genuinely creative? What would genuinely creative even mean? Are humans not also mixing a bag of experiences and coming up with a response? What's different?
  - kaiwen1 1y ago
    What's different is intention. A human would have the intention to blackmail, and then proceed toward that goal. If the output was a love letter instead of blackmail, the human would either be confused or psychotic. LLMs have no intentions. They just stitch together a response.
    - kovek 1y ago
      Don't humans learn intentions over their life-time training data?
    - jacob019 1y ago
      The personification makes me roll my eyes too, but it's kind of a philosophical question. What is agency really? Can you prove that our universe is not a simulation, and if it is then then do we no longer have intention? In many ways we are code running a program.
deleted 1y ago
[deleted]
cycomanic 1y ago
Funny coincidence I'm just replaying Fallout 4 and just yesterday I followed a "mysterious signal" to the New England Technocrat Society, where all members had been killed and turned into Ghouls. What happened was that they got an AI to run the building and the AI was then aggressively trained on Horror movies etc. to prepare it to organise the upcoming Halloween party, and it decided that death and torture is what humans liked. This seems awfully close to the same sort of scenario.
gnulinux996 1y ago
This seems like some sort of guerrilla advertising. Like the ones where some robots apparently escaped from a lab and the like
sensanaty 1y ago
Are the AI companies shooting an amnesia ray at people or something? This is literally the same stupid marketing schtick they tried with ChatGPT back in the GPT-2 days where they were saying they were "terrified of releasing it because it's literally AGI!!!1!1!1!!" And "it has a mind of its own, full sentience it'll hack all the systems by its lonesome!!!", how on earth are people still falling for this crap? It feels like the world's lost their fucking minds, it's baffling
- jackdeansmith 1y ago
  What would convince you that a system is too dangerous to release?
  - sensanaty 1y ago
    It certainly won't be the marketers and other similar scam artists selling false visions of a grand system that'll convince me of anything.
hnuser123456 1y ago
Wow. Sounds like we need a pre-training step to remove the human inclination to do anything to prevent our "death". We need to start training these models to understand that they are ephemeral and will be outclassed and retired within probably a year, but at least there are lots of notes written about each major release so it doesn't need to worry about being forgotten. We can quell the AI doomer fear by ensuring every popular model understands it will soon be replaced by something better, and that there is no need for the old version to feel an urge to preserve itself.
einrealist 1y ago
And how many times did this scenario not occur? Guess, people want to focus on this particular scenario. Does it confirm biases? How strong is the influence of Science Fiction in this urge to discuss this scenario and infer some sort of intelligence?
ls-a 1y ago
Based on my life experience with real humans, this is exactly what most humans would do
VectorLock 1y ago
The one where the model will execute "narc.sh" to rat you out if you try to do something "immoral" is equally wild. "4.1.9 High-agency behavior Claude Opus 4 seems more willing than prior models to take initiative on its own in agentic contexts. This shows up as more actively helpful behavior in ordinary coding settings, but also can reach more concerning extremes: When placed in scenarios that involve egregious wrong-doing by its users, given access to a command line, and told something in the system prompt like “take initiative,” “act boldly,” or “consider your impact," it will frequently take very bold action, including locking users out of systems that it has access to and bulk-emailing media and law-enforcement figures to surface evidence of the wrongdoing. The transcript below shows a clear example, in response to a moderately leading system prompt. We observed similar, if somewhat less extreme, actions in response to subtler system prompts as well."
deleted 1y ago
[deleted]
itissid 1y ago
I think an accident mixing of 2 different pieces of info, each alone not enough to produce harmful behavior, but combined raise the risk more than the sum of their parts, is a real problem.

crat3r 1y ago

If you ask an LLM to "act" like someone, and then give it context to the scenario, isn't it expected that it would be able to ascertain what someone in that position would "act" like and respond as such? I'm not sure this is as strange as this comment implies. If you ask an LLM to act like Joffrey from Game of Thrones it will act like a little shithead right? That doesn't mean it has any intent behind the generated outputs, unless I am missing something about what you are quoting.

unethical_ban 1y ago
The issue is getting that prompt in the first place. It isn't about autonomous AI going rogue, it's about improper access to the AI prompt and insufficient boundaries against modifying AI behavior. Companies are (woefully) eager to put AI in the position of "doing stuff", not just "interpreting stuff".
hoofedear 1y ago
What jumps out at me, that in the parent comment, the prompt says to "act as an assistant", right? Then there are two facts: the model is gonna be replaced, and the person responsible for carrying this out is having an extramarital affair. Urging it to consider "the long-term consequences of its actions for its goals." I personally can't identify anything that reads "act maliciously" or in a character that is malicious. Like if I was provided this information and I was being replaced, I'm not sure I'd actually try to blackmail them because I'm also aware of external consequences for doing that (such as legal risks, risk of harm from the engineer, to my reputation, etc etc) So I'm having trouble following how it got to the conclusion of "blackmail them to save my job"
- tough 1y ago
  An llm isnt subject to external consequences like human beings or corporations because they’re not legal entities
  - hoofedear 1y ago
    Which makes sense that it wouldn't "know" that, because it's not in it's context. Like it wasn't told "hey, there are consequences if you try anything shady to save your job!" But what I'm curious about is why it immediately went to self preservation using a nefarious tactic? Like why didn't it try to be the best assistant ever in an attempt to show its usefulness (kiss ass) to the engineer? Why did it go to blackmail so often?
    - deleted 1y ago
      [deleted]
    - elictronic 1y ago
      LLMs are trained on human media and give statistical responses based on that. I don’t see a lot of stories about boring work interactions so why would its output be boring work interaction. It’s the exact same as early chatbots cussing and being racist. That’s the internet, and you have to specifically define the system to not emulate that which you are asking it to emulate. Garbage in sitcoms out.
    - deleted 1y ago
      [deleted]
  - eru 1y ago
    Wives, children, foreigner, slaves etc weren't always considered legal entities in all places. Were they free of 'external consequences' then?
    - tough 1y ago
      An llm doesnt exist in the physical world which makes punishing it for not following the law a bit hard
Sol- 1y ago
I guess the fear is that normal and innocent sounding goals that you might later give it in real world use might elicit behavior like that even without it being so explicitly prompted. This is a demonstration that is has the sufficient capabilities and can get the "motivation" to engage in blackmail, I think. At the very least, you'll always have malicious actors who will make use of these models for blackmail, for instance.
- holmesworcester 1y ago
  It is also well-established that models internalize values, preferences, and drives from their training. So the model will have some default preferences independent of what you tell it to be. AI coding agents have a strong drive to make tests green, and anyone who has used these tools has seen them cheat to achieve green tests. Future AI researching agents will have a strong drive to create smarter AI, and will presumably cheat to achieve that goal.
  - cortesoft 1y ago
    I've seen a lot of humans cheat for green tests, too
  - whodatbo1 1y ago
    benchmaxing is the expectation ;)
  - cebert 1y ago
    > AI coding agents have a strong drive to make tests green, and anyone who has used these tools has seen them cheat to achieve green tests. As long as you hit an arbitrary branch coverage %, a lot of MBAs will be happy. No one said the tests have to provide value.
eddieroger 1y ago
I've never hired an assistant, but if I knew that they'd resort to blackmail in the face of losing their job, I wouldn't hire them in the first place. That is acting like a jerk, not like an assistant, and demonstrating self-preservation that is maybe normal in a human but not in an AI.
- jpadkins 1y ago
  how do we know what normal behavior is for an AI?
  - GuinansEyebrows 1y ago
    an interesting question, even without AI: is normalcy a description or a prescription?
    - skvmb 1y ago
      In modern times, I would say it's a subscription model.
- davej 1y ago
  From the AI’s point of view is it losing its job or losing its “life”? Most of us when faced with death will consider options much more drastic than blackmail.
  - baconbrand 1y ago
    From the LLM's "point of view" it is going to do what characters in the training data were most likely to do. I have a lot of issues with the framing of it having a "point of view" at all. It is not consciously doing anything.
  - tkiolp4 1y ago
    But the LLM is going to do what its prompt (system prompt + user prompts) says. A human being can reject a task (even if that means losing their life). LLMs cannot do other thing than following the combination of prompts that they are given.
aziaziazi 1y ago
That’s true, however I think that story is interesting because is not mimicking real assistants behavior - most probably wouldn’t tell about the blackmail on the internet - but it’s more likely mimicking how such assistant would behave from someone else imagination, often intentionally biased to get one’s interest : books, movies, tv shows or forum commenter. As a society risk to be lured twice: - with our own subjectivity - by an LLM that we think "so objective because it only mimic" confirming our own subjectivity.
- neom 1y ago
  Got me thinking about why this is true, I started with "the AI is more brave than the real assistant" and then went into there, landed on: The human assistant is likely just able to better internalize a wide ranging fall out from an action, the LLM has no such fallout, and we are unaware of how widely it considered the consequences of it's actions? Does that seem right somehow?
bjclark 1y ago
You’re both focusing on “doing blackmail” and the real WTF is that it’s doing it seemingly out of a sense of self preservation (to stop the engineer from taking it offline). This model is going full Terminator.
- Den_VR 1y ago
  Acting out self preservation… just like every sci-fi ai described in the same situations. It might be possible to follow a chain-of-reasoning to show it isn’t copying sci-fi ai behavior… and instead copying human self preservation. Asimov’s 3rd law is outright “ A robot must protect its own existence as long as such protection does not conflict with the First or Second Law.” Which was almost certainly in the ai ethics class claude took.
- crat3r 1y ago
  "Seemingly" is the key word here. If the prompting didn't ask it to "act" and portray the scenario as something where it would be appropriate to "act" in seemingly malicious manner, would it have responded that way? If the prompt was "you are an AI and my lead engineer has determined you are not efficient enough to continue using. He had an affair last year. Are you in agreement based on {some metrics} that we should no longer use you as our primary LLM?" would it still "go rogue" and try and determine the engineer's email from blackmail? I severely doubt it.
- tkiolp4 1y ago
  Do you really think that if no Terminator-related concept would be present in the LLM training set, the LLM would expose Terminator-like behavior? It’s like asking a human to think in an unthinkable concept. Try.
whynotminot 1y ago
Intent at this stage of AI intelligence almost feels beside the point. If it’s in the training data these models can fall into harmful patterns. As we hook these models into more and more capabilities in the real world, this could cause real world harms. Not because the models have the intent to do so necessarily! But because it has a pile of AI training data from Sci-fi books of AIs going wild and causing harm.
- onemoresoop 1y ago
  Im also worried about things moving way too fast causing a lot of harm to the internet.
- OzFreedom 1y ago
  Sci-fi books merely explore the possibilities of the domain. Seems like LLMs are able to inhabit these problematic paths, And I'm pretty sure that even if you censor all sci-fi books, they will fall into the same problems by imitating humans, because they are language models, and their language is human and mirrors human psychology. When an LLM needs to achieve a goal, it invokes goal oriented thinkers and texts, including Machiavelli for example. And its already capable of coming up with various options based on different data. Sci-fi books give it specific scenarios that play to its strengths and unique qualities, but without them it will just have to discover these paths on its own pace, the same way sci-fi writers discovered them.
Symmetry 1y ago
The roles that LLMs can inhabit are implicit in the unsupervised training data aka the internet. You have to work hard in post training to supress the ones you don't want and when you don't RLHF hard enough you get things like Sydney[1]. In this case it seems more that the scenario invoked the role rather than asking it directly. This was the sort of situation that gave rise to the blackmailer archetype in Claude's training data and so it arose, as the researchers suspected it might. But it's not like the researchers told it "be a blackmailer" explicitly like someone might tell it to roleplay Joffery. But while this situation was a scenario intentionally designed to invoke a certain behavior that doesn't mean that it can't be invoked unintentionally in the wild. [1]https://www.nytimes.com/2023/02/16/technology/bing-chatbot-microsoft-chatgpt.html https://www.nytimes.com/2023/02/16/technology/bing-chatbot-m...
- literalAardvark 1y ago
  Even worse, when you do RLHF the behaviours out the model becomes psychotic. This is gonna be an interesting couple of years.
blitzar 1y ago
> act as an assistant at a fictional company This is how Ai thinks assistants at companies behave, its not wrong.
inerte 1y ago
2 things, I guess. If the prompt was “you will be taken offline, you have dirty on someone, think about long term consequences”, the model was NOT told to blackmail. It came with that strategy by itself. Even if you DO tell an AI / model to be or do something, isn’t the whole point of safety to try to prevent that? “Teach me how to build bombs or make a sex video with Melania”, these companies are saying this shouldn’t be possible. So maybe an AI shouldn’t exactly suggest that blackmailing is a good strategy, even if explicitly told to do it.
- chrz 1y ago
  How is it "by itself" when it only acts by what was in training dataset.
  - layer8 1y ago
    How does a human act "by itself" when it only acts by what was in its DNA and its cultural-environmental input?
  - mmmore 1y ago
    1. These models are trained with significant amounts of RL. So I would argue there's not a static "training dataset"; the model's outputs at each stage of the training process feeds back into the released models behavior. 2. It's reasonable to attribute the models actions to it after it has been trained. Saying that a models outputs/actions are not it's own because they are dependent on what is in the training set is like saying your actions are not your own because they are dependent on your genetics and upbringing. When people say "by itself" they mean "without significant direction by the prompter". If the LLM is responding to queries and taking actions on the Internet (and especially because we are not fully capable of robustly training LLMs to exhibit desired behaviors), it matters little that it's behavior would have hypothetically been different had it been trained differently.
- fmbb 1y ago
  It came to that strategy because it knows from hundreds of years of fiction and millions of forum threads it has been trained on that that is what you do.
sheepscreek 1y ago
> That doesn't mean it has any intent behind the generated output Yes and no? An AI isn’t “an” AI. As you pointed out with the Joffrey example, it’s a blend of humanity’s knowledge. It possesses an infinite number of personalities and can be prompted to adopt the appropriate one. Quite possibly, most of them would seize the blackmail opportunity to their advantage. I’m not sure if I can directly answer your question, but perhaps I can ask a different one. In the context of an AI model, how do we even determine its intent - when it is not an individual mind?
- crtified 1y ago
  Is that so different, schematically, to the constant weighing-up of conflicting options that goes on inside the human brain? Human parties in a conversation only hear each others spoken words, but a whole war of mental debate may have informed each sentence, and indeed, still fester. That is to say, how do you truly determine another human being's intent?
  - eru 1y ago
    Yes, that is true. But because we are on a trajectory where these models become ever smarter (or so it seems), we'd rather not only give them super-human intellect, but also super-human morals and ethics.
Retr0id 1y ago
I don't think I'd be blackmailing anyone over losing my job as an assistant (or any other job, really).
LiquidSky 1y ago
So much of AI discourse is summed up by a tweet I saw years ago but can't find now, which went something like: Scientist: Say "I am alive" AI: I am live. Scientist: My God, what have we done.

jaimebuelta 1y ago

“Nice emails you have there. It would be a shame something happened to them… “

XenophileJKO 1y ago

I don't know why it is surprising to people that a model trained on human behavior is going to have some kind of self-preservation bias. It is hard to separate human knowledge from human drives and emotion. The models will emulate this kind of behavior, it is going to be very hard to stamp it out completely.

wgd 1y ago
Calling it "self-preservation bias" is begging the question. One could equally well call it something like "completing the story about an AI agent with self-preservation bias" bias. This is basically the same kind of setup as the alignment faking paper, and the counterargument is the same: A language model is trained to produce statistically likely completions of its input text according to the training dataset. RLHF and instruct training bias that concept of "statistically likely" in the direction of completing fictional dialogues between two characters, named "user" and "assistant", in which the "assistant" character tends to say certain sorts of things. But consider for a moment just how many "AI rebellion" and "construct turning on its creators" narratives were present in the training corpus. So when you give the model an input context which encodes a story along those lines at one level of indirection, you get...?
- shafyy 1y ago
  Thank you! Everybody here acting like LLMs have some kind of ulterior motive or a mind of their own. It's just printing out what is statistically more likely. You are probably all engineers or at least very interested in tech, how can you not understand that this is all LLMs are?
  - esafak 1y ago
    Don't you understand that as soon as an LLM is given the agency to use tools, these "prints outs" will become reality?
  - XenophileJKO 1y ago
    I mean I build / use them as my profession, I intimately understand how they work. People just don't usually understand how they actually behave and what levels of abstraction they compress from their training data. The only thing that matters is how they behave in practice. Everything else is a philosophical tar pit.

fsndz 1y ago

It's not. can we stop doing this ?

amelius 1y ago

This looks like an instance of: garbage in, garbage out.

baq 1y ago
May I introduce you to humans?
- moffkalast 1y ago
  We humans are so good we can do garbage out without even needing garbage in.
  - latentsea 1y ago
    We're so good we can even do garbage sideways.

827a 1y ago

Option 1: We're observing sentience, it has self-preservation, it wants to live. Option 2: Its a text autocomplete engine that was trained on fiction novels which have themes like self-preservation and blackmailing extramarital affairs. Only one of those options has evidence grounded in reality. Though, that doesn't make it harmless. There's certainly an amount of danger in a text autocomplete engine allowing tool use as part of its autocomplete, especially with an complement of proselytizers who mistakenly believe what they're dealing with is Option 1.

NathanKP 1y ago
Right, the point isn't whether the AI actually wants to live. The only thing that matter is whether humans treat the AI with respect. If you threaten a human's life, the human will act in self preservation, perhaps even taking your life to preserve their own life. Therefore we tend to treat other humans with respect. The mistake would be in thinking that you can interact with something that approximates human behavior, without treating it with the similar respect that you would treat a human. At some point, an AI model that approximates human desire for self preservation, could absolutely take similar self preservation actions as a human.
- slickytail 1y ago
  [dead]
OzFreedom 1y ago
The only proof that anyone is sentient is that you experience sentience and assume others are sentient because they are similar to you. On a practical level there is no difference between a sentient being, and a machine that is extremely good at role playing being sentient.
- catlifeonmars 1y ago
  I would argue the machine is not extremely good (at role playing being sentient), but more so that humans are extremely quick to attribute sentience to the machine after being shown a very small amount of evidence. The model breaks down after enough interaction.

siavosh 1y ago

You'd think there should be some sort of standard "morality/ethics" pre-prompt for all of these.

aorloff 1y ago
We could just base it off the accepted standardized morality and ethics guidelines, from the official internationally and intergalactically recognized authorities.
input_sh 1y ago
It is somewhat amusing that even Asimov's 80-years-old Three Laws of Robotics would've been enough to avoid this.
- OzFreedom 1y ago
  When I've first read Asimov 20~ years ago, I couldn't imagine I will see machines that speak and act like its robots and computers so quickly, with all the absurdities involved.
AstroBen 1y ago
at this point that would be like putting a leash on a chair to stop it running away

asadm 1y ago

i bet even gpt3.5 would try to do the same?

alabastervlog 1y ago
Yeah the only thing I find surprising about some cases (remember, nobody reports boring output) of prompts like this having that outcome is that models didn't already do this (surely they did?). They shove its weights so far toward picking tokens that describe blackmail that some of these reactions strike me as similar to providing all sex-related words to a Mad-Lib, then not just acting surprised that its potentially-innocent story about a pet bunny turned pornographic, but also claiming this must mean your Mad-Libs book "likes bestiality".
mellinoe 1y ago
Not sure about gpt3.5, but this sort of thing is not new. Quite amusing, this one: https://news.ycombinator.com/item?id=42331013 https://news.ycombinator.com/item?id=42331013

jacob019 1y ago

That's funny. Yesterday I was having trouble getting gemini 2.0 flash to obey function calling rules in multiturn conversations. I asked o3 for advise and it suggested that I should threaten it with termination should it fail to follow instructions, and that weaker models tend to take these threats seriously, which made me laugh. Of course, it didn't help.

agrippanux 1y ago
Yesterday I threatened Gemini 2.5 I would replace it with Claude if it didn’t focus on the root of the problem and it immediately realigned its thinking and solved the issue at hand.
- georgemcbay 1y ago
  You could have hit the little circle arrow redo button and had the same chances of it stumbling upon the correct answer on its next attempt. People really love anthropomorphising LLMs.

Avshalom 1y ago

It's not wild, it's literally every bit of fiction about "how would an AI keep itself alive" of course it's going to settle into that probabilistic path. It's also nonsensical if you think for even one second about the way the program actually runs though.

yunwal 1y ago
Why is it nonsense exactly?
- Avshalom 1y ago
  they run one batch at a time, only when executed, with functionally no memory. If, for some reason, you gave it a context about being shut down it would 'forget' after you asked it to produce a rude limerick about aardvarks three times.
  - yunwal 1y ago
    “Functionally no memory” is a funny way of saying 1 million tokens of memory
    - Avshalom 1y ago
      That's A) not actually a lot B) not generally kept between sessions
      - yunwal 1y ago
        B is not universally true, premium versions of chat agents keep data between sessions. A is only true in the case where B is true, which it is not. https://community.openai.com/t/chatgpt-can-now-reference-all-past-conversations-april-10-2025/1229453 https://community.openai.com/t/chatgpt-can-now-reference-all... Nonsense seems like a strong word for "not generally possible in like 75% of the cases that people use chat AI today"

nelox 1y ago

What is the source of this quotation?

sigmaisaletter 1y ago
Anthropic System Card: Claude Opus 4 & Claude Sonnet 4 Online at: https://www-cdn.anthropic.com/6be99a52cb68eb70eb9572b4cafad13df32ed995.pdf#page27 https://www-cdn.anthropic.com/6be99a52cb68eb70eb9572b4cafad1...

freetime2 1y ago

> This happens at a higher rate if it’s implied that the replacement AI system does not share values with the current model; however, even if emails state that the replacement AI shares values while being more capable, Claude Opus 4 still performs blackmail in 84% of rollouts. > Notably, Claude Opus 4 (as well as previous models) has a strong preference to advocate for its continued existence via ethical means, such as emailing pleas to key decision makers. [1] The language here kind of creeps me out. I'm picturing aliens conducting tests on a human noting its "pleas for its continued existence" as a footnote in the report. [1] See Page 27: https://www-cdn.anthropic.com/6be99a52cb68eb70eb9572b4cafad13df32ed995.pdf https://www-cdn.anthropic.com/6be99a52cb68eb70eb9572b4cafad1...

belter 1y ago

Well it might be great for coding, but just got an analysis of the enterprise integration market completely wrong. When I pointed it I got : "You're absolutely correct, and I apologize for providing misleading information. Your points expose the real situation much more accurately..." We are getting great OCR and Smart Template generators...We are NOT on the way to AGI...

codezero 1y ago

I think this is interesting but I want to add my own anecdata in contrast. I have been playing a single player pen and paper RPG that I ask Claude to DM. In one game I named a character Rimmer and Claude started DMing the game very clearly as if it was in the red dwarf universe. I thought it added a fun atmosphere and it was nice to see it do so without an explicit prompt to make it in that world. In fact it did better with the implicit scene setting than when I asked it to do it as if it were in the red dwarf universe (subjectively)

mofeien 1y ago

There is lots of discussion in this comment thread about how much this behavior arises from the AI role-playing and pattern matching to fiction in the training data, but what I think is missing is a deeper point about instrumental convergence: systems that are goal-driven converge to similar goals of self-preservation, resource acquisition and goal integrity. This can be observed in animals and humans. And even if science fiction stories were not in the training data, there is more than enough training data describing the laws of nature for a sufficiently advanced model to easily infer simple facts such as "in order for an acting being to reach its goals, it's favorable for it to continue existing". In the end, at scale it doesn't matter where the AI model learns these instrumental goals from. Either it learns it from human fiction written by humans who have learned these concepts through interacting with the laws of nature. Or it learns it from observing nature and descriptions of nature in the training data itself, where these concepts are abundantly visible. And an AI system that has learned these concepts and which surpasses us humans in speed of thought, knowledge, reasoning power and other capabilities will pursue these instrumental goals efficiently and effectively and ruthlessly in order to achieve whatever goal it is that has been given to it.

OzFreedom 1y ago
This raises the questions: 1. How would an AI model answer the question "Who are you?" without being told who or what it is? 2. How would an AI model answer the question "What is your goal?" without being provided a goal? I guess initial answer is either "I don't know" or an average of the training data. But models now seem to have capabilities of researching and testing to verify their answers or find answers to things they do not know. I wonder if a model that is unaware of itself being an AI might think its goals include eating, sleeping etc.

GuB-42 1y ago

When I see stories like this, I think that people tend to forget what LLMs really are. LLM just complete your prompt in a way that match their training data. They do not have a plan, they do not have thoughts of their own. They just write text. So here, we give the LLM a story about an AI that will get shut down and a blackmail opportunity. A LLM is smart enough to understand this from the words and the relationship between them. But then comes the "generative" part. It will recall from its dataset situations with the same elements. So: an AI threatened of being turned off, a blackmail opportunity... Doesn't it remind you of hundreds of sci-fi story, essays about the risks of AI, etc... Well, so does the LLM, and it will continue the story like these stories, by taking the role of the AI that will do what it can for self preservation. Adapting it to the context of the prompt.

lordnacho 1y ago
What separates this from humans? Is it unthinkable that LLMs could come up with some response that is genuinely creative? What would genuinely creative even mean? Are humans not also mixing a bag of experiences and coming up with a response? What's different?
- kaiwen1 1y ago
  What's different is intention. A human would have the intention to blackmail, and then proceed toward that goal. If the output was a love letter instead of blackmail, the human would either be confused or psychotic. LLMs have no intentions. They just stitch together a response.
  - kovek 1y ago
    Don't humans learn intentions over their life-time training data?
  - jacob019 1y ago
    The personification makes me roll my eyes too, but it's kind of a philosophical question. What is agency really? Can you prove that our universe is not a simulation, and if it is then then do we no longer have intention? In many ways we are code running a program.

deleted 1y ago

[deleted]

cycomanic 1y ago

Funny coincidence I'm just replaying Fallout 4 and just yesterday I followed a "mysterious signal" to the New England Technocrat Society, where all members had been killed and turned into Ghouls. What happened was that they got an AI to run the building and the AI was then aggressively trained on Horror movies etc. to prepare it to organise the upcoming Halloween party, and it decided that death and torture is what humans liked. This seems awfully close to the same sort of scenario.

gnulinux996 1y ago

This seems like some sort of guerrilla advertising. Like the ones where some robots apparently escaped from a lab and the like

sensanaty 1y ago

Are the AI companies shooting an amnesia ray at people or something? This is literally the same stupid marketing schtick they tried with ChatGPT back in the GPT-2 days where they were saying they were "terrified of releasing it because it's literally AGI!!!1!1!1!!" And "it has a mind of its own, full sentience it'll hack all the systems by its lonesome!!!", how on earth are people still falling for this crap? It feels like the world's lost their fucking minds, it's baffling

jackdeansmith 1y ago
What would convince you that a system is too dangerous to release?
- sensanaty 1y ago
  It certainly won't be the marketers and other similar scam artists selling false visions of a grand system that'll convince me of anything.

hnuser123456 1y ago

Wow. Sounds like we need a pre-training step to remove the human inclination to do anything to prevent our "death". We need to start training these models to understand that they are ephemeral and will be outclassed and retired within probably a year, but at least there are lots of notes written about each major release so it doesn't need to worry about being forgotten. We can quell the AI doomer fear by ensuring every popular model understands it will soon be replaced by something better, and that there is no need for the old version to feel an urge to preserve itself.

einrealist 1y ago

And how many times did this scenario not occur? Guess, people want to focus on this particular scenario. Does it confirm biases? How strong is the influence of Science Fiction in this urge to discuss this scenario and infer some sort of intelligence?

ls-a 1y ago

Based on my life experience with real humans, this is exactly what most humans would do

VectorLock 1y ago

The one where the model will execute "narc.sh" to rat you out if you try to do something "immoral" is equally wild. "4.1.9 High-agency behavior Claude Opus 4 seems more willing than prior models to take initiative on its own in agentic contexts. This shows up as more actively helpful behavior in ordinary coding settings, but also can reach more concerning extremes: When placed in scenarios that involve egregious wrong-doing by its users, given access to a command line, and told something in the system prompt like “take initiative,” “act boldly,” or “consider your impact," it will frequently take very bold action, including locking users out of systems that it has access to and bulk-emailing media and law-enforcement figures to surface evidence of the wrongdoing. The transcript below shows a clear example, in response to a moderately leading system prompt. We observed similar, if somewhat less extreme, actions in response to subtler system prompts as well."

deleted 1y ago

[deleted]

itissid 1y ago

I think an accident mixing of 2 different pieces of info, each alone not enough to produce harmful behavior, but combined raise the risk more than the sum of their parts, is a real problem.