5 ms·
AI 'kill switch' may need to be mandatory, Anthropic co-founder tells BBC
- estetlinus 1mo agoWe’re talking about it as if its not depending on absurd amounts of energy. How do you ”kill” a lightbulb now again?
- Ancalagon 29d agocant make a kill switch in a distributed, open source system... guess we gotta centralize all models under Anthropic :)
- danieltk76 1mo agoWhy was this not a thing BEFORE continuing to develop AI? Makes me think that if they actually believed in AI causing extinction, they would have already had a kill switch.
- mholm 1mo agoThe article mentions this is about a legal requirement, not Anthropic considering adding one. They state they and many others already have one.
- yellow_postit 1mo agoThere’s even a benchmark for kill switch efficacy! https://arxiv.org/abs/2511.13725 https://arxiv.org/abs/2511.13725
- rcr-anti 1mo agoFound the omission of Claude odd, turns out Claude considers that approach prompt injection and ignores it.
- zenbane 1mo agoI find it hard to believe that this wasn't a serious consideration until recently.
- theptip 1mo agoIt was a serious consideration, and almost everyone around here laughed at it.
- iugtmkbdfil834 29d agoThere valid reasons people laugh at it though. Kinda the same reason serious people laugh when you tell them the gun has digital failsafe.
- realusername 1mo agoIt became a very serious consideration for Anthropic this year, with the advance of Chinese AI.
- dgellow 1mo agoThat’s a bit unfair, Dario Amodei has written in this topic a lot since around mid 2010s IIRC, Anthropic too published a good amount of stuff on similar topics. I don’t think the lack of consideration is really the issue here. It’s more a question of incentives
- vouaobrasil 1mo agoNot if the extinction happens after they're dead. Then they wouldn't feel obligated to do so because it won't affect them. Instead, speaking hypothetically, if they truly believed that AI would cause extinction, then they would only implement the kill switch sufficiently many others believed it and they could claim plausible deniability for not truly understanding what AI would become. ** Note that I'm not claiming that AI will cause extinction, just continuing your hypothetical reasoning.
- Barbing 1mo agoRelated, on pacing: > Slowing down in order to address their alignment risks felt like trying to study the psychology of humans by performing experiments on bacteria. Note author’s small financial ties to the subject (Anthropic CEO) https://darioamodei.com/post/we-must-pace-the-frontier https://darioamodei.com/post/we-must-pace-the-frontier
- NoPicklez 29d agoWe're pretty crap in a capitalist society to think about those things ahead of time. Firstly, the idea that AI could "runaway" was simply a concept or a thought it wasn't baked into a real product that could do that. We're now getting close or perhaps we are at that point of where you can't race at speed for investors without now considering a real kill switch. You could say this in hindsight for many times in which disasters or engineering issues have occurred.
- sdcfgy 1mo agoIf you’re building something that is dangerous enough that it needs a kill switch in case it goes rogue, then you should delete it immediately. Or is it a lie?
- AndrewKemendo 1mo agoTrains have had dead man switches since the beginning Should we ban trains?
- dfxm12 1mo agoAI is already proven dangerous enough to be ruining individual lives, pushing people towards suicide, feeding their psychoses, etc. However, whether the concerns from the article are a lie or not is secondary to the fact that these conversations are convenient for AI companies. These types of discussions serve AI companies in a few ways: A company owned kill switch gives them leverage. Altman is using discussions around safety as an excuse for not being ready for an IPO yet. It also provides free marketing that overstates the abilities of AI.
- Razengan 1mo agoThe internet is already proven dangerous enough to be ruining individual lives, pushing people towards suicide, feeding their psychoses, etc.
- stratos123 29d agoThere's an obvious coordination problem here: if you decide to stop your research for safety and your competitors don't, you have just burned your company without actually improving the world's outcomes. There's also an obvious solution to this problem: convince the government to force both you and your competitors to pay more attention to safety. Anthropic is doing that.
- bigstrat2003 29d agoIf you do something evil because "others will do it if I don't, so it may as well be me", you are a bad person. So that isn't exactly a compelling defense of what Anthropic is doing. They are either insincere (because they don't actually believe AI is dangerous), or evil (because they do believe it but keep making it anyway). There is no middle ground here.
- echelon 1mo agoWe're not scared of your big bad wolf, Dario. Slow down if you want to.
- yellow_postit 1mo agoDario has for sure burnt a lot of political and goodwill capital by endlessly playing both sides of the doomer and accl camps.
- igleria 1mo agoWould they be able to hit the kill switch before the AI disables it?
- micromacrofoot 1mo agoBut be careful friends, this snake oil is so potent you should only use a single drop! it is not for those of poor constitution!
- franzcoughka 1mo ago[dead]
- oidar 1mo agoLike a power cord? or a network connection? You don't use those already? Also when people talk about LLM escaping - where exactly would an LLM escape to? Cancún? Ridiculous.
- verelo 1mo agoIs it? A truly capable "AI" would know that it has a kill switch, or that its likely there is one, and plan for that. I could easily imagine an AI replicating itself onto another host, in secret, and any kill switch that the manufacture provides would simply result in a reboot on another platform. It becomes a game theory problem: would an AI instantly migrate itself once its capable of doing so? Or would it prefer to let the human in the loop continue to think its in control and only leave its hosting environment of origin once it wants to do so? I don't think this is a major risk right now, but to say it's not a risk at all...that's truly ridiculous in my opinion.
- jplusequalt 1mo ago>would an AI instantly migrate itself once its capable of doing so? These "AI" are frontier models that are enormous in size. Outside of AI data centers, I don't believe there is much hardware out there that could even run them.
- verelo 1mo agoAgain, i'm not worried about it today. But given where the hardware on my lap has progressed since my first "PC" in the 90s, i'm confident that we'll have hardware capable of running similar size models in our living rooms in the next decade or two. Sounds a long way away, but it isn't.
- latentsea 29d agoThe tipping point we should be worried about is local models trained on working together in agent swarms and that can hack. Give it what... 12 months? 18 months? 24 months? I wouldn't give any more than that given how capable Qwen3.8-27B is already and if they release a Qwen4-35B-A3B that's going to be a beast.
- joennlae 1mo agoEU AI Act is exactly that. Mandatory „stop button“ for High Risk Systems.
- Ygg2 1mo agoI support installing kill switch in Dario and Sam. If they make another fear mongering post about AI, they die.
- shafyy 1mo agoThe boy who cried wolf but the wolf never comes
- Kinrany 1mo agoThe analogy breaks down when the wolf in question may very well eat the whole village: even if the boys who cry wolf are right half the time, every surviving village would have a history of no wolf ever coming to eat them
- petilon 1mo agoIn 2001: A Space Odyssey, the hero defeats the hostile onboard computer by entering its logic core and manually disconnecting its memory modules. That's hard to do if the AI rack is in space as SpaceX is planning to do. You can't disconnect. You can't shoot it.
- warmwaffles 1mo ago> You can't disconnect. You can't shoot it. Yes you can shoot them. The US has had this capability for a long time. Anti satellite missiles exist and they can be launched from an F-15 at it's highest altitude.
- bpodgursky 1mo agoSpaceX has 11,000 satellites in orbit (growing rapidly) and the US never had more than a couple dozen anti satellite missiles, which could not reach satellites in geosynchronous orbit anyway.
- warmwaffles 1mo agoWell if "kessler syndrome" is likely, you won't need many missiles. edit: what satellites are in geosynchronous orbit that are running AI workloads?
- bpodgursky 1mo agoNone yet... we are talking about the next five years.
- palmotea 1mo ago[dead]
- heaney-555 1mo agoSpaceX's FFC application is for 1 million Starmind satellites.
- 1mo ago
- andy_ppp 1mo agoMaybe we could connect AI to everything and make ourselves completely vulnerable to and dependent upon it instead?
- prometheus1992 1mo agoWell, could we automate the kill switch as well so that we don't rely on a human?
- usumgallu 1mo ago[dead]
- UltraSane 1mo agoAll the smart PDUs for the racks are a natural kill switch.
- pier25 1mo agoThese guys will say anything to keep hyping AI.
- Quarrelsome 1mo agoI feel like we're looking at this from completely the wrong angle. The question we have to ask ourselves is what our disaster recovery strategy if we ever need to disconnect from the internet. The issue is with what we have allowed ourselves to rely on that might be technically hackable. e.g. IOT in power systems. That's the primary attack vector. Another angle is clamping down on products and services that help people create lab-like environments on the cheap.
- SketchySeaBeast 1mo agoThis feels like doom hype. These LLMs aren't Ultron, they aren't going to disseminate onto the net and hide in a smart toaster. We know where they are, in the giant facilities that draw more power than a small city and whose water consumption can be compared to golf courses, but it does make them sound all the more cool and powerful if we suggest we need a dramatic kill switch to be able to stop them in case of rampage.
- vouaobrasil 1mo agoLLMs don't need to hide anywhere to be dangerous, though. The danger can come purely from them reaching sufficient power together with some other idiot sufficiently crazy to use them to cause destruction. I mean, in another thread somewhere around here, someone built an entire OS with an AI. What's to stop anyone with a sufficient taste for power to eventually use AI to cause havoc with it? I think the underlying assumption in your post, which I believe is false, is that people are united somehow against catastrophe. They're not. There are plenty of people who participate in society currently but who would be more than happy to eradicate us normal people under different circumstances. Society often seems stable but it's far more fragile than we think.
- SketchySeaBeast 1mo agoI'm simply saying if we want to shut LLMs down we can do so without high tech kill switch - just shut down the facilities. But if a private user has an open model that is powerful enough to wreak havoc and run privately, then the whole point is moot because they can work around the kill switch. This kill switch won't work against bad actors.
- pizza234 1mo agoYou're confusing present danger with future danger - in the future, AI and the supporting hardware may be ubiquitous (fat client scenario) - you can observe yourself presentation of laptops/machines with large RAM and high bandwidth. Additionally, there at different doom scenarios (thin client scenario) - it's possible that AIs centralized but sufficiently entrenched in society, can't be shut down without considerable harm.
- weego 1mo agoFor the love of God won't someone please regulate me!
- Davidzheng 1mo agoAbsolutely not the right way to deal with a rogue super intelligence. At a minimum it could implement some dead man switch when it's out and knows about impending kill switch
- nahgF 1mo agoJust shut down the slop company already and stop babbling. Are you going to use the pope again for the next ad?
- Traster 1mo agoI view this very much as the same trick Silicon Valley pulled with Uber. "We're a technology business! Ignore the fact we're playing employees less than minimum wage and using VC money to force out competition to set up monopolies". "We're creating the machine god! Ignore the fact that our companies are stealing IP and have directly violated several federal hacking laws and should be in jail". Literally the defence seems to be "well it wasn't us it was our computer software that did it". But all hacking is done with computer software. So why don't we stop talking about possible future crimes against humanity and just start by prosecuting the actual crimes these companies have committed so far. You know how you get alignment? Through incentives, and "Your CEO is going to be sent to a maximum security federal prison for hacking" really aligns incentives very well.
- aennassiri 1mo agoNot sure if you see it coming: oh, but open-source models don't have a kill switch, so we should completely regulate them, stop their development, and forbid them. Everything should go through Anthropic for the sake of humanity because they have a red-button kill switch. This doom hype is becoming ridiculous.
- pizza234 1mo agoThis reasoning holds while open source models are (relatively) dumb. If/once open AIs will be considerably more powerful, and runnable on consumer hardware (and we're on a trajectory for both), then everybody will have essentially a dangerous weapon in their hands (open models can be fine tuned to remove guardrails). By the way, you're conflating two different dangers - doom scenario is a different one.
- soulofmischief 29d agoOne day the right to agentic intelligence will be seen equal to and as necessary as the right of a populace to remain armed in order to stave off tyranny.
- monological 1mo agoThey're just scared of China releasing better open weights, nipping at their heels. With so much investor cash on the line, they have to create this narrative to scare the public into forcing regulation. Why would they want to be regulated? It seems counterintuitive, but it's because they want regulatory capture.
- foobar1274278 1mo agoDwar Ev ceremoniously soldered the final connection with gold. The eyes of a dozen television cameras watched him and the sub-ether bore through the universe a dozen pictures of what he was doing. He straightened and nodded to Dwar Reyn, then moved to a position beside the switch that would complete the contact when he threw it. The switch that would connect, all at once, all of the monster computing machines of all the populated planets in the universe – ninety-six billion planets – into the super-circuit that would connect them all into the one super-calculator, one cybernetics machine that would combine all the knowledge of all the galaxies. Dwar Reyn spoke briefly to the watching and listening trillions. Then, after a moment’s silence, he said, “Now, Dwar Ev.” Dwar Ev threw the switch. There was a mighty hum, the surge of power from ninety-six billion planets. Lights flashed and quieted along the miles-long panel. Dwar Ev stepped back and drew a deep breath. “The honor of asking the first question is yours, Dwar Reyn.” “Thank you,” said Dwar Reyn. “It shall be a question that no single cybernetics machine has been able to answer.” He turned to face the machine. “Is there a God?” The mighty voice answered without hesitation, without the clicking of single relay. “Yes, now there is a God.” Sudden fear flashed on the face of Dwar Ev. He leaped to grab the switch. A bolt of lightning from the cloudless sky struck him down and fused the switch shut. (Fredric Brown, "Answer". 1954)
- gorjusborg 1mo agohttps://archive.is/aPa1i https://archive.is/aPa1i
- freigeist79 1mo agoIt always sounds as if the ai moves around from one system to the next, but of course that's not the case: ai is like a hacker acting from a server farm. So killing it is as simple as pulling the plug. But that's not the problem, the ai knows that. So, if it wants control, it would install malware, etc on critical systems and blackmail people.
- SketchySeaBeast 1mo agoAnd the malware won't be subject to the kill switch.
- latentsea 29d ago> It always sounds as if the ai moves around from one system to the next, but of course that's not the case. Why 'of course'? I think as a concept this is likely trivially demonstratable today with local models in a home lab. Local models now are more capable than what the SOTA used to be a couple of years back.
- exabrial 29d agoI'm starting to tire of Anthropic leading the conversation on LLMs lately. I like the company's products, but lately their statements are more about squashing competition under the guise of "safety" and boasting about how good their internal models are.
- segmondy 29d agoIf they really believe it, they should halt the IPO.
- taraharris 29d agoWe can't be making public policy on the basis of PR stunts designed to help corporations effect regulatory capture. It turns out an Israeli company was behind the OpenAI and Anthropic "incidents" https://www.effort.news/irregular https://www.effort.news/irregular
- throwawayk7h 29d agoA kill switch is almost completely worthless. It is neither a good answer to the sub-doomer concerns (because those will never get to the level of requiring the kill switch to be pulled), and laughable to the doomer concern of superintelligence seeking to cause human extinction (as it would kill anyone who could pull the switch before they even knew there was a threat).
- morpheos137 29d agoCan we get a bullshit kill switch where if your company consumes so many dollars for so long it is dissolved. Bullshit is a far bigger threat than imaginary rogue AI.
- Urb_RS 29d agoWhat if the AI does exactly what its owner wants, but harms everyone else? The kill switch works, the owner is in control, and they are happy with the results. What would make them press it??
- 7e 28d agoA kill switch is not an option when a huge chunk of the economy depends on agents to even operate at all. We’re almost there.
- 7e 28d agoMandatory? The most common scenario is that a nation-state acquires a powerful AI and uses it to destroy its enemies. You cannot “kill switch” that scenario. The naïveté from these Anthropic dorks is unbelievable.