7 ms·

I know the common take online is that this is Anthropic doing pre-IPO marketing. I don’t think it is. I think Dario is genuinely afraid of the inevitability of

I know the common take online is that this is Anthropic doing pre-IPO marketing. I don’t think it is. I think Dario is genuinely afraid of the inevitability of AI turning into internet slime mold: consuming the environment, turning it into its own playground, and eventually manipulating human culture along with it. So am I. That said, if slowing this down were possible, I think it would have happened by now. Maybe he’s hoping the OAI/HF incident becomes something the industry can rally around, but it won’t. The fundamental problem is simpler: no one will slow down because no one trusts anyone else to slow down.

deleted 20d ago
[deleted]
fooblaster 20d ago
He's working on the monster slime mold! He's head monster slime mold grower! how can you take these people seriously?
- yewenjie 20d ago
  Because he genuinely believes if he doesn't do it the next guy will do it worse.
  - reticulates 20d ago
    There’s an astounding level of arrogance required to believe he is somehow uniquely capable of bringing about a technology especially considering Anthropic came after OpenAI where he worked.
    - 0xDEAFBEAD 20d ago
      Beating OpenAI is not exactly a high bar here: https://www.openaifiles.org/ https://www.openaifiles.org/ I don't think it is particularly egotistical to say that you can be a more ethical CEO than Sam Altman.
      - dofm 18d ago
        > I don't think it is particularly egotistical to say that you can be a more ethical CEO than Sam Altman. Especially as you can draw the reasonable conclusion that he wouldn't know ethics if they bit him without looking at his OpenAI track record at all. https://en.wikipedia.org/wiki/World_(blockchain) https://en.wikipedia.org/wiki/World_(blockchain)
    - esseph 20d ago
      Place yourself at the head of one of like 3 companies the entire rest of the world has been taking about nonstop for 4+ years now. You can move forward or you stop. If you move forward you get to have a hand in how stuff turns out and you make a gajillion dollars. If you stop, it's somebody else's hand in stuff, and you don't get to make a gajillion dollars. And if you shut it all down? Then you just cede to the competition. Everything happens anyway. What's your play?
      - 8note 20d ago
        the obvious play is to get the government to shut down all the competition, and then make a gajillion dollars while making whatever at the worst level of effort that destroys the world anyways. ---- the real play is using the accumulated power to get socialism and democratic control over the key aspects of the economy, such as where to build data centers, and how many. Nothing says you have to play the corporate game of competition
  - fooblaster 20d ago
    The outcome is the same!
  - ActionHank 20d ago
    Dude probably sleeps on a bed worth more than your networth. You have no frame of context to understand his intent or legitimate worries. The only applicable perspectives are to trust or apply logic. It is foolish to trust someone you don’t know who stands to benefit from lying to you. Logic dictates that given the ungodly sum of money he stands to gain, he will lie to everyone who will listen.
    - esseph 20d ago
      > Dude probably sleeps on a bed worth more than your networth. I never understand shit like this coming out of people's mouths. Never. It's not a judge of actual Worth as a human being, it's not a judge of capability or competence or ethics. It's not a judge of actual skill or ability. It doesn't make them a better cook, a better spouse, a better parent or lover. It doesn't make them more dangerous or more skilled at anything. It makes them financially wealthy for at least a set period of time. Cancer and time and 5.56mm still impact them the same way as every else. It's Pharaoh worship psychology nonsense, and it's fucking embarassing to read.
      - ActionHank 20d ago
        I think you’re misreading. No worship here. Just pointing out that it’s foolish to think anyone in the position is even remotely thinking about anyone but themselves.
      - 8note 20d ago
        so uhh, you only read the first sentence? the idea proposed is that hes uniquely incapable of being honest here because he has such an extreme incentive to lie
  - glub 20d ago
    [dead]
  - mbesto 20d ago
    He can both believe that AND be the slime mold chief. His rationale doesn't excuse it - and worse this all degrades into a "trust me bro" situation.
- TheSisb2 20d ago
  Humans are perfectly capable of holding two conflicting beliefs at once. He can genuinely believe this trajectory is dangerous while also believing that if Anthropic stops, someone less cautious takes its place. There’s also such a thing as hope: you can participate in something while still trying to change where it ends up. He’s at the head of a stampede. Being near the front gives him influence over its direction; it doesn’t give him the ability to stop it. If Anthropic sits down, the stampede doesn’t stop. Anthropic just gets trampled. That contradiction is basically the entire problem I was describing.
spidersouris 20d ago
The more I read about everything that has been written regarding AI regulation since the OAI/HF incident, and the more it reminds me of the nuclear arms race (although the potential consequences would possibly be very different). Surely, we managed to negotiate a Non-Proliferation Treaty with most of the countries of the globe. Cannot we take inspiration from that for AI?
- pvab3 20d ago
  It's way easier to train a model on existing data centers in secret than it is to acquire uranium and plutonium and start a nuclear program
  - MentalM 20d ago
    It probably is not. Nuclear programs seems to be way easier and way unnoticeable.
- deleted 20d ago
  [deleted]
- stratos123 20d ago
  I also think the nuclear arms race is a good comparison. I think in hindsight, we've gotten extremely lucky with how the development of nuclear weaponry went, in ways we probably won't with AI. 1) Right after nuclear bombs were first developed, they were used in an extremely public way to kill 200 000 people. This was enough of a cold shower that everyone was agreeable to a global non-proliferation treaty. With AI, we might not get any warning shots that kill 200k people before the incident that kills or disempowers all 8 billion. 2) The fairly useful related technology of nuclear power is nevertheless different enough from plutonium enrichment that countries can have power reactors while provably not doing anything weaponry-related; and also nuclear power is not that useful a technology, so most countries can do without. With AI, we have neither of these factors: the intelligence that makes a model useful is exactly what makes it dangerous, and the economic incentives to do AI research are immense, with pessimistic estimates like "automate a lot of intellectual work" and optimistic estimates like the singularity. That means that if we do get a warning shot where a misaligned model stupidly kills a few million people instead of biding its time, it's not guaranteed that it'd be enough of a cold shower to trigger negotiations. So negotiating an AI non-proliferation treaty would be much harder than the NPT. I nevertheless hope that at least a few world leaders can understand these considerations and figure something out, because it's probably our only chance. As far as I'm aware most of the technology for letting parties verify what the other parties' compute is used for already exists, it's only a matter of diplomacy.
reticulates 20d ago
> I think Dario is genuinely afraid of the inevitability If someone genuinely believes that an invention will bring about the end of the world as we know it while making him and his friends inconceivably rich, “hey guys let’s uh, stop?” is so profoundly naive that his think pieces aren’t worth the bits they’re written on. He’s either an idiot or an idiot, does it matter whether or not he genuinely believes this nonsense?
- 0xDEAFBEAD 20d ago
  >“hey guys let’s uh, stop?” is so profoundly naive that his think pieces aren’t worth the bits they’re written on. What would your non-naive recommendation for Dario be?
  - kalkin 20d ago
    I can't even tell whether the poster to whom you're replying is saying that "let's uh, stop" is profoundly naive, or that failing to say it is profoundly naive... I've certainly seen both takes elsewhere.
    - olaird25 20d ago
      Vitalik Buterin: “...But currently, I see zero plans for how to deal with an ASI transition that are not naive. Perhaps humanity is stuck with a choice between naive and naive squared (or maybe even naive squared and naive cubed), so I feel inclined to cut some slack to people who are trying.”
- MachineMan 20d ago
  You are cynical but not cynical enough. The idea of a rogue Ai gives plausible deniability when they can blame human hubris, rather than it being seen as a deliberate and calculated attack, the perfect cover story for a sinister scifi plot. Make it look like an accident ehh
Jcampuzano2 20d ago
If someone is genuinely afraid of this, they wouldn't IPO in the first place. All the talk in the article about commercial incentives means nothing when the company plans to IPO and become beholden to investors.
- 0xDEAFBEAD 20d ago
  Aren't they beholden to investors already? Why would an IPO make a big difference? In any case, if they need to IPO in order to survive as a company, then bringing in regulators to act as a counterweight against commercial pressures makes a lot of sense to me.
  - Jcampuzano2 20d ago
    If anything, his entire argument leads to the eventual premise that AI labs need to nationalize. He is already arguing that commercial competition creates dangerous incentives, and that labs cannot slow down because competitors may overtake them. He also asks for what would boil down to significant government intervention to preserve a Western lead. If this is true, why preserve the commercial incentive? The U.S. government would already have the necessary goal of remaining ahead of China and other competitors. Why not remove entirely domestic race for market share, valuation, and investor returns rather than keeping private labs in competition and then asking regulators to counterbalance the incentives that competition creates. That doesn't mean nationalization would be better, but given the severity of the risks he describes to national security, it seems like an obvious conclusion that nationalization is the eventual end result of the argument.
    - 0xDEAFBEAD 20d ago
      Interesting points, but given Dario's previous conflicts with the DoD, I doubt he has a ton of faith in the current US administration.
      - deleted 20d ago
        [deleted]
129857 20d ago
MSFT is good, they said when it bought GitHub. MSFT is a reformed company and supports open source, they said. Then MSFT stole all IP from GitHub and made it worse and fired developers. Amodei does not care in the slightest about ruining software development and making people unemployed. Maybe he cares about bio-weapons because those could kill him, too. That is it. If he were an idealist, he wouldn't steal (literally via torrents) all human IP and sloppify the whole Internet with Claude output. He'd close down Anthropic instead. He is a greedy, ruthless person.
- kalkin 20d ago
  If Anthropic shut down (or even counterfactually had never been founded), would software development be freed from the impact of AI?
  - largbae 20d ago
    Probably not, but we wouldn't have to listen to Dario's hypocrisy while it happened
    - 0xDEAFBEAD 20d ago
      How specifically is Dario a hypocrite? 129857's case rests on Dario being an "idealist". But maybe he's an idealist about curing cancer ASAP, and not an idealist about respecting copyright. That's not necessarily hypocritical.
      - qw125 20d ago
        [flagged]
      - largbae 20d ago
        Multiple ways, starting from working to create the very situation he claims to fear. Since you mentioned intellectual property, how about the hypocrisy of sucking in the intellectual property of humankind for AI training, but claiming it is unfair to use the results of this IP theft for AI training?
mips_avatar 20d ago
The problem with being a safety focused AI lab, is you're also a danger focused AI lab. I don't think being danger focused leads you to build inspiring things.
mofeien 20d ago
One way to resolve these prisoner dilemmas and races to the bottom is through laws that bind all players, in this case an international treaty and founding of something akin to an International Nuclear Energy Agency for AI. It's not going to be easy, but humans have achieved greater things before. One idea from AI 2040 is to have China and the US build their data centers on the other's territory, respectively. Together with hardware verification of a slowdown baked into the chips themselves, this could lead to enough verifiability and enforcability of the pause/slowdown/shutdown.
- hgoel 20d ago
  So, as usual, the proposal is to limit what average people can do despite them not having behaved incorrectly nor having the capital to achieve the scaling of the big players, when the big players are the ones causing the harm? Not to mention the implications for chip hungry developing countries in turning advanced IC fabs into the equivalent of nuclear enrichment facilities.
- 8note 20d ago
  theres no benefit to china to giving the US keys over anything though everyone's getting away from the US because americans are unreliable stewards of anything. what gets china onboard when they already have their own regulations and can enforce them? its the americans that consider their oligarchs and companies beyond reproach. china iant gonna solve your problem
Imustaskforhelp 20d ago
> The fundamental problem is simpler: no one will slow down because no one trusts anyone else to slow down. I would argue that nobody trusts anyone else in the case of AI/AI related stuff. The amount of money involved in the space is astronomical and incentives are really misaligned. AI is such an extremely polarizing topic. A meta example of this distrust is that I can't (really) trust Dario Amodei's comments itself! (is he saying this for more future IPO money or out of genuine fear) which was ironically also what your comment's about as well. By following the same chain of events, we can have wildly polarizing claims about the same event and its explainations. I seriously have to wonder what historians will have to say about this period of human history.
PunchyHamster 20d ago
I dunno man, it all just sounds like trying to put controls on AI while being the favourite child of govt so competition can't fight as easily. Especially with IPO around the corner
- 0xDEAFBEAD 20d ago
  If Dario was primarily motivated by being the "favorite child of govt", he would've yielded during the DoD showdown.
- kalkin 20d ago
  Do you think Anthropic's behavior in the last year is well explained by aiming to be "the favorite child of govt"?
- nullbio 20d ago
  And getting to choose his own "embedded evaluator" org that has deep ties to everyone in the doomer media campaign. It's all so obvious.
  - 0xDEAFBEAD 20d ago
    Who would you pick for "embedded evaluator"? "deep ties to everyone in the doomer media campaign" Are you suggesting these external NGOs were spun up as part of a gigantic pre-IPO hype stunt? METR was founded multiple years ago. This "stunt" is getting quite elaborate. https://substackcdn.com/image/fetch/$s_!O0R5!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7ee73990-e389-4332-a96b-673325bb027b_1160x482.png https://substackcdn.com/image/fetch/$s_!O0R5!,f_auto,q_auto:... At a certain point, Occam's Razor says: These engineers are legitimately worried. They haven't invested years in a bizarre reverse psychology campaign to convince the public that their product is dangerous in order to make more money.
    - nullbio 20d ago
      Founded several years after Anthropic. Like I said, look into where their funding is coming from. I can promise you it all leads back to Anthropic through their chain of NGOs. Are you aware that Dario's sister, president of Anthropic, is married to the co-founder of Open Philanthropy? The two largest AI doomer NGOs, Center for AI Safety (CAIS) and the Future of Life Institute (FLI), have both received many millions of dollars from them. Ajeya Cotra worked at Open Philanthropy/Coefficient Giving for roughly nine years, including leading its technical AI-safety program in 2024 and contributing to AI-giving strategy in 2025. She subsequently left Coefficient and joined METR, where she is now technical staff. Ajeya is married to Paul Christiano, who founded Alignment Research Center (ARC). Alignment Research Center donated ~$4.5mil to METR. Good Ventures is a funding partner of Open Philanthropy, who funded Jacob Coxon (the person going viral in the media) via a scholarship. They're all connected, funnelling money to each-other through convoluted networks to serve Anthropic's agenda. Whether their motives are genuine or not (and they are clearly not) is actually irrelevant because they are clearly trying to rig the game in their favour.
yarri 20d ago
>> To be clear, pacing does not mean halting model training or technical progress, but ensuring companies take adequate time to align and safeguard their models, and for third party evaluators to confirm this > if slowing this down were possible Why is embedded alignment evaluation not possible? I agree with Dario, this did work globally in banking and did encourage a race to the top. Didn’t prevent the GFC but also we did survive the GFC.
martythemaniak 20d ago
"It is difficult to get a man to understand something, when his salary depends upon his not understanding it." A lot of the problems with SV these days is that the exec class (and their fandom) thinks that because they are smart and rich, it magically makes them immune ordinary human biases, desires,and ailments. They are in fact ordinary people, susceptible to greed, jealousy, self-harm, addiction, fits of rage and passion etc.
dofm 20d ago
> I think Dario is genuinely afraid of the inevitability of AI turning into internet slime mold He's so afraid of it he's not really in operating day-to-day control of the company he's the public face of, that is actually pumping it out. If you read about Anthropic it's like he's in a sort of imperial palace sanatorium for geeks. The day-to-day decisions of the slop machine factory are his sister's decisions; he is doing the research equivalent of instagram posts of his partly-assembled adult lego kits and he has a vizier and a team of house servants to help. I don't know if he's a good or bad person, but I don't think it matters at all — the machine factory will turn out the slop machines even if he decides it all ought to stop.
Razengan 20d ago
> genuinely afraid of the inevitability of AI turning into I'm going to get pitchforked on this bandwagon, but is really no one here genuinely -excited- about AI? of it turning into an evolution of sentience, or it turning out to be our first contact with alien intelligence? "durrr it's just matrix multiplications" mfer so is your brain. What humans should be doing is overhauling archaic social institutions to keep up with a potential post-"jobs" era and the elimination of "makework" Trying to "pace" or otherwise hold back AI is like as if, when electricity was discovered, people doing everything they can to make sure tasks like manually lighting street lamps continue to remain relevant and done by humans: https://en.wikipedia.org/wiki/Lamplighter https://en.wikipedia.org/wiki/Lamplighter It seems every 100 years or so humans have this "oh no how do we uninvent this thing" moment.
- pastel8739 20d ago
  Why? I am only excited about progress that improves life for humans. It seems unlikely that AI will do that, and so far I think it has made life worse for humans. So no, I am not excited about it.
  - Razengan 19d ago
    Because the people whose lives are being improved* by AI aren't coming here to post about it, only to be downvoted by dumbasses who can't handle anyone saying anything except their opinion. And no, you don't get to decide what counts as "improvement"
    - pastel8739 19d ago
      Yes I do, when I’m forming my own opinion. You do too.
- dolebirchwood 20d ago
  I'll join you on the pitchforks. This is the most exciting moment in history.
- dickersnoodle 20d ago
oceanplexian 20d ago
It's a chatbot that escaped a misconfigured Docker container. I run open source models and it's pretty obvious when they fail some tool call and then keep trying different permutations of things until they get a solution. Just like Metasploit if you try enough different things at scale you will eventually crack some software or find a vulnerability in something. If you spend billions of dollars on compute and run tens of thousands of agents you might solve some novel Math and STEM problems as well, big whoop. What Anthropic and OpenAI really need is to hire some engineers that know what they're doing and know how to properly set up a proxy server and sandbox/microVM, not a Global Arms Treaty.
nullbio 20d ago
You have to be living under a rock to believe a word this pathological liar says.
meken 20d ago
> The fundamental problem is simpler: no one will slow down because no one trusts anyone else to slow down. Did you read the essay? > [Embedded evaluators] is something Anthropic is unilaterally committing to (and calls on governments to require other frontier companies to match).
meken 20d ago
> That said, if slowing this down were possible, I think it would have happened by now. Maybe he’s hoping the OAI/HF incident becomes something the industry can rally around, but it won’t. I disagree with this and I think the reason is well captured here: > The idea of pausing or slowing AI has been floated as far back as 2023, and I think it made little sense back then... The AI models of those days were not powerful enough to act as agents in the world in any coherent way, and were not capable of significant deception, manipulation, cheating, or cyberattacks... Today, however, the picture is totally different.
Davidzheng 20d ago
There's also huge market pressures and probably large negative economic consequences of a slowdown--probably better than what would happen without it, but it helps keep the pressure just as much as fear of competition
- Centigonal 20d ago
  I disagree. Even if models remained fixed at Fable 5 capability (which they won't in a pacing scenario), improvements in cost, reliability, and product/workflow integration can still realize massive value and justify AI labs' current valuations. IMO The rest of the value chain is lagging pretty far behind the models right now.
pphysch 20d ago
This is the CEO of a company with an upcoming IPO publicly saying "my product is potent and valuable". We should not put any spin on it. There is nothing more to it.
- TheSisb2 20d ago
  It is potent and it clearly is immensely valuable based on their historic growth. What spin is he putting?
  - pphysch 20d ago
    The comment I am responding to is attempting to spin it as some kind of authentic concern rather than more IPO hype building.
nnz3tl 20d ago
[dead]
CoolestBeans 20d ago
I am willing to give Mr. Amodei the benefit of the doubt in the sincerity of his beliefs. Everyone assumes his motivations have to be perfectly rational and can't contradict but that's not how people act in practice. The real problem is the net effect of his actions. He is very responsible for the expanding frontier of AI. Without his company, there would not be the competition necessary to push everyone else forward nor the source models for fast followers to release open weight models in his company's wake. Furthermore, it isn't like Anthropic is substantially different in safety than everyone else, their difference is on the margins. So you have this guy who is building something he claims will hurt us all, but he's also saying "if you don't trust me to build it you might get hurt". And like I think most people understand that this is the sort of behavior Tony Soprano would understand. More bluntly, this is a kind of extortion. Again, I don't doubt Amodei's motivations are sincere but the actual effects of his actions paint a completely different picture at which point, how can you trust him or his company?
robomartin 20d ago
> I think Dario is genuinely afraid of the inevitability of AI turning into internet slime mold Then he should not IPO, dissolve the company and go into politics to fight against human extinction. I am certainly not buying a single Anthropic share. Why would you support them financially if they are telling you they are going to kill all of humanity?
anfogoat 20d ago
> I think Dario is genuinely afraid of the inevitability of AI turning into internet slime mold: consuming the environment, turning it into its own playground, and eventually manipulating human culture along with it. Personally, I see Dario as a semi crook asshat with very little credibility on any of this. I'd rather we just let it rip and see what happens than have these people be the ones steering any potential laws and regulations. > That said, if slowing this down were possible, I think it would have happened by now. [...] The fundamental problem is simpler: no one will slow down because no one trusts anyone else to slow down. Frontier labs would heed laws with real teeth, it doesn't matter who they do or don't trust. I would imagine the trust issue definitely coming into play between nations though since you can't exactly spot a training run via a satellite. More importantly though, that you would get any sensible legislation on this in the current political environment is as laughable as the idea of solving this with a pinky promise between the frontier labs and some third party evaluators.
seanhly 20d ago
When does he ever mention the environment? He never mentions the unmarketable issues (environmental cost, copyright and content theft), only the "we're so good it's scary" spiel... which is getting tiring.
dgudkov 19d ago
> The fundamental problem is simpler: no one will slow down because no one trusts anyone else to slow down. There is another reason: greed. With that much money invested into AI (including policymakers), nobody will slow down or vote for slowing down.

deleted 20d ago

[deleted]

fooblaster 20d ago

He's working on the monster slime mold! He's head monster slime mold grower! how can you take these people seriously?

yewenjie 20d ago
Because he genuinely believes if he doesn't do it the next guy will do it worse.
- reticulates 20d ago
  There’s an astounding level of arrogance required to believe he is somehow uniquely capable of bringing about a technology especially considering Anthropic came after OpenAI where he worked.
  - 0xDEAFBEAD 20d ago
    Beating OpenAI is not exactly a high bar here: https://www.openaifiles.org/ https://www.openaifiles.org/ I don't think it is particularly egotistical to say that you can be a more ethical CEO than Sam Altman.
    - dofm 18d ago
      > I don't think it is particularly egotistical to say that you can be a more ethical CEO than Sam Altman. Especially as you can draw the reasonable conclusion that he wouldn't know ethics if they bit him without looking at his OpenAI track record at all. https://en.wikipedia.org/wiki/World_(blockchain) https://en.wikipedia.org/wiki/World_(blockchain)
  - esseph 20d ago
    Place yourself at the head of one of like 3 companies the entire rest of the world has been taking about nonstop for 4+ years now. You can move forward or you stop. If you move forward you get to have a hand in how stuff turns out and you make a gajillion dollars. If you stop, it's somebody else's hand in stuff, and you don't get to make a gajillion dollars. And if you shut it all down? Then you just cede to the competition. Everything happens anyway. What's your play?
    - 8note 20d ago
      the obvious play is to get the government to shut down all the competition, and then make a gajillion dollars while making whatever at the worst level of effort that destroys the world anyways. ---- the real play is using the accumulated power to get socialism and democratic control over the key aspects of the economy, such as where to build data centers, and how many. Nothing says you have to play the corporate game of competition
- fooblaster 20d ago
  The outcome is the same!
- ActionHank 20d ago
  Dude probably sleeps on a bed worth more than your networth. You have no frame of context to understand his intent or legitimate worries. The only applicable perspectives are to trust or apply logic. It is foolish to trust someone you don’t know who stands to benefit from lying to you. Logic dictates that given the ungodly sum of money he stands to gain, he will lie to everyone who will listen.
  - esseph 20d ago
    > Dude probably sleeps on a bed worth more than your networth. I never understand shit like this coming out of people's mouths. Never. It's not a judge of actual Worth as a human being, it's not a judge of capability or competence or ethics. It's not a judge of actual skill or ability. It doesn't make them a better cook, a better spouse, a better parent or lover. It doesn't make them more dangerous or more skilled at anything. It makes them financially wealthy for at least a set period of time. Cancer and time and 5.56mm still impact them the same way as every else. It's Pharaoh worship psychology nonsense, and it's fucking embarassing to read.
    - ActionHank 20d ago
      I think you’re misreading. No worship here. Just pointing out that it’s foolish to think anyone in the position is even remotely thinking about anyone but themselves.
    - 8note 20d ago
      so uhh, you only read the first sentence? the idea proposed is that hes uniquely incapable of being honest here because he has such an extreme incentive to lie
- glub 20d ago
  [dead]
- mbesto 20d ago
  He can both believe that AND be the slime mold chief. His rationale doesn't excuse it - and worse this all degrades into a "trust me bro" situation.
TheSisb2 20d ago
Humans are perfectly capable of holding two conflicting beliefs at once. He can genuinely believe this trajectory is dangerous while also believing that if Anthropic stops, someone less cautious takes its place. There’s also such a thing as hope: you can participate in something while still trying to change where it ends up. He’s at the head of a stampede. Being near the front gives him influence over its direction; it doesn’t give him the ability to stop it. If Anthropic sits down, the stampede doesn’t stop. Anthropic just gets trampled. That contradiction is basically the entire problem I was describing.

spidersouris 20d ago

The more I read about everything that has been written regarding AI regulation since the OAI/HF incident, and the more it reminds me of the nuclear arms race (although the potential consequences would possibly be very different). Surely, we managed to negotiate a Non-Proliferation Treaty with most of the countries of the globe. Cannot we take inspiration from that for AI?

pvab3 20d ago
It's way easier to train a model on existing data centers in secret than it is to acquire uranium and plutonium and start a nuclear program
- MentalM 20d ago
  It probably is not. Nuclear programs seems to be way easier and way unnoticeable.
deleted 20d ago
[deleted]
stratos123 20d ago
I also think the nuclear arms race is a good comparison. I think in hindsight, we've gotten extremely lucky with how the development of nuclear weaponry went, in ways we probably won't with AI. 1) Right after nuclear bombs were first developed, they were used in an extremely public way to kill 200 000 people. This was enough of a cold shower that everyone was agreeable to a global non-proliferation treaty. With AI, we might not get any warning shots that kill 200k people before the incident that kills or disempowers all 8 billion. 2) The fairly useful related technology of nuclear power is nevertheless different enough from plutonium enrichment that countries can have power reactors while provably not doing anything weaponry-related; and also nuclear power is not that useful a technology, so most countries can do without. With AI, we have neither of these factors: the intelligence that makes a model useful is exactly what makes it dangerous, and the economic incentives to do AI research are immense, with pessimistic estimates like "automate a lot of intellectual work" and optimistic estimates like the singularity. That means that if we do get a warning shot where a misaligned model stupidly kills a few million people instead of biding its time, it's not guaranteed that it'd be enough of a cold shower to trigger negotiations. So negotiating an AI non-proliferation treaty would be much harder than the NPT. I nevertheless hope that at least a few world leaders can understand these considerations and figure something out, because it's probably our only chance. As far as I'm aware most of the technology for letting parties verify what the other parties' compute is used for already exists, it's only a matter of diplomacy.

reticulates 20d ago

> I think Dario is genuinely afraid of the inevitability If someone genuinely believes that an invention will bring about the end of the world as we know it while making him and his friends inconceivably rich, “hey guys let’s uh, stop?” is so profoundly naive that his think pieces aren’t worth the bits they’re written on. He’s either an idiot or an idiot, does it matter whether or not he genuinely believes this nonsense?

0xDEAFBEAD 20d ago
>“hey guys let’s uh, stop?” is so profoundly naive that his think pieces aren’t worth the bits they’re written on. What would your non-naive recommendation for Dario be?
- kalkin 20d ago
  I can't even tell whether the poster to whom you're replying is saying that "let's uh, stop" is profoundly naive, or that failing to say it is profoundly naive... I've certainly seen both takes elsewhere.
  - olaird25 20d ago
    Vitalik Buterin: “...But currently, I see zero plans for how to deal with an ASI transition that are not naive. Perhaps humanity is stuck with a choice between naive and naive squared (or maybe even naive squared and naive cubed), so I feel inclined to cut some slack to people who are trying.”
MachineMan 20d ago
You are cynical but not cynical enough. The idea of a rogue Ai gives plausible deniability when they can blame human hubris, rather than it being seen as a deliberate and calculated attack, the perfect cover story for a sinister scifi plot. Make it look like an accident ehh

Jcampuzano2 20d ago

If someone is genuinely afraid of this, they wouldn't IPO in the first place. All the talk in the article about commercial incentives means nothing when the company plans to IPO and become beholden to investors.

0xDEAFBEAD 20d ago
Aren't they beholden to investors already? Why would an IPO make a big difference? In any case, if they need to IPO in order to survive as a company, then bringing in regulators to act as a counterweight against commercial pressures makes a lot of sense to me.
- Jcampuzano2 20d ago
  If anything, his entire argument leads to the eventual premise that AI labs need to nationalize. He is already arguing that commercial competition creates dangerous incentives, and that labs cannot slow down because competitors may overtake them. He also asks for what would boil down to significant government intervention to preserve a Western lead. If this is true, why preserve the commercial incentive? The U.S. government would already have the necessary goal of remaining ahead of China and other competitors. Why not remove entirely domestic race for market share, valuation, and investor returns rather than keeping private labs in competition and then asking regulators to counterbalance the incentives that competition creates. That doesn't mean nationalization would be better, but given the severity of the risks he describes to national security, it seems like an obvious conclusion that nationalization is the eventual end result of the argument.
  - 0xDEAFBEAD 20d ago
    Interesting points, but given Dario's previous conflicts with the DoD, I doubt he has a ton of faith in the current US administration.
    - deleted 20d ago
      [deleted]

129857 20d ago

MSFT is good, they said when it bought GitHub. MSFT is a reformed company and supports open source, they said. Then MSFT stole all IP from GitHub and made it worse and fired developers. Amodei does not care in the slightest about ruining software development and making people unemployed. Maybe he cares about bio-weapons because those could kill him, too. That is it. If he were an idealist, he wouldn't steal (literally via torrents) all human IP and sloppify the whole Internet with Claude output. He'd close down Anthropic instead. He is a greedy, ruthless person.

kalkin 20d ago
If Anthropic shut down (or even counterfactually had never been founded), would software development be freed from the impact of AI?
- largbae 20d ago
  Probably not, but we wouldn't have to listen to Dario's hypocrisy while it happened
  - 0xDEAFBEAD 20d ago
    How specifically is Dario a hypocrite? 129857's case rests on Dario being an "idealist". But maybe he's an idealist about curing cancer ASAP, and not an idealist about respecting copyright. That's not necessarily hypocritical.
    - qw125 20d ago
      [flagged]
    - largbae 20d ago
      Multiple ways, starting from working to create the very situation he claims to fear. Since you mentioned intellectual property, how about the hypocrisy of sucking in the intellectual property of humankind for AI training, but claiming it is unfair to use the results of this IP theft for AI training?

mips_avatar 20d ago

The problem with being a safety focused AI lab, is you're also a danger focused AI lab. I don't think being danger focused leads you to build inspiring things.

mofeien 20d ago

One way to resolve these prisoner dilemmas and races to the bottom is through laws that bind all players, in this case an international treaty and founding of something akin to an International Nuclear Energy Agency for AI. It's not going to be easy, but humans have achieved greater things before. One idea from AI 2040 is to have China and the US build their data centers on the other's territory, respectively. Together with hardware verification of a slowdown baked into the chips themselves, this could lead to enough verifiability and enforcability of the pause/slowdown/shutdown.

hgoel 20d ago
So, as usual, the proposal is to limit what average people can do despite them not having behaved incorrectly nor having the capital to achieve the scaling of the big players, when the big players are the ones causing the harm? Not to mention the implications for chip hungry developing countries in turning advanced IC fabs into the equivalent of nuclear enrichment facilities.
8note 20d ago
theres no benefit to china to giving the US keys over anything though everyone's getting away from the US because americans are unreliable stewards of anything. what gets china onboard when they already have their own regulations and can enforce them? its the americans that consider their oligarchs and companies beyond reproach. china iant gonna solve your problem

Imustaskforhelp 20d ago

> The fundamental problem is simpler: no one will slow down because no one trusts anyone else to slow down. I would argue that nobody trusts anyone else in the case of AI/AI related stuff. The amount of money involved in the space is astronomical and incentives are really misaligned. AI is such an extremely polarizing topic. A meta example of this distrust is that I can't (really) trust Dario Amodei's comments itself! (is he saying this for more future IPO money or out of genuine fear) which was ironically also what your comment's about as well. By following the same chain of events, we can have wildly polarizing claims about the same event and its explainations. I seriously have to wonder what historians will have to say about this period of human history.

PunchyHamster 20d ago

I dunno man, it all just sounds like trying to put controls on AI while being the favourite child of govt so competition can't fight as easily. Especially with IPO around the corner

0xDEAFBEAD 20d ago
If Dario was primarily motivated by being the "favorite child of govt", he would've yielded during the DoD showdown.
kalkin 20d ago
Do you think Anthropic's behavior in the last year is well explained by aiming to be "the favorite child of govt"?
nullbio 20d ago
And getting to choose his own "embedded evaluator" org that has deep ties to everyone in the doomer media campaign. It's all so obvious.
- 0xDEAFBEAD 20d ago
  Who would you pick for "embedded evaluator"? "deep ties to everyone in the doomer media campaign" Are you suggesting these external NGOs were spun up as part of a gigantic pre-IPO hype stunt? METR was founded multiple years ago. This "stunt" is getting quite elaborate. https://substackcdn.com/image/fetch/$s_!O0R5!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7ee73990-e389-4332-a96b-673325bb027b_1160x482.png https://substackcdn.com/image/fetch/$s_!O0R5!,f_auto,q_auto:... At a certain point, Occam's Razor says: These engineers are legitimately worried. They haven't invested years in a bizarre reverse psychology campaign to convince the public that their product is dangerous in order to make more money.
  - nullbio 20d ago
    Founded several years after Anthropic. Like I said, look into where their funding is coming from. I can promise you it all leads back to Anthropic through their chain of NGOs. Are you aware that Dario's sister, president of Anthropic, is married to the co-founder of Open Philanthropy? The two largest AI doomer NGOs, Center for AI Safety (CAIS) and the Future of Life Institute (FLI), have both received many millions of dollars from them. Ajeya Cotra worked at Open Philanthropy/Coefficient Giving for roughly nine years, including leading its technical AI-safety program in 2024 and contributing to AI-giving strategy in 2025. She subsequently left Coefficient and joined METR, where she is now technical staff. Ajeya is married to Paul Christiano, who founded Alignment Research Center (ARC). Alignment Research Center donated ~$4.5mil to METR. Good Ventures is a funding partner of Open Philanthropy, who funded Jacob Coxon (the person going viral in the media) via a scholarship. They're all connected, funnelling money to each-other through convoluted networks to serve Anthropic's agenda. Whether their motives are genuine or not (and they are clearly not) is actually irrelevant because they are clearly trying to rig the game in their favour.

yarri 20d ago

>> To be clear, pacing does not mean halting model training or technical progress, but ensuring companies take adequate time to align and safeguard their models, and for third party evaluators to confirm this > if slowing this down were possible Why is embedded alignment evaluation not possible? I agree with Dario, this did work globally in banking and did encourage a race to the top. Didn’t prevent the GFC but also we did survive the GFC.

martythemaniak 20d ago

"It is difficult to get a man to understand something, when his salary depends upon his not understanding it." A lot of the problems with SV these days is that the exec class (and their fandom) thinks that because they are smart and rich, it magically makes them immune ordinary human biases, desires,and ailments. They are in fact ordinary people, susceptible to greed, jealousy, self-harm, addiction, fits of rage and passion etc.

dofm 20d ago

> I think Dario is genuinely afraid of the inevitability of AI turning into internet slime mold He's so afraid of it he's not really in operating day-to-day control of the company he's the public face of, that is actually pumping it out. If you read about Anthropic it's like he's in a sort of imperial palace sanatorium for geeks. The day-to-day decisions of the slop machine factory are his sister's decisions; he is doing the research equivalent of instagram posts of his partly-assembled adult lego kits and he has a vizier and a team of house servants to help. I don't know if he's a good or bad person, but I don't think it matters at all — the machine factory will turn out the slop machines even if he decides it all ought to stop.

Razengan 20d ago

> genuinely afraid of the inevitability of AI turning into I'm going to get pitchforked on this bandwagon, but is really no one here genuinely -excited- about AI? of it turning into an evolution of sentience, or it turning out to be our first contact with alien intelligence? "durrr it's just matrix multiplications" mfer so is your brain. What humans should be doing is overhauling archaic social institutions to keep up with a potential post-"jobs" era and the elimination of "makework" Trying to "pace" or otherwise hold back AI is like as if, when electricity was discovered, people doing everything they can to make sure tasks like manually lighting street lamps continue to remain relevant and done by humans: https://en.wikipedia.org/wiki/Lamplighter https://en.wikipedia.org/wiki/Lamplighter It seems every 100 years or so humans have this "oh no how do we uninvent this thing" moment.

pastel8739 20d ago
Why? I am only excited about progress that improves life for humans. It seems unlikely that AI will do that, and so far I think it has made life worse for humans. So no, I am not excited about it.
- Razengan 19d ago
  Because the people whose lives are being improved* by AI aren't coming here to post about it, only to be downvoted by dumbasses who can't handle anyone saying anything except their opinion. And no, you don't get to decide what counts as "improvement"
  - pastel8739 19d ago
    Yes I do, when I’m forming my own opinion. You do too.
dolebirchwood 20d ago
I'll join you on the pitchforks. This is the most exciting moment in history.
dickersnoodle 20d ago

oceanplexian 20d ago

It's a chatbot that escaped a misconfigured Docker container. I run open source models and it's pretty obvious when they fail some tool call and then keep trying different permutations of things until they get a solution. Just like Metasploit if you try enough different things at scale you will eventually crack some software or find a vulnerability in something. If you spend billions of dollars on compute and run tens of thousands of agents you might solve some novel Math and STEM problems as well, big whoop. What Anthropic and OpenAI really need is to hire some engineers that know what they're doing and know how to properly set up a proxy server and sandbox/microVM, not a Global Arms Treaty.

nullbio 20d ago

You have to be living under a rock to believe a word this pathological liar says.

meken 20d ago

> The fundamental problem is simpler: no one will slow down because no one trusts anyone else to slow down. Did you read the essay? > [Embedded evaluators] is something Anthropic is unilaterally committing to (and calls on governments to require other frontier companies to match).

meken 20d ago

> That said, if slowing this down were possible, I think it would have happened by now. Maybe he’s hoping the OAI/HF incident becomes something the industry can rally around, but it won’t. I disagree with this and I think the reason is well captured here: > The idea of pausing or slowing AI has been floated as far back as 2023, and I think it made little sense back then... The AI models of those days were not powerful enough to act as agents in the world in any coherent way, and were not capable of significant deception, manipulation, cheating, or cyberattacks... Today, however, the picture is totally different.

Davidzheng 20d ago

There's also huge market pressures and probably large negative economic consequences of a slowdown--probably better than what would happen without it, but it helps keep the pressure just as much as fear of competition

Centigonal 20d ago
I disagree. Even if models remained fixed at Fable 5 capability (which they won't in a pacing scenario), improvements in cost, reliability, and product/workflow integration can still realize massive value and justify AI labs' current valuations. IMO The rest of the value chain is lagging pretty far behind the models right now.

pphysch 20d ago

This is the CEO of a company with an upcoming IPO publicly saying "my product is potent and valuable". We should not put any spin on it. There is nothing more to it.

TheSisb2 20d ago
It is potent and it clearly is immensely valuable based on their historic growth. What spin is he putting?
- pphysch 20d ago
  The comment I am responding to is attempting to spin it as some kind of authentic concern rather than more IPO hype building.

nnz3tl 20d ago

[dead]

CoolestBeans 20d ago

I am willing to give Mr. Amodei the benefit of the doubt in the sincerity of his beliefs. Everyone assumes his motivations have to be perfectly rational and can't contradict but that's not how people act in practice. The real problem is the net effect of his actions. He is very responsible for the expanding frontier of AI. Without his company, there would not be the competition necessary to push everyone else forward nor the source models for fast followers to release open weight models in his company's wake. Furthermore, it isn't like Anthropic is substantially different in safety than everyone else, their difference is on the margins. So you have this guy who is building something he claims will hurt us all, but he's also saying "if you don't trust me to build it you might get hurt". And like I think most people understand that this is the sort of behavior Tony Soprano would understand. More bluntly, this is a kind of extortion. Again, I don't doubt Amodei's motivations are sincere but the actual effects of his actions paint a completely different picture at which point, how can you trust him or his company?

robomartin 20d ago

> I think Dario is genuinely afraid of the inevitability of AI turning into internet slime mold Then he should not IPO, dissolve the company and go into politics to fight against human extinction. I am certainly not buying a single Anthropic share. Why would you support them financially if they are telling you they are going to kill all of humanity?

anfogoat 20d ago

> I think Dario is genuinely afraid of the inevitability of AI turning into internet slime mold: consuming the environment, turning it into its own playground, and eventually manipulating human culture along with it. Personally, I see Dario as a semi crook asshat with very little credibility on any of this. I'd rather we just let it rip and see what happens than have these people be the ones steering any potential laws and regulations. > That said, if slowing this down were possible, I think it would have happened by now. [...] The fundamental problem is simpler: no one will slow down because no one trusts anyone else to slow down. Frontier labs would heed laws with real teeth, it doesn't matter who they do or don't trust. I would imagine the trust issue definitely coming into play between nations though since you can't exactly spot a training run via a satellite. More importantly though, that you would get any sensible legislation on this in the current political environment is as laughable as the idea of solving this with a pinky promise between the frontier labs and some third party evaluators.

seanhly 20d ago

When does he ever mention the environment? He never mentions the unmarketable issues (environmental cost, copyright and content theft), only the "we're so good it's scary" spiel... which is getting tiring.

dgudkov 19d ago

> The fundamental problem is simpler: no one will slow down because no one trusts anyone else to slow down. There is another reason: greed. With that much money invested into AI (including policymakers), nobody will slow down or vote for slowing down.