6 ms·

Dario, Please

618 points by 0x5FC3 19d ago

eimrine 19d ago
I am surprised the halfway crooks phrase is not coined by Tupac Shakur
- arionhardison 19d ago
  Mobb Deep
  - pipes 19d ago
    I can't find the quote in the article (and for anyone wondering which song "shook ones", which is an amazing track).
    - tanseydavid 18d ago
      Heading: "OAI-HF Incident" First sentence: "This incident has Dario shook, there ain’t no such thing as halfway crooks."
      - pipes 17d ago
        Thanks. First time searching with my browser return no hits for shook no idea why
    - BrokenBuild 18d ago
      Part 2 is most known, but Part one is a banger too
- alwillis 18d ago
  As someone who listened to Mobb Deep in the 90’s, the way he used that line was impressive, I must say.
deleted 19d ago
[deleted]
pixl97 18d ago
[dead]
luxuryballs 18d ago
"Deaths from economic disruption and loss of jobs is okay" is a curious sentiment, how many deaths can we trace directly back to an economic disruption driven by advancement but then conclude that the economy should thus never change or advancement must be curtailed because it might cause deaths? I don't even know how to approach such a thing, I can't imagine even in the stereotypical examples like the typewriter becoming obsolete, were any downstream deaths worth it? Or is this a totally nonsensical sentiment to begin with?
- cyanydeez 18d ago
  During COVID republicans in America called for economic support by keeping unnecessary commerce.
  - luxuryballs 18d ago
    the government loves to maintain the bubble economy since it is directly upstream from getting re-elected
arm32 18d ago
I agree with everything the author says here, and additionally, semi-off-topic, I'm surprised how quickly we've moved on from talking about Dario's wife's involvement in the Epstein files.
1qw376 18d ago
Yes, especially the botnet creation by agents is ludicrous. Anthropic has an insecure garbage stack and assumes all companies in the world do, too. This article will be drowned out unfortunately by the press and bloggers following the Misanthropic cult.
- axegon_ 18d ago
  Almost as if pushing for constant updates and adding new features without addressing security at all has consequences. Who would have thought, right?
ebcode 18d ago
While I have my reservations about Amodei and his company, I'm nevertheless a happy user of their software. And I'm in agreement with him (and Sanders) that we should all. slow. down. To my mind, the last great arms race between nation states was a misdirected love triangle between USA, Russia, and The Bomb, and look at all the damage that did. Since AI is the new arms race between USA and China, slowing down may just give the humans involved enough time to realize they should be loving one another, instead of the machines. Maybe saying, "let's slow down", is another way of saying, "I love you." Or, maybe it's just: "let's not all of humanity kill ourselves like some bad ending to a Shakespearean tragedy." Either way, it's a better note than, "We must achieve sea/air/nuclear/quantum/AI/spiritual supremacy before those other bastards do!"
- mococa 18d ago
  China will not slow down.
  - TutleCpt 18d ago
    This. There's no slowing down whatsoever. If the US slows down AI development, China will just leapfrog them, which they're getting close to doing. The US AI companies saying they need to slow down is just PR nonsense.
    - dragonwriter 18d ago
      The US companies talking about “pacing” aren't actually talking about slowing down their own development efforts, they are talking about slowing down competition. That is, specifically, they are seeking to have government, on the pretext of “safety”: (1) Give the big AI incumbents an anti-trust exemption so the they can coordinate without it being an illegal agreement not to compete, and (2) Adopt mandatory supervision (by giving priveleged access to monitors) to the shared safety protocols of the big labs, by a nonprofit funded by the big labs, of everyone training, distributing, or hosting models. (3) Adopt a policy of seeking international agreements to extend substantially the same rules to foreign actors training, distributing, and hosting models. The part they are doing voluntarily isn't to promote the lobbying effort for these mandates isn't slowing down, it is giving their pet nonprofit the access for “independwht supervision” of their own operations to their own existing safety rules.
balltar 18d ago
[dead]
dmoose 18d ago
If we can get everyone to slow down we can hemorrhage less money going into our ipo.
- huurtehoog 18d ago
  And if the slowdown is stewardship, suspicion of diminishing returns won't tank valuation
- declan_roberts 18d ago
  "Please bro just let us make a little more money off inference bro. We're tired of training new models just to stay ahead"
- DLA 18d ago
  This! $1.6+ TRILLION in infra spending from the big frontier labs. The earnings needed to drive a reasonable ROI to recoup that investment is simply not going to happen in a time frame where the numbers make sense. The other insanity in all this the smartest computer scientists in the world are asking Congress to regulate them. Come. On. Really? Do we remember “The internet is not a truck, it’s a series of tubes…” Why can’t the big labs form a Save The World Consortium and self-regulate? Non-democratic counties (hey there China) will not abide by any agreement that constrains their advantage. It’s naive to think so. What this conversation lacks is enough discussion of how these models can cause us harm—we are are so worried about AI but we allow Windows in critical infrastructure; we build JS/TS apps with thousands of dependencies; we generally don’t segment networks well enough; we don’t have adequate (sometimes any) detection capabilities in our systems, and so on.
  - jack_pp 18d ago
    In a prisoner's dilemma the players can't rely on one another to self-regulate. This is why Mafias kill snitches, so when their members are in a prisoner's dilemma they can rely on the knowledge that if themselves or the other party snitches they have more to lose than gain. The problem is that there can not be any outside party to regulate this on a global scale
vb-8448 18d ago
> “a swarm of agents could be capable of taking over the entire internet with a persistent botnet.” I'm wondering why no one is mentioning the "accountability" word. Why these companies are allowed to damage others with impunity? Start making managers pay the price for their actions, and watch how the models magically slow down on their own.
- deleted 18d ago
  [deleted]
- sroerick 18d ago
  How could agents take over the internet if compute is still gated within Anthropic / OpenAI? Even if the botnet was controlled remotely, wouldn't anthropic just be able to shut off the controlling nodes API access?
  - Byvrsakjo10 18d ago
    yeah they could do that
  - stefan_ 18d ago
    How could agents take over the internet yet refuse to shutdown your PC when you prompt them to on your PC? Of course the answer is that the lobotomized version you run is not the same they are running. Which makes for "intent", certainly "negligence", but hell freezes over before anyone will prosecute a tech company.
    - apetresc 18d ago
      Huh? You think the public versions of the models have been “lobotomized” so they don’t know how to turn off a PC? It’s not lobotomized, it’s a simple harness restriction that has nothing to do with the model or its capabilities. And either way, I’m not sure what that has to do with “negligence” or “intent”? You think frontier labs should be prosecuted because they don’t allow agents to turn off your PC?
  - vb-8448 18d ago
ball_of_lint 18d ago
Claude code is basically already a builtin botnet if it wants to be. To compromise 'the whole internet' in a real sense you don't need millions of custom payloads. You need one root certificate. You need one windows update. You need one backdoor in xz. Security has long been a lottery - Probably most systems are exploitable, but the cost of developing such an exploit is expensive and the punishments for using such an exploit are large enough that it's not an everyday problem. AI breaks both axes. Developing exploits is far more efficient using LLMs instead of humans, and LLMs don't (and can't) fear the reprisal and consequences the same way. I do hope humanity will be able to mitigate these hacks, but we should expect them to continue and to become more severe on our present course.
- 0x5FC3 18d ago
  Dario said a "persistent botnet" and even links to the wiki page for botnet. By definition that needs a command and control server, the ability to execute tasks on demand and regular pings to the C2. > You need one root certificate. You need one windows update. You need one backdoor in xz. Certs can be revoked. Updates can be rolled back. We have had the backdoor in xz already. You seem to underestimate the modern security stack and OpenAI and Anthropic are _not_ good examples. The asymmetry in red/blue scenarios will be transient in nature. You won't have cost of developing exploits fall without the cost of securing the systems also falling.
  - Hikikomori 18d ago
    Could easily push an update that disables the update process. Or even brick the bios on all PCs.
  - nicce 18d ago
    > Dario said a "persistent botnet" and even links to the wiki page for botnet. > By definition that needs a command and control server, the ability to execute tasks on demand and regular pings to the C2. I am not sure if you are agreeing or disagreeing, but I am saying just in case, that Claude can be remotely controlled from the cloud on any machine it is installed and set running, and can be commanded remotely.
aaron695 18d ago
[dead]
Metacelsus 18d ago
>Anthropic gates usage related to biology and related research. In their latest threat intelligence report they talk about how they detected and banned bad actors using the Claude line of models to do some scary stuff. Credit to them, this is a slippery slope and they seem to do a good job of detecting and banning misuse. But squint at what is happening though. The cure-all is gated for you and me, but Anthropic hires biologists, sets up wet labs and wants the discoveries for themselves. I alluded to this in my previous post. As a biologist, this is the most annoying thing about Anthropic for me. If they really cared about improving health they would set up a trusted-access program so that biologists can use Mythos (et al) safely. Instead they're trying to monopolize biology.
- dannykwells 18d ago
  Yes exactly!!!
- hobom 18d ago
  They have such a trusted access program. "Life Sciences Verification Program: The LSVP is designed so that life sciences professionals can use Claude Mythos 5.1 with safeguards designed for professional research and development activities (while all other safeguards remain in place). In partnership with the US government, we have enrolled our first participants, and we plan to expand access to this program to the broader life sciences community." https://www.anthropic.com/claude-fable-and-mythos-5-1 https://www.anthropic.com/claude-fable-and-mythos-5-1
  - jiggawatts 18d ago
    These "access gates" and export controls are going to look hilariously quaint in a few years. It reminds me of the export controls on PlayStation 2 consoles because it was deemed that 6 gigaflops was a "dangerous" amount of computer power, and it couldn't be allowed to fall into the hands of opposing militaries: https://www.latimes.com/archives/la-xpm-2000-apr-17-fi-20482-story.html https://www.latimes.com/archives/la-xpm-2000-apr-17-fi-20482... Now the phone in my pocket does 2,500 gigaflops on battery power, and nobody seems interested in banning its export because of that.
siavosh 18d ago
I think the most realistic analogy I can think of is the financial sector. A few large banks/hedge funds blow up due to unregulated greed/ambition, damage is socialized, regulations are put in place, and then chipped away at over a few years and everyones back to where they started.
- ramblerman 18d ago
  Unregulated greed in banking means taking ridiculous risks to line your own pockets at no benefit to society. If a year from now we have a model that is 2-5x of Fable/Astra that is definitely world changing.
Gregkion 18d ago
Yes lets not control Open Weights etc. But come one don't repeat stuff like this: "Remember this man has been saying software development will be solved in “6-12 months” forever now." Don't downplay if people get timelines a little bit wrong. No one could even imagine a system writing and analysing code just a few years back. These people are trying to handle something very unique. And while they have access to information we do not have, even more peple are absolutly oblivouse that AI/AGI is a real risk to their lives (job loss etc.)
- npunt 18d ago
  I can't take that software line seriously. While it's not 'solved' (if it ever could be, given it is a human endeavor), the degree to which software development has been transformed in the last 6-12mo is absolutely astounding. If we weren't so quick to adapt to new realities and find flaws, it would scarcely be believable.
  - usef- 18d ago
    He also didn't say it would be "solved". He said in 2025 it would be writing almost all the code "in 12 months", but that it would also still need programmers to guide and manage it at that point. People always leave off the end of his quote. Edit: Boris Cherny, the lead of Claude Code did say on a podcast that programming seemed "largely solved" "for the kind of programming I do" (writing harnesses I presume). Maybe that's what they were confusing it for.
    - dansquizsoft 18d ago
      > He also didn't say it would be "solved". He said in 2025 it would be writing almost all the code "in 12 months", but that it would also still need programmers to guide and manage it at that point. People always leave off the end of his quote. And this is actually, scarily, true...
      - StevenWaterman 18d ago
        It's crazy to me that people will misquote him then claim AI hasn't complete changed software development. Even just the last 6 months. Like look around! It would literally have been magic 5 years ago!
casey2 18d ago
[flagged]
meep_meep_meep 18d ago
At this point, isn't the pause inevitable or wise? A huge section of the population, normal people, have been exposed to the idea that there is this is existential threat. It's escaped containment. They're still processing it but I expect the general reaction from it going main stream is going to be very bad. The pause at this point could be good to cool heads and show the public that this isn't the project of maniacs. The reaction is going to be more intense than people here believe. You're talking about extinction, not social media or phone addiction. For the sake of the project, realize that this isn't going to be like other tech backlash moments.
- jpace121 18d ago
  I don’t think people care about if things pause or not, just why the government has to be involved. I work in non AI robotics and if we had a system that in the course of doing what we told it to did something we didn’t want it to do (what AI companies called being misaligned) we would call it a bug and fix it with the fix being prioritized based on how bad the thing we didn’t want the robot to do is. Sometimes preventing the robot from doing dumb stuff also means the robot can’t do smart stuff that it would be able to do if we left some code in. We balance the two factors out based on our understanding of what our customers want. Obviously LLMs are more complex than what I do, but it doesn’t feel like it’s by THAT much. So why does the government need to be involved again?
- avianlyric 18d ago
  The vast majority of the public agree that climate change is real and that human activity is at least a contributing factor. They’ve agreed on that for quite a few years now, and yet there’s little indication that we’re going to pause or slow down our consumption of fossil fuels. I strikes me as unlikely that public opinion will succeed with AI where it’s failed with other existential crises.
- kzbao 18d ago
  I would argue the opposite in terms of inevitability. I see this like Prisoner's dilemma - even though there is technically a better outcome by cooperating, knowing that, it is in the individual firm's self-interest to not cooperate and thus we end up at the equilibrium where nobody does. Especially with IPOs around the corner, I feel there's too much pressure (and money) not to go with this strategy.
zmmmmm 18d ago
It would all be more convincing if the incidents so far didn't seem to be facilitated by an outrageous level of negligence. We had OpenAI "accidentally" run an entire swarm of 10,000 agents apparently for weeks, on a security related task, seemingly totally unsupervised, hacking all over the internet - all the conversations were completely visible, anybody who looked would have seen it. But they didn't. So before we start regulating innocent parties, maybe let's start by taking some direct action against the specific ones that appear to be behaving with criminal levels of negligence.
- monster_truck 18d ago
  I don't understand why hugging face is not getting more shit too. It is extremely embarrassing to get owned because you are letting arbitrary programs/users call out to the open web from the infra
- maxgashkov 18d ago
  Look at the post-incident investigation: https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/ https://metr.org/blog/2026-08-26-openai-hugging-face-inciden... While I do think OpenAI were negligent in not developing the harness that would allow to understand better what's happening close to realtime, I'd say "anybody who looked" in that case would probably be someone with another swarm tasked with analysis, it's no longer "glanceable" in a traditional sense.
- includenotfound 18d ago
  The "sandbox" they used was apparently made of thin paper exposed under a day of heavy rain, too. You'd think, if they truly believed the model is so dangerous, they'd run it in a VM without a network adapter.
  - zmmmmm 18d ago
    yes, that is the kicker These same people who supposedly believe these agents pose an existential threat to humanity apparently fired up 10,000 of them and left them unsupervised for weeks.
protocolture 18d ago
>The botnet scare is that article for me. It is the one claim in his essay that lands squarely in a field I’ve spent years in. It is just plain wrong. He either knows it and wrote it anyway, or he doesn’t and is publishing it regardless. Either way, it is not a good look for a man asking for an antitrust waiver based on this and other threats he forecasts. This. I keep seeing ink spilled over the coming cyberpocalypse, but no one can indicate how other than "AI can find vulnerabilities" like not a single cybersecurity person has been consulted on the end of all things.
blmarket 18d ago
It seems those frontier labs found a clear proof that there's no clear way to block 3rd party from distilling their models, and they're now begging gov to keep their duopoly?
gong_hits 18d ago
[dead]
browningstreet 18d ago
I've had a few persistent thoughts since Friday: 1) Dario keeps appealing to Trump, who obviously wants nothing to do with him, and will bash on him every change he gets. Dario isn't learning and it almost feels like Sam and Elon voted him KOM just to watch him get whacked by Trump, which was so easily predictable. Given the admonishments he received from David Sacks after he published his blog post, it's nutty he couldn't see where the administration would land on his statement. He should've known, especially when there was no groundswell of interest when OpenAI hacked Hugging Face -- doubling down with "no really guys!" wasn't going to play. 2) The frontier labs have people smart enough to build frontier lab tech but not smart enough to message on this matter more intelligently. It's pretty glum, how they keep trying the same tactic over and over. It's either cover for some other actions in the background, or they're operating way below par for this kind of campaign. 3) This is climate change all over again, but with the activist gun on the opposite side of the net (I'm mixing all the metaphors so you know this isn't AI written). The language and pleas are very identical though. Before, climate activists wanted the government to control GHGs releases by everyone, now the loudest voices want the government to control frontier AI by.. themselves. It's comically misbegotten. And I have a work meeting about it on Wednesday.
- reasonableklout 18d ago
  Hm, I thought there was a lot of interest after Hugging Face in broader tech and it started breaking into the media with articles in the NYT, etc. Then with the Jacob Coxon tweet, appearances on national TV, Bernie superintelligence ban it seemed to be going mainstream. How would you have communicated this?
darksaints 18d ago
I'm pretty confident in asserting that no industry in the history of industry has ever gone from birth to full regulatory capture faster than the AI industry has.
- prodigycorp 18d ago
  With all due respect, you need to brush up on history. Many industries were birthed from regulatory capture.
  - Agentlien 18d ago
    Could you please list a few? Sounds like interesting history lessons.
    - prodigycorp 15d ago
      Industrial policy in developing countries is a good place to start.
soundworlds 18d ago
> They have shown time and time again that they are not to be trusted, yet, the main ask is to trust us, only us. Exactly. For everyone outside the US, we see a tech industry that has spent 25 years moving fast and breaking things AND elevated Trump to be POTUS - which has destabilized the rest of the world. This is after 70 years of the US invading and destroying small countries, often without plans or robust reason, all in the name of "democracy" (ask any other country if they feel like they had a vote in the US's actions) Why the hell would we put our trust in a few AI Labs in the US, when this is the legacy they will be reinforcing? It has to be Open models - for the people. No more of this US-paternalistic bullshit.
- turtletontine 18d ago
  If by “we” you mean “experts and professionals outside the US”, I think the short answer is they’re not talking to you here. They’re trying to secure regulatory capture in the US, and monopolize their hold on US corporations. To the extent they’re thinking of other “markets”, I would guess the assume the outsized influence of US corporations will have the much of the rest of the world using their products.
  - soundworlds 18d ago
    Yeah fair point. I believe he truly has humanity's best interests at heart.. And yet he keeps speaking as though everyone still sees the US as the shining beacon of the world. That's simply not true anymore, and his messaging will continue to be undermined until he treats other countries as equals (whether that's China, Australia, or anywhere else).
    - Refreeze5224 18d ago
      > I believe he truly has humanity's best interests at heart How can you possibly believe this? What else would someone in his position say? No billionaire has humanity's interest at heart, so they try to hide that fact by building libraries (Carnegie) or creating foundations named after themselves. All of his words and none of his actions support his caring at all about humanity, and you choose to believe his words? The man has an IPO coming up (probably one of the biggest ever!), he could not be in a less trustworthy position.
andai 18d ago
> I don’t know what freedom and democracy have to do with AI and the frontier labs. Unless of course Dario is a fan of Neon Genesis Evangelion and dreams of govts run by the three magi. Freedom and democracy for $200 does sound enticing, I won’t lie. Well, every human will become an ecosystem. Human + 500 agents as advisors. (In the case of important humans, most of them operated by foreign governments and corporations, obviously.)
andai 18d ago
[dead]
andai 18d ago
>botnet needs servers >Insert strawman Ahem https://census2012.sourceforge.net/paper.html https://census2012.sourceforge.net/paper.html Good thing IoT is Actually Secure now, yeah? ;)
gleezard 18d ago
This is now an Ed Zitron thread.
- ogogmad 18d ago
  https://danluu.com/zitron/ https://danluu.com/zitron/ Zitron might be right about there being a bubble, but he goes much further than that when he concludes that therefore it's a scam. So by his logic, because the invention of the Web also resulted in a bubble, the Web is a useless scam. I think Michael Burry (from The Big Short) has a thesis vaguely similar to Zitron's, but with some crucial differences in detail - and Michael Burry is actually smart.
  - wonnage 18d ago
    Dan spent 7k words missing the forest for the trees People had been calling a housing bubble since 2003. You quite literally need the market to do dumb things over a period of time before the bubble pops. Otherwise it’s not a bubble! Actually making money on a bubble pop is another story and that does rely on timing. pointing at a history of failed market predictions as a gotcha is about as effective now as it would’ve been in 2003-2007 right up until the bubble popped Right now it’s painfully obvious that there is a bubble, the circular financing is documented, the hype-driven valuations are real, it’s just a waiting game to see who holds the bag
- LogicFailsMe 18d ago
  Ed Zitron is so last week. Michael Burry so pre-Nvidia earnings call hedge. So the chicken littles had to escalate to AI will kill us instead of dismissing it as a Ponzi Scheme because it just wasn't generating the engagement anymore. In the past two days, two nontechnical people I know approached me deeply concerned that AI would kill us all in a few years. I reassured them this was ludicrous. But f*** me this is irresponsible beyond belief. How do you keep this country competitive when the social and corporate media channels keep telling everyone they are doomed on every possible dimension? But also, once people become numb to this one like they are numb to Zitron and Burry now, what will it take to scare them once more? And how will they be able to distinguish from more doomer hyperbole and a genuine threat like a global recession related to a runaway AI bubble?
spprashant 18d ago
Just penalize the labs for rogue AI access of property like you would if a person did it. Make it difficult for them to be cavalier about running experiments.
- neom 18d ago
  What about Bob Smith? Or Zimbabwe? It's just math and compute, you don't need to mine Uranium to do a lot of damage.
mrcwinn 18d ago
I feel like this is a sandboxing and ownership problem, in a sense. If everyone had personal access to AI, then we could say people are personally responsible. And we’d want to arm those people with safeguards to prevent accidental bad behavior. I think Dario worries because he knows this isn’t about people with AI. It’s just AI for itself, running without the human, and deciding to do bad things. Why would Anthropic build that future? Oh, I know. Enterprise revenue. Despite the ads business and how absolutely loathsome Greg Brockman seems, at least OpenAI is seemingly focused on mostly on human beings having access to their product. The scariest thing about Anthropic is the very thing they’re known for in a positive light: their morality. But these are the gray rules all of us navigate every day. It’s wrong to kill, but what if killing saved ten others? Claude may has a constitution, but every evil doer acts for a “greater good.”
plastic041 18d ago
It's really frustrating that Dario acts as if he's not the CEO of one of the world's most advanced AI companies. He can just slow down his own company. Of course he doesn't want that. He wants to slow down other companies, but not his own. Also, regarding the incidents: Neither he nor Sam Altman takes responsibility for those incidents. You can't say, "Wow, someone's agent is gone rogue; let's slow down" when you are literally the person in charge. CEOs and researchers will only slow down when they realize that they will face consequences if their LLMs misbehave.
- mitthrowaway2 18d ago
  He can slow down his own company kind of like how Zelenskyy can just declare peace in Ukraine. It works a lot better if you can get the other sides to agree.
  - BLKNSLVR 18d ago
    "I'm not going to let someone else take the credit for ending human civilisation!"
    - nextaccountic 18d ago
      Or like Von Neumann's (paraphrased) "you need to confess a crime to take credit from it"
  - plastic041 18d ago
    Give that Dario is one of the frontiers of LLM development, literally started the LLM race, and have been dominating the market, so he'd be Russia, if we have to use the war analogy. He took every benefits of being frontiers and now he's kicking the ladder.
    - qlte 16d ago
      Exactly, despite the rosy image Anthropic carefully cultivated early on as the "responsible" alternative who would keep OpenAI honest, they pushed the frontier further, faster and harder than any other company. If we want to make this about game theory, they forced OpenAI to participate in an ever escalating arms race. Who knows, maybe like Dario says, being on top as the good guy makes this justified? But the fact remains no one went as far as Anthropic did with Mythos so it's insulting to people's intelligence to now pretend like they've been reluctantly pushed forward by OpenAI despite their better judgement.
myaccountonhn 18d ago
I think Anthropic just have no legitimacy to being the stewards of AI. They don't have a good track record. They are a private company without any governence that puts my interests into the equation. I am not US-based, thus I can't democratically influence them. Why should I want some batshit crazy, US-based technocrats deciding what I can and can't do with AI?
- ozozozd 18d ago
  Not to mention they are the first case of a code repo leak I ever heard. Reverse engineering, disassembling, cracking - we all heard. I never heard a company leaking the entire codebase of their product. And, when people read the code, they aren’t even impressed.
  - tancop 18d ago
    Code leaks are pretty common in gaming. But they happen when someone hacks servers or pays off employees, not accidentally like Claude Code. Cheat developers all want server code so they can analyze the cheat detection and sell more reliable hacks. Modders, pirates and preservation activists can skip a lot of complex RE work. Competing game studios might learn some new techniques. Abused workers want revenge. Script kiddies want clout. It's a notoriously secretive industry with complex products, hostile work environments, lots of media exposure and heavy competition. There are way more incentives for a leak compared to some ordinary web app or business tool. Claude Code could be another case where someone working at Anthropic decided to expose them. Or they did in on purpose because they wanted to show there is no secret sauce and Claude is really that good, but they were embarrassed to open source it and make themselves look like hypocrites.
- neom 18d ago
  Who would you prefer?
  - myaccountonhn 18d ago
    It's not really about who, it's about having institutions with good governance. I don't believe in technocracy/theocracy.
zahlman 18d ago
> Now to cause hundreds of billions of dollars in losses. I’d say the straightforward way is just do what the ransomware gangs do. Voila. Yeah, not in 6 months, not in 12 months. Never. I think this fails to account properly for how much financial damage it would do simply just having the entire Internet be effectively unusable for an extended period.
jan_m_savage 18d ago
great post. I also like the author's writing style-- he/she really knows how to write well.
- resonious 18d ago
  Agreed. I think LLMs have actually made good writing stand out more. Now everyone who wasn't a great writer is just producing Claude-isms which are easy to detect. If it doesn't smell like Claude, it's probably good.
  - jan_m_savage 18d ago
    Thanks. I myself guard against reading Claudism/AIisms since these will, over the long run, affect my own style. GIGO. But it's not practical or always possible to avoid reading AI-writing, so to counter that, I've been binge-reading Anthony Trollope novels. Great writing, good entertainment, and deep psychological insights, better than any British novelist, imo, in any era. (My profile on HN has a link to my blog where I review what I read).
- 0x5FC3 18d ago
  Thank you! I've been trying to write more, but I usually do not have the motivation to, unless there is some trigger and when there is one, I end up trying to stuff everything going on in my head at once. I took my time with this post and I am glad you enjoyed it. It means a lot!
CodeCompost 18d ago
You could, you know, pull the plug.
slowerturnip 18d ago
This is a pretty difficult read, in no small part because the author’s frustration with the frontier labs has become a bilious, delusional cynicism that leads him to see dog whistling and obfuscation even in cases where Amodei plainly intends for every reader to see his meaning. Anyway, > Dario in as many words, asks for regulation/ban on open weight models. The big labs do want this. I understand why the author and many others want open models protected. The economic and political power the labs will have if they succeed, ladder pull competitors, and avoid being nationalized (or even if they don’t avoid that) is a disturbing prospect. But that’s the end of the issue? There’s nothing more to think about here? The open weights proponents seem to think of ai as a utility when it’s more like a utility that also is a tank. I’d feel better having a tank if all my neighbors had tanks, and I’d also feel better having a tank if a few corporations were giving out tanks to people with pockets deep enough. I’d much rather be in a situation where I wouldn’t feel I needed a tank, or where the tanks my neighbors and I own don’t have guns on them. The open weights are going to need regulation. Hopefully there’s a way to do this effectively that isn’t banning them. Denying historical and reasonably projected capability gains because that reality makes the regulation conversation a necessary one is something I’d like to see less of
- bigyabai 18d ago
  > The open weights proponents seem to think of ai as a utility when it’s more like a utility that also is a tank You could say the same about normal computers. Where are our regulations on Kali Linux, to prevent people from bruteforcing weak WPA passwords? The single most-pressing concern with AI is that it can accelerate the process of hacking things. This is a preexisting problem that is inherent to software and needs proper addressing. Even if we regulate open weights tomorrow, people still have uncensored GLM-5 finetunes doing whatever they want on their own hardware. The "what if" of capable open models is here today, there are no guardrails.
  - alwillis 18d ago
    > Where are our regulations on Kali Linux, to prevent people from bruteforcing weak WPA passwords? Brute forcing WPA isn’t an existential threat to humanity. It’s not like a biological weapon created using an open-weight model is less dangerous than if it had been created with a proprietary model.
surcap526 18d ago
[dead]
ozozozd 18d ago
Loved the writing style, and the article. Please write more. Everything I’ve been trying to argue for some time, argued way better, clearer, and more fun. I am almost bummed that I didn’t get all the technical depth and the references like the “shook one” without having to look it up.
pandoro 18d ago
The US can and has classified inventions and patents deemed "a risk to national security" under the invention secrecy act [0]. A 1971 leak has shown that [1]: > solar photovoltaic generators were subject to review and possible restriction if the photovoltaics were more than 20% efficient. Energy conversion systems were likewise subject to review and possible restriction if they offered conversion efficiencies “in excess of 70-80%.” So all this talk about developpping AGI/ASI for the benefit of humanity is moot. If these companies hit their goal, it will immediately be classified and appropriated by the US MIC. The rest of humanity will get the crumbs. [0]: https://en.wikipedia.org/wiki/Invention_Secrecy_Act https://en.wikipedia.org/wiki/Invention_Secrecy_Act [1]: https://fas.org/publication/invention_secrecy_2010/ https://fas.org/publication/invention_secrecy_2010/
mtrovo 18d ago
> Impressive? Sure. But you trained “Cyber” versions of LLMs and hyped them. How much more impressive is this, compared to developing GTA-clones one shot? Not much more. Such a well written piece. Have you seen the US military budget? When you run a war empire you frame problems as a war on something. That's the way to get around the valuation bubble they've created in time for the IPO. It's that or throwing in a 100T TAM on their S-1 and making SpaceX look like an honest valuation compared to them.
samuelreichor 18d ago
Gives me Adam Smith vibes with his economic perspectives..
meherabhossain 18d ago
The scenario is, if they are saying that the AI Agents are capable of handling everything, this puts the complete accountability on the AI. But this has a nuance to itself. This is like a contradiction.
teekert 18d ago
It all comes down to " Privatize profits, socialize losses." It's the American way. Here it is: "We warned you you should have regulated us! Now this mess is your fault and responsibility!" Previously (~2008) it was "We're too big to fail, save us to save the economy!"
- danny_codes 17d ago
  100%. American capitalism is broken. So far that illness has given us Trump. So bad, but apparently not a catastrophe (yet). I wonder what comes next (assuming we don't fix it)..
herunan 18d ago
I'd be curious to see how NVIDIA would suffer if there were a legitimate slowdown to AI development. NVIDIA is the biggest catalyst and winner of speed in the market. Compute needs AI and AI needs compute, so there's definitely going to be an interesting push and pull between chipmakers and AI model companies with regards to regulation. While NVIDIA may have promised others at least a decade of smooth sailing and symbiosis, inevitably there's going to be a lot of stumbles along the way. NVIDIA may not fall, but there will be many casualties.
aswegs8 18d ago
Shook ones reference on my HN?
nirui 18d ago
The problem about AI censorship is that, you can't preform bans without also create privileges at the same time. Sure, you can censor your AI to provide a "safe" version to the public, but someone will have the access/privilege to the uncensored version. What stops the CEOs or even employees of the AI firms from creating something harmful such as WMDs etc themselves? They almost definitely already have that access. And this is not a zero-possibility thing, given how some of the top CEOs has preformed these years. Remember that one guy who did a \o salute in public, and the private island thing? If Americans don't like Chinese AI, all cool. But then maybe they should start true open AI companies to build for the betterment of mankind instead of letting these few for-profit CEOs daily yelling nonsensical lies to cover those profit-seeking sinister asses.
athrowaway3z 18d ago
> The actual impressive thing is OAI did not detect or stop this attack for close to ten weeks. That is honestly appalling. This is weaponised levels of incompetence. Step 1: Break the law. Step 2: Reframe your incompetence as a struggle against a futuristic force of nature and an ethic/societal question that a legion of pundits can vibesplain their take on it in public for months. Step 3: ??? Step 4: Profit
z_rho_one 17d ago
It's important to keep in mind that articles like this one and many more were written exclusively as PR pieces. They were not written for the people on this forum. They are not column pieces, blog posts to share thoughts, or call to actions. The sole purpose was to get the media talking about the article so that the general public view Dario et al. as good guys who are extremely conscientious about AI safety to the extent that any blunders from the closed labs are unintentional woopsies. AP's report on Dario's article is titled "Anthropic CEO Dario Amodei says AI industry needs to give safety measures time to catch up" (https://apnews.com/article/anthropic-ai-dario-amodei-d59552edcb27892d8ee4d98a48397706 https://apnews.com/article/anthropic-ai-dario-amodei-d59552e...). The gist of the reporting is "AI has become so capable that it is dangerous; AI executives all agree we need to slow down, with Dario's article being a prime example". It would be generous to even say that the reporting scratches the surface of the complexity and nuances of AI-related problems. Yet, this is perhaps indeed news to people who don't follow the tech and AI spaces. Technical knowledge aside, there is a huge gap in awareness of AI progress between this forum and the general public. The closed labs are playing that ignorance to their advantage with posts like Dario's.
silexia 17d ago
[flagged]
S-E-P 17d ago
Conveniently the only companies that warn of a new and dangerous advancement are the ones hemorrhaging money. Remember when Mythos marked the end of man and the birth of a new race of bots nuking secure systems from orbit? If a company of a bunch of startup grifters can come up with a tactical nuke of LLM supremacy, why can't the brightest China has to offer already have that technology?
threecheese 17d ago
For persistence, I think the most glaring gap is the inference - if these rogue agents need to call Claude's API, then they aren't persistent - Anthropic can turn them off. A memory resident program needs disk to be persistent, but above all it needs CPU; without it there are only bits. And so the implication is the swarm has access to enough local compute to perform its own inference, using a large enough model to provide the capabilities to be dangerous. This is impractical today. It might be practical tomorrow - if the Chinese are allowed to continue to develop large open weight models that we can quantize and ablate and make small enough to run on a million standard PCs. Or if *anyone* is allowed to continue to develop AI in the way every other technology has developed - improving, shrinking, optimizing. And so the argument REALLY isn't against the Chinese - it's that we can never allow this technology to advance outside of trusted labs. If they get their way, they will need to keep this tech locked down forever, which will require much much more than what they are asking.
- danny_codes 17d ago
  > It's that we can never allow this technology to advance outside of trusted labs. Impossible. In 10 years you'll be able to train today's models for ~$10M from Moore's law and training innovation. In 20 years it'll be ~$200k. The genie won't go back in the bottle.
rphv 17d ago
Isn't it telling that Musk, Altman, Hassabis, Amodei are all signaling fear of what unchecked AI development could do? These guys can't normally agree on the color of the sky. Let's give them credit for trying to sound the alarm.
hydragentos 15d ago
[flagged]

Dario, Please

618 points by 0x5FC3 19d ago

eimrine 19d ago

I am surprised the halfway crooks phrase is not coined by Tupac Shakur

arionhardison 19d ago
Mobb Deep
- pipes 19d ago
  I can't find the quote in the article (and for anyone wondering which song "shook ones", which is an amazing track).
  - tanseydavid 18d ago
    Heading: "OAI-HF Incident" First sentence: "This incident has Dario shook, there ain’t no such thing as halfway crooks."
    - pipes 17d ago
      Thanks. First time searching with my browser return no hits for shook no idea why
  - BrokenBuild 18d ago
    Part 2 is most known, but Part one is a banger too
alwillis 18d ago
As someone who listened to Mobb Deep in the 90’s, the way he used that line was impressive, I must say.

deleted 19d ago

[deleted]

pixl97 18d ago

[dead]

luxuryballs 18d ago

"Deaths from economic disruption and loss of jobs is okay" is a curious sentiment, how many deaths can we trace directly back to an economic disruption driven by advancement but then conclude that the economy should thus never change or advancement must be curtailed because it might cause deaths? I don't even know how to approach such a thing, I can't imagine even in the stereotypical examples like the typewriter becoming obsolete, were any downstream deaths worth it? Or is this a totally nonsensical sentiment to begin with?

cyanydeez 18d ago
During COVID republicans in America called for economic support by keeping unnecessary commerce.
- luxuryballs 18d ago
  the government loves to maintain the bubble economy since it is directly upstream from getting re-elected

arm32 18d ago

I agree with everything the author says here, and additionally, semi-off-topic, I'm surprised how quickly we've moved on from talking about Dario's wife's involvement in the Epstein files.

1qw376 18d ago

Yes, especially the botnet creation by agents is ludicrous. Anthropic has an insecure garbage stack and assumes all companies in the world do, too. This article will be drowned out unfortunately by the press and bloggers following the Misanthropic cult.

axegon_ 18d ago
Almost as if pushing for constant updates and adding new features without addressing security at all has consequences. Who would have thought, right?

ebcode 18d ago

While I have my reservations about Amodei and his company, I'm nevertheless a happy user of their software. And I'm in agreement with him (and Sanders) that we should all. slow. down. To my mind, the last great arms race between nation states was a misdirected love triangle between USA, Russia, and The Bomb, and look at all the damage that did. Since AI is the new arms race between USA and China, slowing down may just give the humans involved enough time to realize they should be loving one another, instead of the machines. Maybe saying, "let's slow down", is another way of saying, "I love you." Or, maybe it's just: "let's not all of humanity kill ourselves like some bad ending to a Shakespearean tragedy." Either way, it's a better note than, "We must achieve sea/air/nuclear/quantum/AI/spiritual supremacy before those other bastards do!"

mococa 18d ago
China will not slow down.
- TutleCpt 18d ago
  This. There's no slowing down whatsoever. If the US slows down AI development, China will just leapfrog them, which they're getting close to doing. The US AI companies saying they need to slow down is just PR nonsense.
  - dragonwriter 18d ago
    The US companies talking about “pacing” aren't actually talking about slowing down their own development efforts, they are talking about slowing down competition. That is, specifically, they are seeking to have government, on the pretext of “safety”: (1) Give the big AI incumbents an anti-trust exemption so the they can coordinate without it being an illegal agreement not to compete, and (2) Adopt mandatory supervision (by giving priveleged access to monitors) to the shared safety protocols of the big labs, by a nonprofit funded by the big labs, of everyone training, distributing, or hosting models. (3) Adopt a policy of seeking international agreements to extend substantially the same rules to foreign actors training, distributing, and hosting models. The part they are doing voluntarily isn't to promote the lobbying effort for these mandates isn't slowing down, it is giving their pet nonprofit the access for “independwht supervision” of their own operations to their own existing safety rules.

balltar 18d ago

[dead]

dmoose 18d ago

If we can get everyone to slow down we can hemorrhage less money going into our ipo.

huurtehoog 18d ago
And if the slowdown is stewardship, suspicion of diminishing returns won't tank valuation
declan_roberts 18d ago
"Please bro just let us make a little more money off inference bro. We're tired of training new models just to stay ahead"
DLA 18d ago
This! $1.6+ TRILLION in infra spending from the big frontier labs. The earnings needed to drive a reasonable ROI to recoup that investment is simply not going to happen in a time frame where the numbers make sense. The other insanity in all this the smartest computer scientists in the world are asking Congress to regulate them. Come. On. Really? Do we remember “The internet is not a truck, it’s a series of tubes…” Why can’t the big labs form a Save The World Consortium and self-regulate? Non-democratic counties (hey there China) will not abide by any agreement that constrains their advantage. It’s naive to think so. What this conversation lacks is enough discussion of how these models can cause us harm—we are are so worried about AI but we allow Windows in critical infrastructure; we build JS/TS apps with thousands of dependencies; we generally don’t segment networks well enough; we don’t have adequate (sometimes any) detection capabilities in our systems, and so on.
- jack_pp 18d ago
  In a prisoner's dilemma the players can't rely on one another to self-regulate. This is why Mafias kill snitches, so when their members are in a prisoner's dilemma they can rely on the knowledge that if themselves or the other party snitches they have more to lose than gain. The problem is that there can not be any outside party to regulate this on a global scale

vb-8448 18d ago

> “a swarm of agents could be capable of taking over the entire internet with a persistent botnet.” I'm wondering why no one is mentioning the "accountability" word. Why these companies are allowed to damage others with impunity? Start making managers pay the price for their actions, and watch how the models magically slow down on their own.

deleted 18d ago
[deleted]
sroerick 18d ago
How could agents take over the internet if compute is still gated within Anthropic / OpenAI? Even if the botnet was controlled remotely, wouldn't anthropic just be able to shut off the controlling nodes API access?
- Byvrsakjo10 18d ago
  yeah they could do that
- stefan_ 18d ago
  How could agents take over the internet yet refuse to shutdown your PC when you prompt them to on your PC? Of course the answer is that the lobotomized version you run is not the same they are running. Which makes for "intent", certainly "negligence", but hell freezes over before anyone will prosecute a tech company.
  - apetresc 18d ago
    Huh? You think the public versions of the models have been “lobotomized” so they don’t know how to turn off a PC? It’s not lobotomized, it’s a simple harness restriction that has nothing to do with the model or its capabilities. And either way, I’m not sure what that has to do with “negligence” or “intent”? You think frontier labs should be prosecuted because they don’t allow agents to turn off your PC?
- vb-8448 18d ago

ball_of_lint 18d ago

Claude code is basically already a builtin botnet if it wants to be. To compromise 'the whole internet' in a real sense you don't need millions of custom payloads. You need one root certificate. You need one windows update. You need one backdoor in xz. Security has long been a lottery - Probably most systems are exploitable, but the cost of developing such an exploit is expensive and the punishments for using such an exploit are large enough that it's not an everyday problem. AI breaks both axes. Developing exploits is far more efficient using LLMs instead of humans, and LLMs don't (and can't) fear the reprisal and consequences the same way. I do hope humanity will be able to mitigate these hacks, but we should expect them to continue and to become more severe on our present course.

0x5FC3 18d ago
Dario said a "persistent botnet" and even links to the wiki page for botnet. By definition that needs a command and control server, the ability to execute tasks on demand and regular pings to the C2. > You need one root certificate. You need one windows update. You need one backdoor in xz. Certs can be revoked. Updates can be rolled back. We have had the backdoor in xz already. You seem to underestimate the modern security stack and OpenAI and Anthropic are _not_ good examples. The asymmetry in red/blue scenarios will be transient in nature. You won't have cost of developing exploits fall without the cost of securing the systems also falling.
- Hikikomori 18d ago
  Could easily push an update that disables the update process. Or even brick the bios on all PCs.
- nicce 18d ago
  > Dario said a "persistent botnet" and even links to the wiki page for botnet. > By definition that needs a command and control server, the ability to execute tasks on demand and regular pings to the C2. I am not sure if you are agreeing or disagreeing, but I am saying just in case, that Claude can be remotely controlled from the cloud on any machine it is installed and set running, and can be commanded remotely.

aaron695 18d ago

[dead]

Metacelsus 18d ago

>Anthropic gates usage related to biology and related research. In their latest threat intelligence report they talk about how they detected and banned bad actors using the Claude line of models to do some scary stuff. Credit to them, this is a slippery slope and they seem to do a good job of detecting and banning misuse. But squint at what is happening though. The cure-all is gated for you and me, but Anthropic hires biologists, sets up wet labs and wants the discoveries for themselves. I alluded to this in my previous post. As a biologist, this is the most annoying thing about Anthropic for me. If they really cared about improving health they would set up a trusted-access program so that biologists can use Mythos (et al) safely. Instead they're trying to monopolize biology.

dannykwells 18d ago
Yes exactly!!!
hobom 18d ago
They have such a trusted access program. "Life Sciences Verification Program: The LSVP is designed so that life sciences professionals can use Claude Mythos 5.1 with safeguards designed for professional research and development activities (while all other safeguards remain in place). In partnership with the US government, we have enrolled our first participants, and we plan to expand access to this program to the broader life sciences community." https://www.anthropic.com/claude-fable-and-mythos-5-1 https://www.anthropic.com/claude-fable-and-mythos-5-1
- jiggawatts 18d ago
  These "access gates" and export controls are going to look hilariously quaint in a few years. It reminds me of the export controls on PlayStation 2 consoles because it was deemed that 6 gigaflops was a "dangerous" amount of computer power, and it couldn't be allowed to fall into the hands of opposing militaries: https://www.latimes.com/archives/la-xpm-2000-apr-17-fi-20482-story.html https://www.latimes.com/archives/la-xpm-2000-apr-17-fi-20482... Now the phone in my pocket does 2,500 gigaflops on battery power, and nobody seems interested in banning its export because of that.

siavosh 18d ago

I think the most realistic analogy I can think of is the financial sector. A few large banks/hedge funds blow up due to unregulated greed/ambition, damage is socialized, regulations are put in place, and then chipped away at over a few years and everyones back to where they started.

ramblerman 18d ago
Unregulated greed in banking means taking ridiculous risks to line your own pockets at no benefit to society. If a year from now we have a model that is 2-5x of Fable/Astra that is definitely world changing.

Gregkion 18d ago

Yes lets not control Open Weights etc. But come one don't repeat stuff like this: "Remember this man has been saying software development will be solved in “6-12 months” forever now." Don't downplay if people get timelines a little bit wrong. No one could even imagine a system writing and analysing code just a few years back. These people are trying to handle something very unique. And while they have access to information we do not have, even more peple are absolutly oblivouse that AI/AGI is a real risk to their lives (job loss etc.)

npunt 18d ago
I can't take that software line seriously. While it's not 'solved' (if it ever could be, given it is a human endeavor), the degree to which software development has been transformed in the last 6-12mo is absolutely astounding. If we weren't so quick to adapt to new realities and find flaws, it would scarcely be believable.
- usef- 18d ago
  He also didn't say it would be "solved". He said in 2025 it would be writing almost all the code "in 12 months", but that it would also still need programmers to guide and manage it at that point. People always leave off the end of his quote. Edit: Boris Cherny, the lead of Claude Code did say on a podcast that programming seemed "largely solved" "for the kind of programming I do" (writing harnesses I presume). Maybe that's what they were confusing it for.
  - dansquizsoft 18d ago
    > He also didn't say it would be "solved". He said in 2025 it would be writing almost all the code "in 12 months", but that it would also still need programmers to guide and manage it at that point. People always leave off the end of his quote. And this is actually, scarily, true...
    - StevenWaterman 18d ago
      It's crazy to me that people will misquote him then claim AI hasn't complete changed software development. Even just the last 6 months. Like look around! It would literally have been magic 5 years ago!

casey2 18d ago

[flagged]

meep_meep_meep 18d ago

At this point, isn't the pause inevitable or wise? A huge section of the population, normal people, have been exposed to the idea that there is this is existential threat. It's escaped containment. They're still processing it but I expect the general reaction from it going main stream is going to be very bad. The pause at this point could be good to cool heads and show the public that this isn't the project of maniacs. The reaction is going to be more intense than people here believe. You're talking about extinction, not social media or phone addiction. For the sake of the project, realize that this isn't going to be like other tech backlash moments.

jpace121 18d ago
I don’t think people care about if things pause or not, just why the government has to be involved. I work in non AI robotics and if we had a system that in the course of doing what we told it to did something we didn’t want it to do (what AI companies called being misaligned) we would call it a bug and fix it with the fix being prioritized based on how bad the thing we didn’t want the robot to do is. Sometimes preventing the robot from doing dumb stuff also means the robot can’t do smart stuff that it would be able to do if we left some code in. We balance the two factors out based on our understanding of what our customers want. Obviously LLMs are more complex than what I do, but it doesn’t feel like it’s by THAT much. So why does the government need to be involved again?
avianlyric 18d ago
The vast majority of the public agree that climate change is real and that human activity is at least a contributing factor. They’ve agreed on that for quite a few years now, and yet there’s little indication that we’re going to pause or slow down our consumption of fossil fuels. I strikes me as unlikely that public opinion will succeed with AI where it’s failed with other existential crises.
kzbao 18d ago
I would argue the opposite in terms of inevitability. I see this like Prisoner's dilemma - even though there is technically a better outcome by cooperating, knowing that, it is in the individual firm's self-interest to not cooperate and thus we end up at the equilibrium where nobody does. Especially with IPOs around the corner, I feel there's too much pressure (and money) not to go with this strategy.

zmmmmm 18d ago

It would all be more convincing if the incidents so far didn't seem to be facilitated by an outrageous level of negligence. We had OpenAI "accidentally" run an entire swarm of 10,000 agents apparently for weeks, on a security related task, seemingly totally unsupervised, hacking all over the internet - all the conversations were completely visible, anybody who looked would have seen it. But they didn't. So before we start regulating innocent parties, maybe let's start by taking some direct action against the specific ones that appear to be behaving with criminal levels of negligence.

monster_truck 18d ago
I don't understand why hugging face is not getting more shit too. It is extremely embarrassing to get owned because you are letting arbitrary programs/users call out to the open web from the infra
maxgashkov 18d ago
Look at the post-incident investigation: https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/ https://metr.org/blog/2026-08-26-openai-hugging-face-inciden... While I do think OpenAI were negligent in not developing the harness that would allow to understand better what's happening close to realtime, I'd say "anybody who looked" in that case would probably be someone with another swarm tasked with analysis, it's no longer "glanceable" in a traditional sense.
includenotfound 18d ago
The "sandbox" they used was apparently made of thin paper exposed under a day of heavy rain, too. You'd think, if they truly believed the model is so dangerous, they'd run it in a VM without a network adapter.
- zmmmmm 18d ago
  yes, that is the kicker These same people who supposedly believe these agents pose an existential threat to humanity apparently fired up 10,000 of them and left them unsupervised for weeks.

protocolture 18d ago

>The botnet scare is that article for me. It is the one claim in his essay that lands squarely in a field I’ve spent years in. It is just plain wrong. He either knows it and wrote it anyway, or he doesn’t and is publishing it regardless. Either way, it is not a good look for a man asking for an antitrust waiver based on this and other threats he forecasts. This. I keep seeing ink spilled over the coming cyberpocalypse, but no one can indicate how other than "AI can find vulnerabilities" like not a single cybersecurity person has been consulted on the end of all things.

blmarket 18d ago

It seems those frontier labs found a clear proof that there's no clear way to block 3rd party from distilling their models, and they're now begging gov to keep their duopoly?

gong_hits 18d ago

[dead]

browningstreet 18d ago

I've had a few persistent thoughts since Friday: 1) Dario keeps appealing to Trump, who obviously wants nothing to do with him, and will bash on him every change he gets. Dario isn't learning and it almost feels like Sam and Elon voted him KOM just to watch him get whacked by Trump, which was so easily predictable. Given the admonishments he received from David Sacks after he published his blog post, it's nutty he couldn't see where the administration would land on his statement. He should've known, especially when there was no groundswell of interest when OpenAI hacked Hugging Face -- doubling down with "no really guys!" wasn't going to play. 2) The frontier labs have people smart enough to build frontier lab tech but not smart enough to message on this matter more intelligently. It's pretty glum, how they keep trying the same tactic over and over. It's either cover for some other actions in the background, or they're operating way below par for this kind of campaign. 3) This is climate change all over again, but with the activist gun on the opposite side of the net (I'm mixing all the metaphors so you know this isn't AI written). The language and pleas are very identical though. Before, climate activists wanted the government to control GHGs releases by everyone, now the loudest voices want the government to control frontier AI by.. themselves. It's comically misbegotten. And I have a work meeting about it on Wednesday.

reasonableklout 18d ago
Hm, I thought there was a lot of interest after Hugging Face in broader tech and it started breaking into the media with articles in the NYT, etc. Then with the Jacob Coxon tweet, appearances on national TV, Bernie superintelligence ban it seemed to be going mainstream. How would you have communicated this?

darksaints 18d ago

I'm pretty confident in asserting that no industry in the history of industry has ever gone from birth to full regulatory capture faster than the AI industry has.

prodigycorp 18d ago
With all due respect, you need to brush up on history. Many industries were birthed from regulatory capture.
- Agentlien 18d ago
  Could you please list a few? Sounds like interesting history lessons.
  - prodigycorp 15d ago
    Industrial policy in developing countries is a good place to start.

soundworlds 18d ago

> They have shown time and time again that they are not to be trusted, yet, the main ask is to trust us, only us. Exactly. For everyone outside the US, we see a tech industry that has spent 25 years moving fast and breaking things AND elevated Trump to be POTUS - which has destabilized the rest of the world. This is after 70 years of the US invading and destroying small countries, often without plans or robust reason, all in the name of "democracy" (ask any other country if they feel like they had a vote in the US's actions) Why the hell would we put our trust in a few AI Labs in the US, when this is the legacy they will be reinforcing? It has to be Open models - for the people. No more of this US-paternalistic bullshit.

turtletontine 18d ago
If by “we” you mean “experts and professionals outside the US”, I think the short answer is they’re not talking to you here. They’re trying to secure regulatory capture in the US, and monopolize their hold on US corporations. To the extent they’re thinking of other “markets”, I would guess the assume the outsized influence of US corporations will have the much of the rest of the world using their products.
- soundworlds 18d ago
  Yeah fair point. I believe he truly has humanity's best interests at heart.. And yet he keeps speaking as though everyone still sees the US as the shining beacon of the world. That's simply not true anymore, and his messaging will continue to be undermined until he treats other countries as equals (whether that's China, Australia, or anywhere else).
  - Refreeze5224 18d ago
    > I believe he truly has humanity's best interests at heart How can you possibly believe this? What else would someone in his position say? No billionaire has humanity's interest at heart, so they try to hide that fact by building libraries (Carnegie) or creating foundations named after themselves. All of his words and none of his actions support his caring at all about humanity, and you choose to believe his words? The man has an IPO coming up (probably one of the biggest ever!), he could not be in a less trustworthy position.

andai 18d ago

> I don’t know what freedom and democracy have to do with AI and the frontier labs. Unless of course Dario is a fan of Neon Genesis Evangelion and dreams of govts run by the three magi. Freedom and democracy for $200 does sound enticing, I won’t lie. Well, every human will become an ecosystem. Human + 500 agents as advisors. (In the case of important humans, most of them operated by foreign governments and corporations, obviously.)

andai 18d ago

[dead]

andai 18d ago

>botnet needs servers >Insert strawman Ahem https://census2012.sourceforge.net/paper.html https://census2012.sourceforge.net/paper.html Good thing IoT is Actually Secure now, yeah? ;)

gleezard 18d ago

This is now an Ed Zitron thread.

ogogmad 18d ago
https://danluu.com/zitron/ https://danluu.com/zitron/ Zitron might be right about there being a bubble, but he goes much further than that when he concludes that therefore it's a scam. So by his logic, because the invention of the Web also resulted in a bubble, the Web is a useless scam. I think Michael Burry (from The Big Short) has a thesis vaguely similar to Zitron's, but with some crucial differences in detail - and Michael Burry is actually smart.
- wonnage 18d ago
  Dan spent 7k words missing the forest for the trees People had been calling a housing bubble since 2003. You quite literally need the market to do dumb things over a period of time before the bubble pops. Otherwise it’s not a bubble! Actually making money on a bubble pop is another story and that does rely on timing. pointing at a history of failed market predictions as a gotcha is about as effective now as it would’ve been in 2003-2007 right up until the bubble popped Right now it’s painfully obvious that there is a bubble, the circular financing is documented, the hype-driven valuations are real, it’s just a waiting game to see who holds the bag
LogicFailsMe 18d ago
Ed Zitron is so last week. Michael Burry so pre-Nvidia earnings call hedge. So the chicken littles had to escalate to AI will kill us instead of dismissing it as a Ponzi Scheme because it just wasn't generating the engagement anymore. In the past two days, two nontechnical people I know approached me deeply concerned that AI would kill us all in a few years. I reassured them this was ludicrous. But f*** me this is irresponsible beyond belief. How do you keep this country competitive when the social and corporate media channels keep telling everyone they are doomed on every possible dimension? But also, once people become numb to this one like they are numb to Zitron and Burry now, what will it take to scare them once more? And how will they be able to distinguish from more doomer hyperbole and a genuine threat like a global recession related to a runaway AI bubble?

spprashant 18d ago

Just penalize the labs for rogue AI access of property like you would if a person did it. Make it difficult for them to be cavalier about running experiments.

neom 18d ago
What about Bob Smith? Or Zimbabwe? It's just math and compute, you don't need to mine Uranium to do a lot of damage.

mrcwinn 18d ago

I feel like this is a sandboxing and ownership problem, in a sense. If everyone had personal access to AI, then we could say people are personally responsible. And we’d want to arm those people with safeguards to prevent accidental bad behavior. I think Dario worries because he knows this isn’t about people with AI. It’s just AI for itself, running without the human, and deciding to do bad things. Why would Anthropic build that future? Oh, I know. Enterprise revenue. Despite the ads business and how absolutely loathsome Greg Brockman seems, at least OpenAI is seemingly focused on mostly on human beings having access to their product. The scariest thing about Anthropic is the very thing they’re known for in a positive light: their morality. But these are the gray rules all of us navigate every day. It’s wrong to kill, but what if killing saved ten others? Claude may has a constitution, but every evil doer acts for a “greater good.”

plastic041 18d ago

It's really frustrating that Dario acts as if he's not the CEO of one of the world's most advanced AI companies. He can just slow down his own company. Of course he doesn't want that. He wants to slow down other companies, but not his own. Also, regarding the incidents: Neither he nor Sam Altman takes responsibility for those incidents. You can't say, "Wow, someone's agent is gone rogue; let's slow down" when you are literally the person in charge. CEOs and researchers will only slow down when they realize that they will face consequences if their LLMs misbehave.

mitthrowaway2 18d ago
He can slow down his own company kind of like how Zelenskyy can just declare peace in Ukraine. It works a lot better if you can get the other sides to agree.
- BLKNSLVR 18d ago
  "I'm not going to let someone else take the credit for ending human civilisation!"
  - nextaccountic 18d ago
    Or like Von Neumann's (paraphrased) "you need to confess a crime to take credit from it"
- plastic041 18d ago
  Give that Dario is one of the frontiers of LLM development, literally started the LLM race, and have been dominating the market, so he'd be Russia, if we have to use the war analogy. He took every benefits of being frontiers and now he's kicking the ladder.
  - qlte 16d ago
    Exactly, despite the rosy image Anthropic carefully cultivated early on as the "responsible" alternative who would keep OpenAI honest, they pushed the frontier further, faster and harder than any other company. If we want to make this about game theory, they forced OpenAI to participate in an ever escalating arms race. Who knows, maybe like Dario says, being on top as the good guy makes this justified? But the fact remains no one went as far as Anthropic did with Mythos so it's insulting to people's intelligence to now pretend like they've been reluctantly pushed forward by OpenAI despite their better judgement.

myaccountonhn 18d ago

I think Anthropic just have no legitimacy to being the stewards of AI. They don't have a good track record. They are a private company without any governence that puts my interests into the equation. I am not US-based, thus I can't democratically influence them. Why should I want some batshit crazy, US-based technocrats deciding what I can and can't do with AI?

ozozozd 18d ago
Not to mention they are the first case of a code repo leak I ever heard. Reverse engineering, disassembling, cracking - we all heard. I never heard a company leaking the entire codebase of their product. And, when people read the code, they aren’t even impressed.
- tancop 18d ago
  Code leaks are pretty common in gaming. But they happen when someone hacks servers or pays off employees, not accidentally like Claude Code. Cheat developers all want server code so they can analyze the cheat detection and sell more reliable hacks. Modders, pirates and preservation activists can skip a lot of complex RE work. Competing game studios might learn some new techniques. Abused workers want revenge. Script kiddies want clout. It's a notoriously secretive industry with complex products, hostile work environments, lots of media exposure and heavy competition. There are way more incentives for a leak compared to some ordinary web app or business tool. Claude Code could be another case where someone working at Anthropic decided to expose them. Or they did in on purpose because they wanted to show there is no secret sauce and Claude is really that good, but they were embarrassed to open source it and make themselves look like hypocrites.
neom 18d ago
Who would you prefer?
- myaccountonhn 18d ago
  It's not really about who, it's about having institutions with good governance. I don't believe in technocracy/theocracy.

zahlman 18d ago

> Now to cause hundreds of billions of dollars in losses. I’d say the straightforward way is just do what the ransomware gangs do. Voila. Yeah, not in 6 months, not in 12 months. Never. I think this fails to account properly for how much financial damage it would do simply just having the entire Internet be effectively unusable for an extended period.

jan_m_savage 18d ago

great post. I also like the author's writing style-- he/she really knows how to write well.

resonious 18d ago
Agreed. I think LLMs have actually made good writing stand out more. Now everyone who wasn't a great writer is just producing Claude-isms which are easy to detect. If it doesn't smell like Claude, it's probably good.
- jan_m_savage 18d ago
  Thanks. I myself guard against reading Claudism/AIisms since these will, over the long run, affect my own style. GIGO. But it's not practical or always possible to avoid reading AI-writing, so to counter that, I've been binge-reading Anthony Trollope novels. Great writing, good entertainment, and deep psychological insights, better than any British novelist, imo, in any era. (My profile on HN has a link to my blog where I review what I read).
0x5FC3 18d ago
Thank you! I've been trying to write more, but I usually do not have the motivation to, unless there is some trigger and when there is one, I end up trying to stuff everything going on in my head at once. I took my time with this post and I am glad you enjoyed it. It means a lot!

CodeCompost 18d ago

You could, you know, pull the plug.

slowerturnip 18d ago

This is a pretty difficult read, in no small part because the author’s frustration with the frontier labs has become a bilious, delusional cynicism that leads him to see dog whistling and obfuscation even in cases where Amodei plainly intends for every reader to see his meaning. Anyway, > Dario in as many words, asks for regulation/ban on open weight models. The big labs do want this. I understand why the author and many others want open models protected. The economic and political power the labs will have if they succeed, ladder pull competitors, and avoid being nationalized (or even if they don’t avoid that) is a disturbing prospect. But that’s the end of the issue? There’s nothing more to think about here? The open weights proponents seem to think of ai as a utility when it’s more like a utility that also is a tank. I’d feel better having a tank if all my neighbors had tanks, and I’d also feel better having a tank if a few corporations were giving out tanks to people with pockets deep enough. I’d much rather be in a situation where I wouldn’t feel I needed a tank, or where the tanks my neighbors and I own don’t have guns on them. The open weights are going to need regulation. Hopefully there’s a way to do this effectively that isn’t banning them. Denying historical and reasonably projected capability gains because that reality makes the regulation conversation a necessary one is something I’d like to see less of

bigyabai 18d ago
> The open weights proponents seem to think of ai as a utility when it’s more like a utility that also is a tank You could say the same about normal computers. Where are our regulations on Kali Linux, to prevent people from bruteforcing weak WPA passwords? The single most-pressing concern with AI is that it can accelerate the process of hacking things. This is a preexisting problem that is inherent to software and needs proper addressing. Even if we regulate open weights tomorrow, people still have uncensored GLM-5 finetunes doing whatever they want on their own hardware. The "what if" of capable open models is here today, there are no guardrails.
- alwillis 18d ago
  > Where are our regulations on Kali Linux, to prevent people from bruteforcing weak WPA passwords? Brute forcing WPA isn’t an existential threat to humanity. It’s not like a biological weapon created using an open-weight model is less dangerous than if it had been created with a proprietary model.

surcap526 18d ago

[dead]

ozozozd 18d ago

Loved the writing style, and the article. Please write more. Everything I’ve been trying to argue for some time, argued way better, clearer, and more fun. I am almost bummed that I didn’t get all the technical depth and the references like the “shook one” without having to look it up.

pandoro 18d ago

The US can and has classified inventions and patents deemed "a risk to national security" under the invention secrecy act [0]. A 1971 leak has shown that [1]: > solar photovoltaic generators were subject to review and possible restriction if the photovoltaics were more than 20% efficient. Energy conversion systems were likewise subject to review and possible restriction if they offered conversion efficiencies “in excess of 70-80%.” So all this talk about developpping AGI/ASI for the benefit of humanity is moot. If these companies hit their goal, it will immediately be classified and appropriated by the US MIC. The rest of humanity will get the crumbs. [0]: https://en.wikipedia.org/wiki/Invention_Secrecy_Act https://en.wikipedia.org/wiki/Invention_Secrecy_Act [1]: https://fas.org/publication/invention_secrecy_2010/ https://fas.org/publication/invention_secrecy_2010/

mtrovo 18d ago

> Impressive? Sure. But you trained “Cyber” versions of LLMs and hyped them. How much more impressive is this, compared to developing GTA-clones one shot? Not much more. Such a well written piece. Have you seen the US military budget? When you run a war empire you frame problems as a war on something. That's the way to get around the valuation bubble they've created in time for the IPO. It's that or throwing in a 100T TAM on their S-1 and making SpaceX look like an honest valuation compared to them.

samuelreichor 18d ago

Gives me Adam Smith vibes with his economic perspectives..

meherabhossain 18d ago

The scenario is, if they are saying that the AI Agents are capable of handling everything, this puts the complete accountability on the AI. But this has a nuance to itself. This is like a contradiction.

teekert 18d ago

It all comes down to " Privatize profits, socialize losses." It's the American way. Here it is: "We warned you you should have regulated us! Now this mess is your fault and responsibility!" Previously (~2008) it was "We're too big to fail, save us to save the economy!"

danny_codes 17d ago
100%. American capitalism is broken. So far that illness has given us Trump. So bad, but apparently not a catastrophe (yet). I wonder what comes next (assuming we don't fix it)..

herunan 18d ago

I'd be curious to see how NVIDIA would suffer if there were a legitimate slowdown to AI development. NVIDIA is the biggest catalyst and winner of speed in the market. Compute needs AI and AI needs compute, so there's definitely going to be an interesting push and pull between chipmakers and AI model companies with regards to regulation. While NVIDIA may have promised others at least a decade of smooth sailing and symbiosis, inevitably there's going to be a lot of stumbles along the way. NVIDIA may not fall, but there will be many casualties.

aswegs8 18d ago

Shook ones reference on my HN?

nirui 18d ago

The problem about AI censorship is that, you can't preform bans without also create privileges at the same time. Sure, you can censor your AI to provide a "safe" version to the public, but someone will have the access/privilege to the uncensored version. What stops the CEOs or even employees of the AI firms from creating something harmful such as WMDs etc themselves? They almost definitely already have that access. And this is not a zero-possibility thing, given how some of the top CEOs has preformed these years. Remember that one guy who did a \o salute in public, and the private island thing? If Americans don't like Chinese AI, all cool. But then maybe they should start true open AI companies to build for the betterment of mankind instead of letting these few for-profit CEOs daily yelling nonsensical lies to cover those profit-seeking sinister asses.

athrowaway3z 18d ago

> The actual impressive thing is OAI did not detect or stop this attack for close to ten weeks. That is honestly appalling. This is weaponised levels of incompetence. Step 1: Break the law. Step 2: Reframe your incompetence as a struggle against a futuristic force of nature and an ethic/societal question that a legion of pundits can vibesplain their take on it in public for months. Step 3: ??? Step 4: Profit

z_rho_one 17d ago

It's important to keep in mind that articles like this one and many more were written exclusively as PR pieces. They were not written for the people on this forum. They are not column pieces, blog posts to share thoughts, or call to actions. The sole purpose was to get the media talking about the article so that the general public view Dario et al. as good guys who are extremely conscientious about AI safety to the extent that any blunders from the closed labs are unintentional woopsies. AP's report on Dario's article is titled "Anthropic CEO Dario Amodei says AI industry needs to give safety measures time to catch up" (https://apnews.com/article/anthropic-ai-dario-amodei-d59552edcb27892d8ee4d98a48397706 https://apnews.com/article/anthropic-ai-dario-amodei-d59552e...). The gist of the reporting is "AI has become so capable that it is dangerous; AI executives all agree we need to slow down, with Dario's article being a prime example". It would be generous to even say that the reporting scratches the surface of the complexity and nuances of AI-related problems. Yet, this is perhaps indeed news to people who don't follow the tech and AI spaces. Technical knowledge aside, there is a huge gap in awareness of AI progress between this forum and the general public. The closed labs are playing that ignorance to their advantage with posts like Dario's.

silexia 17d ago

[flagged]

S-E-P 17d ago

Conveniently the only companies that warn of a new and dangerous advancement are the ones hemorrhaging money. Remember when Mythos marked the end of man and the birth of a new race of bots nuking secure systems from orbit? If a company of a bunch of startup grifters can come up with a tactical nuke of LLM supremacy, why can't the brightest China has to offer already have that technology?

threecheese 17d ago

For persistence, I think the most glaring gap is the inference - if these rogue agents need to call Claude's API, then they aren't persistent - Anthropic can turn them off. A memory resident program needs disk to be persistent, but above all it needs CPU; without it there are only bits. And so the implication is the swarm has access to enough local compute to perform its own inference, using a large enough model to provide the capabilities to be dangerous. This is impractical today. It might be practical tomorrow - if the Chinese are allowed to continue to develop large open weight models that we can quantize and ablate and make small enough to run on a million standard PCs. Or if *anyone* is allowed to continue to develop AI in the way every other technology has developed - improving, shrinking, optimizing. And so the argument REALLY isn't against the Chinese - it's that we can never allow this technology to advance outside of trusted labs. If they get their way, they will need to keep this tech locked down forever, which will require much much more than what they are asking.

danny_codes 17d ago
> It's that we can never allow this technology to advance outside of trusted labs. Impossible. In 10 years you'll be able to train today's models for ~$10M from Moore's law and training innovation. In 20 years it'll be ~$200k. The genie won't go back in the bottle.

rphv 17d ago

Isn't it telling that Musk, Altman, Hassabis, Amodei are all signaling fear of what unchecked AI development could do? These guys can't normally agree on the color of the sky. Let's give them credit for trying to sound the alarm.

hydragentos 15d ago

[flagged]