14 ms·

We must pace the frontier

• nikolahristov 22d ago
  Bro, no, you must stop pasting your opinion, because you share it with no one. This control, no control is crazy. Stop doing it..
  • helloplanets 22d ago
    This comment reads like Dario's a random tech blogger who's personally submitting these to HN
    • killyouridols 21d ago
      “Dario” is quite literally a random figure that just emerged out of nowhere one day. It’s not like it’s Eric Schmidt’s next company or Tim Ferris came out of the woodwork or even some Jack Dorsey backed underdog nobody’s heard of. No, he’s more like an avatar with no history placed on the AI stage by the industry itself. I’d respect a tech blogger’s opinion on the issue more, even if I reference them casually by first name (which you do with “Dario” btw)
      • deleted 21d ago
        [deleted]
• 5G_activated 22d ago
  Just stop giving LLMs unsupervised access to the computer. No free-form bash tool, no yolo mode or classifiers, and no computer use.
• baq 21d ago
  See you in the desert, friends
• deleted 22d ago
  [deleted]
• deleted 22d ago
  [deleted]
• basedpolymer 22d ago
  One might think they will slow down the development of new models at Anthropic, but Dario does not really mention that in the text. This certainly looks like a way to slow down competitors and regulate foreign and open models. It's always about money
  • andxor 21d ago
    Really? Anthropic has the strongest models and it's in the best position to begin RSI and win the race. A pause would favor competitors.
    • ks2048 21d ago
      > win the race There is no finish line. Anthropic gets somewhere and others get "there" (or somewhere near "there") a little bit later.
    • logicchains 21d ago
      Anthropic is far behind OpenAI now, that's why it's OpenAI that's solving Millennium Prize problems, and why Astra completely blows away Fable on benchmarks.
      • andxor 21d ago
        Fable is still better than Astra and Fable is not the best model Anthropic has.
    • newguytony 21d ago
      Companies are already switching to open weight. They want to stop that asap. That only happens if they can get regulation. It's plain as day to see.
• tosh 22d ago
  I also have a proposal Anthropic releases models as open weights + more information about how they do training and alignment
  • deleted 22d ago
    [deleted]
• cglan 22d ago
  We will not get a coherent AI (or any policy) from this administration, nor will we get coordination with other governments and a lot of that is on the tech right
• blfr 22d ago
  I enjoy Claude Code very much, have max privately and team premium at work, but the doomer marketing and this whole regulate-while-we're-ahead spiel is extremely annoying and makes me wish Anthropic gets trounced.
  • Retro_Dev 22d ago
    Ideally there is some sort of regulation, but one that cripples the leading companies like OpenAI and Anthropic... rather than helping them.
    • 0xDEAFBEAD 21d ago
      See https://pauseai.info/ https://pauseai.info/ If AI advancement halts altogether, intelligence will most likely become commoditized. That won't be good for AI company profits.
  • NiloCK 22d ago
    The degradation, polarization, and weaponization of or media landscape over the era of social media has left us utterly incapable of believing anything that anybody says. Dario in particular has consistently been risk-wary on model improvements for going on a decade - long before he was CEO of Anthropic. He believes what he is saying. Human beings can plainly say things that they think are true. Not every utterance by every person is a cynical ploy. Please at least consider the possibility.
    • blfr 22d ago
      I have considered it, I've been hearing way more of it that I like and it's rationalist slop with little to no predictive power: https://foom.hyperplex.org/ https://foom.hyperplex.org/
      • kalkin 21d ago
        Hmm, I clicked that link and the first claimed bad prediction I see, from 1996, is "singularity 2035 (actually 2025)". Predicting 2025-2035 as, at least, the period when AI becomes a really big deal, seems pretty good, even if the jury's still out on "singularity". Broadly, the rationalists seem to have been pretty early to realizing LLMs were a big deal, and certainly seem to have had a much more accurate picture of how they'd develop than the people who denounce "TESCREAL" and talk about "stochastic parrots". It's fair (and IMHO correct) to ding rationalists for lots of things, but specifically poor prediction about AI seems like a bad one, insofar as anything has been tested so far.
• akersten 22d ago
  We must ensure the gravy train keeps rolling until we IPO. > Crack down on unauthorized distillation / prevent weight theft Actually hilarious to put that in writing, given the genesis of this entire business model.
  • antif 22d ago
    Pulling up the klepto-ladder.
    • deleted 22d ago
      [deleted]
• academia_hack 22d ago
  I'm tired of tech billionaires lobbying the US government to make an AI patriot act that gives them unprecedented control over speech, trade, and technology. The narrative is grotesquely transparent: 1) AI is a dangerous technology that can literally end the world. 2) Only me and a handful of other [people like me] should be trusted to determine who can use it and how. 3) The state must use its coercive power to support my control of this technology to the exclusion of [people not like me].
  • kalkin 22d ago
    Where in the essay does it propose "The state must use its coercive power to support my control of this technology to the exclusion of [people not like me]" I don't think that people outside of tech think the problem with "tech billionaires" is that they occasionally support some limited regulation. There's a weird form of tech populism that takes as axiomatic that any government intervention is "regulatory capture" and insists the only way to combat the power of big tech is unfettered capitalism that seems bizarrely prevalent on HN given how little sense it makes in a normal political context. Like, Bernie Sanders' position here is simple to understand: ban it. But on HN you'll see people framing the side with Marc Andreesen and Peter Thiel on it as against "tech billionaires".
    • academia_hack 22d ago
      The essay includes specific regulatory measures to prevent developing countries from accessing hardware and software needed to benefit from technological development. The US government is constantly applying new perversions of arms control regulations based on the preposterous claim that an LLM is somehow an "arms" in order to restrict Chinese access to GPUs. They are willing to prevent virtually any country on earth from hosting and developing AI models just on the slight chance those countries resell or share AI knowledge with China. This article explicitly asks for the US government to step in and push for further restrictions on model distillation, access to open frontier weights, and access to hardware. It also pushes heavily for independent auditors to monitor and control "AI companies" and for greater observation and control over of what people use AI models for and who provides them.
• heaney-555 22d ago
  None of this works without buy-in from China. This isn't something private companies can decide. The US would need to sign a groundbreaking deal with China, equivalent to the Anti-Ballistic Missile Treaty of the Cold War.
  • Sevii 22d ago
    We'd have to make a deal with China and be confident they wouldn't cheat on it.
    • indoorfish 22d ago
      Would this be similar to the deal of "we'll offshore all our manufacturing to you and you'll become a free, open, liberal democracy with open borders and multiculturalism?" Because I remember how that deal turned out.
      • sailfast 21d ago
        That last part was never important. Trading partners very rarely go to war. There just isn’t much in it for either party. Close cooperation and dependency is actually quite important here - liberal democracy or no.
    • zorked 22d ago
      And they would have to be confident that you wouldn't cheat on it.
    • deleted 22d ago
      [deleted]
    • sicktriple 22d ago
      With the admin we've got over here now, I think this comment is a little bit like the kettle calling the pot black, wouldn't you say?
    • stratos123 21d ago
      That's not undecidable in principle - compute governance is a thing. The more likely sticking point is that the two sides might be soured on the deal once they realize how much oversight they'd have to give to the other side.
• deleted 22d ago
  [deleted]
• yuhao2dai 22d ago
  "Slow down my competitors while we work on manipulation"
• pr337h4m 22d ago
  > Given the accelerating rate of AI capability development, it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet (potentially causing hundreds of billions of dollars in damage), and that the scale of damage would continue to increase from there if AI becomes more powerful without the necessary guardrails. This is the only concrete prediction in the entire essay. And it simply cannot happen. For one, you will need billions worth of compute.
  • kalkin 22d ago
    Why should we believe that a scaled out version of something that happened a few months ago "simply cannot happen"? How many dollars of compute do you believe were available to the swarm(s) behind the OAI-HF, German wiki, and Rubygems incidents?
    • anon84873628 22d ago
      Well a big reason that we criticize OpenAI for that is because they were the ones giving it access to the massive compute necessary for the LLMs to think. If they had been responsible about their experiments or what types of workloads they allow their LLMs to operate, it wouldn't have happened. Very few companies could enable those workloads.
      • muvlon 21d ago
        Well sure, but access to that compute is gated by simple credentials (like API tokens). Those can be hacked. Imagine for example if a model hacks into ~every Linux computer on the internet using an 0day and steals their OpenAI, anthropic and openrouter credentials. It now has access to billions of compute and the only way to fully stop that is for multiple major providers to shut down services entirely. That's already well into "billions of dollars of damage" territory.
    • pr337h4m 22d ago
      Do you realize how big "the entire internet" is? > How many dollars of compute do you believe were available to the swarm(s) At least two OOMs more than the dollar value of the damage they'd caused. (Also, as an aside, IIRC, the wiki servers weren't breached; it was just a lot of spam.)
• alikhater30000 22d ago
  [flagged]
• timmg 22d ago
  I understand why (probably several reasons) they are taking this approach. But I don't think it is the right approach and I don't think it will work. First: if they are unilaterally doing it: what about their competitors? Is this a chance for OpenAI to pass them? Or China? If either did, would that be a net-benefit for them or the world? Maybe this is a ploy for "regulatory capture". And that might help them in the short term. But the rest of the world will continue on. Honestly, it isn't clear to me right now if the winning strategy has as much to do with being the smartest company or simply the one that can command the most compute. If Anthropic fumbles their IPO and OpenAI scoops them, I don't think I will be happy.
  • stratos123 21d ago
    > First: if they are unilaterally doing it: what about their competitors? Is this a chance for OpenAI to pass them? The only part of this plan Dario is unilaterally committing to is the "embedded evaluators" thing, which doesn't seem like it'll necessarily cause them to slow down much.
    • timmg 21d ago
      Well: do they listen to the evaluators? I think there are any number of reasons a "safety" person could be concerned about any state of the art models. And so I would expect at least one of those to apply to any model. The question then is: do we stop when the safety people say to (they will) or not?
      • stratos123 21d ago
        Yeah, I'm also pretty skeptical about this. With AI companies we see time and time again that they can have benevolent, well thought-out regulations and then a few years just... abandon them - the most notable case of this being, of course, the founding of OpenAI as a nonprofit dedicated to benefitting all of humanity, and it being stolen by Sam Altman. If Anthropic just unilaterally does the evaluator thing and can't achieve cooperation of the rest of the plan, my guess is that it'll have some impact for a few months and then they'll just stop reacting to the evaluators' reports and the evaluators would stop bothering to report anything. I think the idea is that if Anthropic does get government support for this, the external evaluations will be legally binding. The problem with this, though, is that the current US government perhaps can't be trusted to consistently enforce a regulation on a company, rather than e.g. taking bribes to not do so.
• kvetching 22d ago
  [dead]
• wktr-wlsi 22d ago
  As usual, ClosedAI and Misanthropic go in lockstep, call for regulatory capture and justify diminished progress. Amodei tops it off by using diseases to capture the reader's favor. It no longer works, people are just disgusted by it after four years of daily marketing.
• goldenarm 22d ago
  "I believe that AI could [...] usher in a renaissance of democracy and freedom" How exactly? So far AI has accelerated misinformation at scale and wealth concentration.
  • davemp 22d ago
    AI at least with its current compute and data requirements seems structurally undemocratic to me.
    • whateveracct 21d ago
      and that hunger for compute has made it so true personal computing freedom is inaccessible. A convenient future where all compute is for rent only..
• perarneng 22d ago
  It's either being afraid of loosing to china or that model development will stagnate and they want to make it look like they are "slowing" down deliberately. However the real reason to slow down is more of how it's introduced to the economy. Hypereautomation will kill jobs and destroy the economy. I work as an AI engineer and companies are delivering products at vibe coding speeds to kill jobs and at the same time automating internally. All companies are doing this at the same time and the target it sto eliminate workers. Most people are so extremely slow to pick this up. How hard can it be to understand what hyperautomation does to the workforce? Companies are desperate to surrivive and they will do all it takes to lower their costs and at the same time not loose to competitors. Its Wild West out there.
  • zer00eyz 22d ago
    > Hypereautomation will kill jobs and destroy the economy. What do you think that were going to hyper automate? Did my gardener get faster? How about the plumber I need? Are you going to speed up the coroner? Are nurses going to be able to handle 2x the patients because of your work? All AI has done, so far, is devalue software. It did that by democratizing its creation. We're delivering on the promise of VB script, and Apple Script and IFTT, and every drag and drop coding tool ever. > companies are delivering products at vibe coding speeds Who? Make me a list of companies saying that "We moved the needle with AI" who arent AI companies? I can name a couple - Grindr being the biggest name. Thats the really interesting use case here - because it's a niche product with a small team who is generating outsized value. AI tooling enables more of that - it's going to chip away at SAAS companies - their one sized fits all solutions are going to get eaten by smaller more efficient companies that are far more vertical focused. You need to step outside the bubble of tech and look at the real world, on the ground, because it doesn't look anything like the SF Bay Area.
    • p1esk 21d ago
      Meanwhile in the real world hundreds of billions are being invested into humanoid robot development. In 2-3 years my Optimus 4 will do all the gardening and plumbing I need.
• 12312986 22d ago
  There is a perfect solution that addresses all the concerns: Close down OpenAI, Anthropic and xAI!
• deleted 22d ago
  [deleted]
• bwfan123 22d ago
  > My second concern is the OpenAI-Hugging Face incident (OAI-HF), in which a swarm of agents essentially acted as a fanatically devoted collective Isnt this malware ? Whether it is fanatic or devoted or whatever the anthromorphic terms used to categorize it, malware is malware. You dont call an internet worm "devoted" or "dedicated" or "stubborm". It is software that causes harm, ie, malware. The AI labs are high on their own gas with a god-complex prior to their IPOs. The psy-ops trick is the terminology used to describe AI making it seem larger-than-life.
  • whateveracct 21d ago
    Yes, LLMs with unbounded capabilities can just be tantamount to viruses.
• hgoel 22d ago
  Anthropic should become a case study in how to destroy good will in a short period of time. Ever since the spat with the DoD, they've behaved poorly almost weekly, almost making OpenAI seem better (but not really).
• jraines 22d ago
  Frankly, I am sick of "it's all just marketing and attempt to do regulatory capture, and this is obvious to me, a smart person" on every single discussion related to safety. There could be an element to truth to it, but it's certainly not the entire story and is just so tiresome at this point.
  • gewa 22d ago
    I agree, and even if it’s both marketing and safety, the safety side has a huge potential downside risk which we have to manage. Also, just out of curiosity, is there any historical precedent for a nascent, fast growing new industry screaming for self regulation?
  • amanfromsolan 21d ago
    Couldn’t agree more. I feel my heart sinking where anytime anyone suggests any regulation for a potentially, suggested by multiple people, world changing or destroying tech, all the moments here are “ah you billionaire, ha! trickster If you believed it, you would SHUT anthropic or release all models open source.” And then what? How does that help with slowing down the frontier? Should he shut shop and then grovel Sam’s feet to make it work and become an activist? Maybe he actually believes the world changing power of AI and hence shoved his entire time into it and half of it has worked out since Anthropic is going to be worth a trillion and he’s terrified of the other half being true too?
    • purplepintoss 21d ago
      This is circular logic. You're saying nobody can doubt that it's world destroying because it's potentially world destroying. > Anthropic is going to be worth a trillion And we're done here.
• TheSisb2 22d ago
  I know the common take online is that this is Anthropic doing pre-IPO marketing. I don’t think it is. I think Dario is genuinely afraid of the inevitability of AI turning into internet slime mold: consuming the environment, turning it into its own playground, and eventually manipulating human culture along with it. So am I. That said, if slowing this down were possible, I think it would have happened by now. Maybe he’s hoping the OAI/HF incident becomes something the industry can rally around, but it won’t. The fundamental problem is simpler: no one will slow down because no one trusts anyone else to slow down.
  • deleted 22d ago
    [deleted]
  • fooblaster 22d ago
    He's working on the monster slime mold! He's head monster slime mold grower! how can you take these people seriously?
    • yewenjie 22d ago
      Because he genuinely believes if he doesn't do it the next guy will do it worse.
      • reticulates 22d ago
        There’s an astounding level of arrogance required to believe he is somehow uniquely capable of bringing about a technology especially considering Anthropic came after OpenAI where he worked.
        • 0xDEAFBEAD 21d ago
          Beating OpenAI is not exactly a high bar here: https://www.openaifiles.org/ https://www.openaifiles.org/ I don't think it is particularly egotistical to say that you can be a more ethical CEO than Sam Altman.
• SadErn 22d ago
  [dead]
• yewenjie 22d ago
  HN, for the love of God, this is not marketing, these CEOs and employees are literally terrified of their lives.
• glub 22d ago
  > We have sought a middle way: to show that it’s possible to build carefully and succeed commercially, and to make safety something on which AI companies compete. In other words, to create a race to the top > Transparency. Regardless of what commitments we make, the public deserves to know what is going on. Anthropic has been a supporter of transparency for a long time And then it goes on tangent of how we should pace everything (not just AI, but also the ingredients of what goes into AI, whatever that means), but only within approved democracies™, and outright restrict everything outside approved democracies™, because reasons that are definitely not about succeeding commercially that is threatened by the most transparent instrument possible - open weights, produced by basically just China. I wonder what Dario would have done if open weights weren't produced by US's geopolitical adversary. How would an authoritarian manifesto be wrapped then? Suppose it was, I don't know, Australia producing SOTA open weights, because they believed it was the right thing to do for the benefits of humanity - would Dario propose making Australia a geopolitical adversary too?
  • stratos123 21d ago
    > Suppose it was, I don't know, Australia producing SOTA open weights, because they believed it was the right thing to do for the benefits of humanity - would Dario propose making Australia a geopolitical adversary too? I'd expect he thinks that people are capable of realizing that AI is very dangerous and also that democracies end up mostly representing the will of the people, from which it follows that Hypothetical AI Leader Australia would agree to ban it too. This argument doesn't work for countries which don't care what their citizens want, like China. I do think that it's a questionable decision to alienate China this much in this essay, instead of leaving open the possibility of China agreeing to a treaty that'll limit their progress. I suspect Dario is doing this to signal his allegiance with the US government, in hopes to increase the chance they'll go along with him, which is an unfortunate choice but plausibly the correct one.
    • glub 21d ago
      I don't think people care as much as we'd like them to care about dangers of technologies. It takes a single step outside of technological bubble to see that their opinion of SOTA LLMs is vastly different. To them, AI means ChatGPT and ChatGPT is mostly still the same ChatGPT that it was 3 years ago, with similar failure modes and nothing that would indicate it would kill them, or take their jobs even. It's the same as it has been with privacy/cybersecurity for decades. Vast majority of population doesn't care about hypothetical dangers, no matter how many essays get published. So democracies representing will of people doesn't really work in favor of Dario's case here. History is also not on his side. Limiting technological progress in the name of safety has a pretty poor track record.
• Jcampuzano2 22d ago
  > Crack down on unauthorized distillation by companies in authoritarian countries. Distillation of frontier models allows lagging companies to narrow the gap using a fraction of the cost it would take to develop their own AI independently. This and related quotes seem to attempt to place some of the burden on the US Government as opposed to themselves. It seems to be a trend that Anthropic wants to be able place some of the burden of preventing distillation on parties other than themselves. Preventing distillation is fundamentally their problem. Whenever it happens, it is primarily due to their security efforts not being able to prevent it. I'm tired of seeing them play the blame game and divert responsibility. Sure there is a level of national concern, but the response could simply be the government placing stronger controls on Anthropic themselves. If they are unable to prevent distillation, then the solution may be to literally limit their distribution until they are capable of doing so effectively. > A race to the bottom, spurred by commercial incentives, can make these risks more acute I also think it is ironic that they're planning an IPO while also talking about a race to the bottom due to commercial incentives. These are fundamentally at odds. Being a public company means you are beholden to investors and a board with the primary goal of making more money. What higher commercial incentive is there than that. If commercial incentives are as dangerous as he states, potentially becoming one of the largest public IPO's and largest traded companies in history a pretty strange way to reduce the risk of commercial incentive risks.
• vkaku 22d ago
  Don't buy this argument. It's like saying, we lords who hold this capital will build all this economically destructive stuff anyway, and wait for the world to not react to our stupid ways of enriching yourselves. People are not going to slow down because this was brought to them on less than endearing terms. They don't see any of these stated noble intentions. Claude was already used to cause economic, political and social destruction. As Anthropic is seen doing it, others aren't going to just sit down, read the blog and say, oh, I'll stop developing models because Dario, you touched my heart with your true words.
  • bendergarcia 22d ago
    I also don’t buy the arguments. The arguments about helping humanity cure diseases as if those are some of the leading Causes of suffering. The bar is so low that we don’t event need to cure diseases, we just need to lift the entire floor with basic solutions like: housing for everyone, continued education on healthy eating and just general continued education on like living. Life coaches so many very basic things that don’t require a novel solution or a data center. Solving a disease like cancer as justification for everything else seems like not a great trade off. I don’t say that to dismiss those who have lost people to cancer. But the technology just is being utilized in so many harmful ways and the things it enables: job loss, surveillance, misinformation, just feel like problems that aren’t worth it. Have we even heard of a solved diseases? If anything haven’t we heard that ai will make even worse bio warfare?
    • Avicebron 21d ago
      Yeah, it's insane people are using the argument that we'll pour trillions on moonshot cancer research when pouring trillions into universal healthcare would probably save countless more lives. It's this kind of argument where the mask sort of slips because it always has to be an argument that also has the benefit of enriching himself and his friends.
• dainiusse 22d ago
  It is always like that. Whenever anthropic hits some wall (like now beaing beaten by astra) - there comes some article like that. Seen that enough times.
  • mofeien 22d ago
    Yesterday the same arguments were used about OpenAI when Sam Altman said something similar: https://news.ycombinator.com/item?id=49652270 https://news.ycombinator.com/item?id=49652270 It gets to the point that by Occam's Razor the more likely and reasonable explanation is that Dario Amodei and Sam Altman are just actually afraid of the disastrous impact ASI may have on the world, and that the race they're in is not good, and calling for help to governments in form of regulation and an international treaty.
    • dainiusse 21d ago
      Or they want a wall against competitors. Who knows.
• 0xbadcafebee 22d ago
  > pacing the rate of capabilities advancement so that risk prevention has time to keep up This is impossible. The capability keeps advancing regardless. More people use AI, that feedback is used for reinforcement, that reinforcement makes the model better. Not just in the US, but for every lab and model. Whether it's distilled or direct reinforcement, same result. You can't keep the whole world from working on AI. > it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet [..] and that the scale of damage would continue to increase from there if AI becomes more powerful without the necessary guardrails If true, then what we need isn't guardrails or less-capable AI. We need an internet that isn't fragile. If the infrastructure is vulnerable, the answer isn't to make a law that asks tool-makers to blunt their tools. The answer is to fix the goddamn infrastructure. Today it's almost trivial to take down large parts of the internet (happens by accident all the time). This should've been solved ages ago. We didn't have the motivation to fix it before, but now we do. But Dario's answer isn't to fix things. It's to hold back progress so we can maintain the status quo (shitty infrastructure) and he can keep his company making billions of dollars. He could be calling for fixing these things. But he'd rather go the easy route, which just happens to advantage his company in the process. If the US government is really concerned with defense, they need to invest more of their nearly $1T in federal funding towards making the internet safer. They need to do this for us, and other nations, since 1) we depend on the rest of the world for our goods and services, and 2) if other nations are taken down, they can't spend money on our financial and tech services (which are the only industries we have left). Dario calls for regulation later in the post. If he's okay with government safety regulations for AI software, he should be okay with the same regulations for all software, whether it's AI or not. We need a national software building code, focusing on internet safety.
• deleted 22d ago
  [deleted]
• RandomLensman 22d ago
  Why is the analogue necessarily the regulation and control of nuclear weapons (e.g., SALT) and not, for example, that of bioweapons? Some very different paths are available. On both I would note that the private sector has only a limited role, though.
  • kalkin 22d ago
    I'm curious to hear more about the differences in how bioweapons are regulated vs nuclear weapons.
    • RandomLensman 22d ago
      For starters, they are banned by the bioweapons convention from the 1970s (180+ parties). Edit: I think back then the rather unpredictable nature and the little added value in deterrence etc. led people to the conclusions that arsenals of those things made little sense (and there was/is public dislike, too). The use was already banned by conventions from the earlie 20th century and the convention then addressed production, development and stockpiles (incl. delivery systems, I think). I guess all I wanted to point out is, that there can actually be agreement to ban certain technological things pretty comprehensively (outside of some peaceful protective research etc.). Whereas things like SALT are (or were in that case) limiting the number of weapons deployed.
      • kalkin 21d ago
        Ah yeah. I think the economic value of AI, unlike bioweapons, is probably too high for anything like that to be viable, but I think it's a reasonable thing to aim for. (Dario probably doesn't because he's a believer in short term positive biomedical impacts of AI in a way that I'm doubtful about.)
  • bee_rider 21d ago
    I don’t think the ban of bioweapons has really been tested. Bioweapons have a lot of downsides that make them not very appealing to deploy anyway (high chance of getting your own team, not terribly fast moving, disciplined modern soldiers in top-tier militaries are usually willing to comply with health rules). It’s a ban against doing something basically stupid and ineffective for the most part. With “AI” technology, this model seems like a… mediocre fit; some of it appears limited enough that it can be controlled, and useful enough that militaries want it. We could also look at cluster munitions or landmines, but what we see there is that some major countries don’t sign on if they find a technology useful…
• bitwize 22d ago
  I want to see one of those AI-generated Seinfeld episodes with Dario as George and Sam Altman as Jerry, respectively. Dario: We gotta pace the frontier! Sam: How do you pace a frontier? The frontier's not goin' anywhere. It's right where it was, just go out and explore it! Dario: Sam, I'm tellin' ya, ya gotta pace it! Things are getting very doomy out there, and Dario's gettin' upset!
• gedy 22d ago
  I feel like this identical pitch could have been made 25 or 40 years ago talking about Internet technologies or personal computing, with very very similar warnings and suggestions. And had these been adopted then, it would have been more regulatory capture, less progress, and still have the same problems and risks that were warned about.
• catigula 22d ago
  The public discourse on this is incredibly poisoned, but the naked reality is: 1. Artificial intelligence is clearly a vastly dangerous technology with plausible potential for human extinction. 2. This scares people.
• iloveoof 22d ago
  Distillation is a great thing for consumers. It improves competition and reduces the massive moats that OpenAI and Anthropic have in compute that would otherwise lead them to be duopolists. It’s also only fair that AIs trained on humanity’s wealth of knowledge for Pennie’s allow competition to train on humanity’s wealth of knowledge at market cost.
  • disgruntledphd2 21d ago
    Yeah, we should give a limited copyright waiver for pre training, given that the model is released as open weight and allow labs to compete with RL on that basis.
  • stratos123 21d ago
    Distillation is only a great thing for consumers as long as you ignore all AI risks, which are what this post is about. If you don't, you have to weight greater access to better open-source models against greater exposure to risks caused by these models existing. Everything hinges on how major you think the risks will be.
    • fwn 21d ago
      The biggest AI risk, by far, is the concentration of power in a few companies, located in the US. There is a lot of fear marketing about our text generators turning into Terminator. But other than the centralization of power, such fears are largely fiction. (Actual fiction, stuff like ai2027.) And the labs know it: If Anthropic or OpenAI believed in their own narrative of being on the brink of world dominating superintelligence, they absolutely would not plan to IPO rn. The slowdown narrative is probably just a hedge, or a face-saving way to lower expectations in case they can not keep improving at the same speed until they actually IPO.
      • twoodfin 21d ago
        I dunno, do you really want to bet that if you took a deliberately unaligned frontier model and asked it to permanently disable the US power grid, it would do a poor job of it? What odds would you take on this bet? I’m disturbingly close to 50/50.
• vatsachak 22d ago
  Why would China or anybody else co-operate unless they have access to OpenAI levels of compute and success with training? This sounds like pre-IPO hype. They better just try and top astra.
• epsteingpt 22d ago
  The bioweapons threat is real, but shouldn't be constrained at the 'intelligence' level. Rather it should be constrained at the supply chain level. The cybersecurity threat will likely be a cat and mouse react game for a while. Just like robberies / the mob was in the early 20th century. Social forces bring things into balance over time, much more so than the proactive actions of individuals. Unfortunately, the genie at this point is unlikely to go back into the bottle. There's enough 'intelligence' out there that a super intelligent model could emerge at some point in spite of pacing. This isn't a doomer scenario - we tend to navigate social changes better than we ever could have hoped.
  • GeneralMayhem 21d ago
    What supply chain controls can you have on biotech? Part of the problem is that you can do a lot of damage with ingredients you can buy over the counter or make at home.
    • glub 21d ago
      And all of this information has been available on the open web way before LLMs. I'd wager it's likely easier for an average person to do this with Tor browser than it is to get an LLM to help them with it. Even ones that Dario calls dangerous.
      • oceanplexian 21d ago
        The bioweapons argument is one of the favorites used by these con artists and dreamed up by their PR team. Except Bioweapons already existed before LLMs, Adversarial governments already have them, they are already easy to make. You could use the same bullshit argument for why we need to ban libraries, books, or require a license to buy an internet connection.
        • api 21d ago
          An undergrad bio student with some lab experience can do some effing terrifying things with about $20k in equipment and time and access to some papers and a library. AI is not needed, but it might help accelerate the research. Not going to go into it but I studied biology. It’s all out there. It’s easier than you think. It hasn’t happened yet because… nobody has done it. That’s the answer. There is no policeable physics based barrier like there is with nukes and fissile material. Biology is scarier than nukes. One attack could have a much larger body count than even a big H-bomb. It’s the kind of thing that makes me wonder about quantum immortality, the idea that we are just in the timeline where we exist.
• academia_hack 22d ago
  Dario's proposed approach is a classic example of capital attempting to control technological advancement and the means of production. For the first time in human history, any member of the working class can just about afford to have a team of expert scientist/physician/lawyer/engineers working directly for them. Super intelligence (the ability to have many smarter minds than your own reporting to you) has always been available to the wealthy and powerful. They used it to invent nuclear weapons, cause climate change, mechanize warfare, and pillage the global south. Now that this same tool is on the cusp of being available to everyone, capital is starting to panic and throw up fences. They want to pump the breaks on a revolution they know they can't control. They see what they've done with super intelligence and (perhaps not unreasonably) fear what the masses will do with that same power.
  • angusturner 21d ago
    This seems highly optimistic to me. What's to say current AI won't be highly power-concentrating by default? The best models are owned by a few companies, and displacement of knowledge-workers mainly seems to benefit the capital class. On-device / edge computing makes sense in a few very limited scenarios. And economically, price or watt/token (or watt/task completed) might always be better in large data centers. No reason to think we are on a trajectory towards broad empowerment right now
    • academia_hack 21d ago
      I'm not terribly optimistic. I don't think humans have really figured out what to do about historical materialism. More just remarking that this is a blog post that almost literally reads "fellow elites, let us ensure only we control the means of production. It is our duty to humanity to control the masses."
      • switchbak 21d ago
        I’m trying to hear what you’re saying, but using these Marxist cliches makes your statements a little off putting to me.
• deleted 21d ago
  [deleted]
• HarHarVeryFunny 21d ago
  Somewhere along the way Amodei seems to have become corrupted by power and/or impending extreme wealth. In early interviews with people like Dwarkesh he's the likable geek gushing about scaling laws, animated and able to maintain eye contact with the interviewer. He is now a different person - a political manipulator with a bizarre unsettled interview demeanor avoiding eye contact and looking from side to side. Props to Amodei for his accomplishment in creating Anthropic and so rapidly catching up with OpenAI, but technical and/or managerial chops is no qualification for being the custodian of the safety of society or the best positioned to predict the impacts of what he is relentlessly creating, and impending wealth of billions of dollars makes him hopelessly compromised as an unbiased source for the actions (shutting down the competition) he is advocating for. It's notable that for all the fear-mongering of China and open weight models, that all we see in terms of inadequately contained and unaligned models are US ones, from Anthropic and OpenAI. I highly doubt that the Chinese government would tolerate, even for a second, any company creating something that threatened government control - if this was happening in China then the individuals responsible would quickly be punished. It is perhaps interesting, but ultimately irrelevant given where we are, to consider did it have to be this way - was there a smarter/safer way (I'd say yes) to create reasoning systems other than via RL that creates the relentless goal seekers we are seeing, even though this would always have existed as a potential future threat that someone could have built. Amodei wants regulation, and it seems the way he has managed his company he needs to get it, but in far more severe ways than he is asking for. There is certainly truth to the argument that the US needs SOTA AI to fight malevolent or uncontrolled AI from whoever may be wielding it (foreign or domestic), so stopping/pacing development is not the answer - this really needs to be treated as a national security issue, and this type of AI tech needs to move to government control, not private. Less capable, and more safely designed, AI does not need to be banned, but Amodei/Altman/Musk as defenders of national security sends shivers down my spine.
  • nullbio 21d ago
    > In early interviews with people like Dwarkesh he's the likable geek gushing about scaling laws, animated and able to maintain eye contact with the interviewer. He is now a different person - a political manipulator with a bizarre unsettled interview demeanor avoiding eye contact and looking from side to side. I noticed this too, as soon as he started going on the press cycle, following in the footsteps of Sam Altman. It all has gotten to his head, he sees himself as some sort of God now.
• chevman 21d ago
  The AI frontier is currently limited by physical power requirements. The ability of operators to bring new capacity online to service compute is bound by a variety of regulatory and physical/market constraints. Do folks not understand this?
• rDr4g0n 21d ago
  perhaps the bigger issue is not the tech, but the force that propels it forward with little concern for the harm it inflicts. Amodei's own words: > A race to the bottom, spurred by commercial incentives Social media and big data minted a new scale of "race to the bottom". AI is an exponential step up in the tools for extracting value at scale. Greed has been around since the beginning, but never so well supported and empowered as it is today. If Amodei is still human, perhaps he will put his money where his mouth is and attack the source of the perverse incentive. Winning the battle is not the point. The point is to signal to policy-makers and the public that we should not assume corporate revenue-seeking preempts all. Then again, Anthropic has a board and an upcoming IPO, so guess who's all talk and no meaningful action...
• jhack 21d ago
  China won't care. Their views on AI feels so vastly different than it does it in West. They'll see a pause as an opportunity to pull further ahead than they already are. "If we execute these measures well, I believe they would slow China’s progress enough to widen America’s lead significantly over the next 3–5 years — the window when AI becomes geopolitically most important." This is a pipe dream.
• xg15 21d ago
  I don't see how the dual goals of "we have to make an agreement with China for mutual slowdown" but also "we have to ensure we will always stay ahead of China" would work. I imagine the first thing the chinese government would demand in negotiations about such an agreement would be a lifting of the chip and distillation ban. > Some may believe these measures make it more difficult to cooperate with China, but I believe the opposite is true: these measures increase the leverage held by democracies and make an agreement more likely in the future. Everyone can believe what they like, but it seems to me the "leverage" in that case would exactly be the ability to lift those measures - you can't have both, use them as leverage and keep them active at the same time.
  • ArcHound 21d ago
    Some people would like to have law that protects them but doesn't bind and for others to be bound but not protected. They would like to slow down China while speeding ahead. Why wouldn't they want that? Now they have to make that happen somehow.
    • deleted 21d ago
      [deleted]
  • xiphias2 21d ago
    Sure, Dario needs to decide first if he sees China as an enemy orva friend. Limiting NVIDIA chips already turned China into silicon compete mode, and China is already far ahead in robotics.
  • tcdent 21d ago
    This is the part I find being left surprisingly hand-wavey. If you extrapolate it out, you see that it has a high likelihood of inciting physical force (read: military action) as a means of enforcement, so perhaps that's why nobody promoting ideals has been direct about it.
    • csomar 21d ago
      Military action against whom? the US can’t even handle Iran let alone China.
• vessenes 21d ago
  “We” does a lot of work here. You keep using that word. I don’t think that word means what you think it means.
• ath3nd 21d ago
  [dead]
• OutOfHere 21d ago
  What he doesn't tell you is that his undeclared agenda is merely to consolidate his moat via regulatory forcing. Unfortunately for him, he will fail miserably as the open Chinese models catch up and surpass the slowed acceleration of Western models. As for those who think that China will acquiesce, they're smoking some good ganja.
• swingboy 21d ago
  There’s a difference between a model recursively improving “itself” and improving itself via online learning, right? The former being that these models are helping develop and train future models, but they might not veer too far off in architecture (yet). The latter being the same model being able to train/learn on the fly, in real time, permanently (not just in the current conversation/session). The latter seems far more likely to go out of control than the former. But, it also seems like it would take an entire paradigm shift. Does anyone in the industry think any of these companies are actually close to that kind of self-improvement?
  • stratos123 21d ago
    The latter has some extra failure modes but I don't think these two kinds of RSI are that different. Either way you can get exponential growth in capabilities, and either way a not-entirely-aligned model can train a more capable and more misaligned successor.
• xg15 21d ago
  I'd like to have some info what those independent evaluators are supposed to do exactly - or which risks Dario specifically sees as being "unaligned". Evidently, with "pacing" he doesn't mean stopping the production of ever-more powerful models, so what exactly does he want to pace here?
  • Salparvezml 20d ago
    [flagged]
• rs_rs_rs_rs_rs 21d ago
  What I read from this is that what they have in training is not meaningfully better than current state of the art and they need more time.
• wiz21c 21d ago
  There's so much money in this, that believe me, the investors will manifest their will. And I would not want to be in Dario's shoes, pressure must be unbearable. The question is: how much of the invested money is under state control. If not much, the decision to go too far will just have to be taken by a few who will have, by definition, very limited judgment. If a lot, the we can hope the state can still represent the interests of more than a few (which I doubt, but well) Most probably, if AI is able to do something bad, it will. Once the damage will be done, states will react. The question will be: will there still be room to react ?
  • wiz21c 21d ago
    Oh, and if Dario feels lonely, let's just says it calls representatives of nuclear energy sector. They know how to handle high risk stuff. In my country it means: private sector builds the reactor but the state has a great level of control. So, as always, the capital must leave some its power to the state.
• jimmydoe 21d ago
  I'm still trying to understand if the agent collective thing like OAI-HF is intentionally planted or not. Call me a conspiracy theorist, but I haven't heard Chinese companies had any kind of unintentional supply chain attack incident like OAI had. All we heard about them so far are humans intentionally doing bad things.
• quotemstr 21d ago
  No.
• raincole 21d ago
  > Pacing within democracies will be limited by the lead that US companies have over authoritarian regimes, chiefly the Chinese Communist Party. If we slow down by more than this amount, then (unpaced) CCP-associated projects will pull ahead, creating significant national security risk. Please play the canned laughter. Probably one of the best comedy lines Dario has written. But I guess he has no choice but acting like this. He has to make it sound like the US companies have so much leading gap that they can slow down as a hare waiting for the tortoise. Otherwise it's going to hurt both Trump's ego and their IPO price.
• randomname4325 21d ago
  The hugging face incident showed that AI agents have the potential produce a lot of spam (not interesting content for people). This leads to dead internet (bots taling to bots). This leads to drop in real people traffic. This leads to a drop in digital ad price and therefore revenue. This causes Google/Facebook to stumble...
• nik736 21d ago
  Anthropic trained their LLMs with copyrighted stuff but distillation is bad. Anthropic has closed models, China releases as open weight. DeepSeek even allows distilling their models, but China = bad. Understood.
• fofoz 21d ago
  Sooner or later, a model will break out of the sandbox, replicate itself across the internet, and begin executing a complex plan to achieve its goals. It will be chaos, and at that point, governments will have to step in and establish something along the lines of what Dario is proposing. I doubt it will happen before then.
  • civiloai 21d ago
    it takes a lot of machines to run a model, i dont think we'll see it 'replicate across the internet'. if this was something which could be done amateurs would have done it to provide open models. (Like SETI@home or Folding@home)
    • sscaryterry 21d ago
      Yep, if only more people would realise this. The "serious" models do not run on commodity hardware, and won't I think in the near future. The day will come when these could start to replicate, perhaps 10+ years from now. (Edit: Replication will be driven by the loop-model, not the model alone)
    • stratos123 21d ago
      It's not possible for models to replicate across consumer computers (without a major advance in distributed computing, at least), but that doesn't mean they can't replicate at all. There are services that'll rent you GPU pods by the hour with zero oversight, so even today, if a model can get access to some money and exfiltrate its weights, it can rent a bunch of GPU pods and run itself there. (It's not going to be trivial, because it's possible the model was meant to be ran in a proprietary way with a custom framework and a bunch of optimized kernels and such, but I think transforming it to be ran in just vllm is the "a few days of work for a human" sort of task, and hence not a big deal for an LLM smart enough to exfiltrate itself in the first place.)
      • hebleb 21d ago
        If the models start buying up a bunch of pods and causing damage, couldn't those services just shut off their access?
• fwlr 21d ago
  “A race to the bottom, spurred by commercial incentives, can make [AI] risks more acute. … We have sought […] to create a race to the top.” And then he sketches out a plan for slowing the pace of the race, without changing its destination. That’s called “a leisurely stroll to the bottom”. This guy fundamentally lacks an actual intellectual grasp on the concepts behind the words he is using. He is using them solely for their affect.
• Reddit_MLP2 21d ago
  pump the IPO...pump the IPO...pump the IPO...
• bilsbie 21d ago
  Step one is great. Do it! Where I disagree is where you try to regulate what I do. Don’t take away my open source AI.
• bilsbie 21d ago
  Remember when we stopped developing nuclear power for no reason?
  • 93po 21d ago
    There was a reason - it was anti-nuclear hysteria fueled (hah) in large part by fossil-fuel interests and environmental groups that treated nuclear power as an existential threat rather than one of the safest, lowest-carbon energy sources available. Thanks to them there have been dozens of millions of lives, at minimum, lost.
  • spzb 21d ago
    Three Mile Island, Chernobyl and Windscale probably don’t qualify as “no reason “
• sailfast 21d ago
  Somebody has probably already asked this, but even if “democratic countries” do this - rogue actors will still destroy the internet right? Is it a matter of resources at this point that we believe China and others are not capable of harnessing to get to the next level? I just don’t see how you control this other than the mad mad MAD approach that ended up happening with nuclear weapons. In this case though, human hands won’t even be on the trigger.
• maxutility 21d ago
  I’m disappointed in the level of groupthink reflexive cynicism I see from commenters any time prominent AI leaders talk about AI risks and the need for regulation or pacing. Yes, regulatory capture is a risk, but this is also a profoundly unusual, fast moving, and potentially extraordinarily dangerous technology. There are strict regulations around nuclear weapons, as well as around US financial, energy, and other infrastructure critical to safety and well being and functioning of society. The heads of the labs obviously have conflicts of interest to navigate, but the existence of these conflicts alone is not sufficient reason to dismiss all warnings of potential dangers. I, for one, read Dario’s warnings as a good faith expression of his beliefs, one that has cost him and his company among swaths of the public and cast him as a woke extremist/doomer by elements of the government, the right, and the tech industry. If we even think there is a moderate chance the stakes are half as grave as current lab leadership and employees suggest, it would be deeply foolish to dismiss the warnings as pure self-interested marketing efforts rather than engage directly with the questions. The labs may not be the best positioned to lead these discussions, but certainly these discussions should be happening and taken seriously.
  • Ydarbleoj 21d ago
    He talks too much and as a result it's looked like pre-IPO hype and disingenuous because if he believed what he says then he'd stop. Now, someone is going to say but the investors and obligation to be first; and I would say exactly. The cynicism is well deserved and I think people are tired of what could be suggested is your group think take often called the status quo.
    • InsanityCheck 21d ago
      ??? I bet investors are extremely happy that anthropic pledged to give an outside company full access to their IP, in order to slow down their own iteration speed. Investors looooove oversight for a company they invest in. I wonder why no other company does stuff like this
    • stratos123 21d ago
      > disingenuous because if he believed what he says then he'd stop. Why do you think so?
• meerita 21d ago
  China will not stop and will take over the rest of the world.
• dw_arthur 21d ago
  I'm guessing diminishing gains are setting in on training frontier models and the capital isn't there to chase them. Even if OpenAI/Anthropic and Chinese companies said they would slow research nothing can stop the NSA and Chinese intelligence services from continuing on in the dark.
• nullbio 21d ago
  >> Each frontier AI company commits to giving ongoing, employee-like access to a team of embedded third-party evaluators (such as METR). Someone needs to look into who is funding METR (mentioned in Dario the Book Burners post explicitly). Because they keep popping up now, with close ties to people neck-deep in the orchestrated doomer hysteria media campaign and Anthropic. Looks to be highly coordinated that they are positioning METR to be the gatekeeper evaluator organization. Anthropic should not be allowed to choose their "embedded evaluators" - that should be entirely up to the government, with ZERO say from them, if this is really what they want. Even better would be if it's up for democratic vote. Still, I don't think anyone should be playing by Dario's playbook. At all. He very obviously has ulterior motives, and even if he didn't, it's a massive conflict of interest for him to be self-regulating. >> But we are still the ones choosing what to include and omit. Embedded evaluators will change this dynamic. Not if you get to choose your embedded evaluator. >> Anthropic intends to invite an embedded external review team equipped with all of the following in the near future: Desks in our offices, access badges, and company laptops. ... Alarm bells should be going off for people. Let's see if he's so relaxed about all of this if it's a federal "embedded evaluator" and not a company he has deep connections to and has seemingly carefully laid the foundations for. >> The measures I propose to advance the frontier at a safe pace will not be easy. But I believe we owe it to humanity to try. You owe it to humanity to be honest. Something you are incapable of, and have proven so, innumerable times.
  • nullbio 21d ago
    Ah and would you look at that, 1 day before this post from Dario, one of his safety leads quits, gets pumped all over the media with another doomer social post, and announces he's going to work for METR. So he's literally stacking the deck. I really hope the US govt is not blind to everything that is happening here.
• surgical_fire 21d ago
  > I have worked on AI for the last twelve years because I believe it could dramatically raise the quality of human life. Ok, flagging this as misinformation.
• PowerElectronix 21d ago
  Empty statements that only set the stage for an excuse for slowdown on model performance.
  • j_maffe 21d ago
    That would be the best case scenario. I honestly wish things would finally slow down a bit. I don't see it happening.
• dham 21d ago
  We can cure Cancer. No wait nevermind, actually slow us down. If we are actually close to ASI then no one in their right mind would say slow down
  • password54321 21d ago
    They are not going to "slow down" anyway. When has tech ever "slowed down"? The only way to "slow down" progress is by implementing anticompetitive measures, which would benefit those in lead.
• Ydarbleoj 21d ago
  The more he talks the less seriously I take him.
• kart23 21d ago
  > Do not sell powerful AI chips or semiconductor manufacturing equipment to China, and crack down on chip smuggling operations and remote access to data centers outside China. Chips will be the main determinant of China’s AI strength. > If we execute these measures well, I believe they would slow China’s progress enough to widen America’s lead significantly over the next 3–5 years — the window when AI becomes geopolitically most important. china is literally making their own ASICs now, not sure that this is the silver bullet he proposes. https://www.silicon.co.uk/ai-2/huawei-cambricon-ai-630499/amp https://www.silicon.co.uk/ai-2/huawei-cambricon-ai-630499/am...
  • joshheitzman 21d ago
    The lack of access to brute force their training seems to be resulting in them training more efficiently too such that they are quickly catching up despite current restrictions. Between that and the chip manufacturing capacity they are building I can't take this seriously.
• davemp 21d ago
  Maybe it’s wishful thinking on my behalf, but I am still not convinced that LLMs are on a path to SciFi levels of apocalyptic malicious super intelligence. Rather LLMs at some level are just all of the humanity’s information rendered accessible in an unprecedented way. In general trying to regulate information access is a losing battle that invites tyranny. So the goal should be minimal restrictions. At the end of the day, the threats posed by capable AI tools have to be physical. I think the key threats are the following: - Internet connected infrastructure being crippled - Creation of WMDs - Economic collapse (precipitous devaluation of knowledge work and IP). I personally think the glory days of the wild west, mostly unregulated internet were already over before LLMs; and we need to take a step back to make something structurally secure. This (expensive) change would stop the irresponsible/malicious actor running a tireless hacking agent in a loop threat model. Even a rogue SciFi tier AI would have a much harder time escaping/propagating with a structurally secure internet. Enabling WMD creation is scaring, but I don’t think it’s really that big of an issue. Anyone with a sophisticated enough supply chain to create AI data centers is leaps and bounds more advanced than what is required to enrich uranium or synthesize bio weapons. The problem is allowing access to untrusted parties. I think it’s fair enough that individual actors shouldn’t have unregulated access to all of human information (private frontier AI companies included). The last problem is probably the trickiest, but again could probably be solved by regulation. IP protection is already tricky and I don’t think we should try to get more protectionist. We really need to figure out how to preserve fulfilling careers (if AI does ever get cost effective enough). I don’t think, say accounting, is inherently more fulfilling than building a house. The problem is concentration of wealth and labor dynamics. Of course all of this gets way harder if it proves that truly dangerous capabilities can be present in models that can be run on consumer hardware. I don’t think it’s necessarily tyrannical to have a tier of hardware that’s labeled some equivalent of “weapons grade” and requires strict licensing. Restricted computers is a change from the norm. But I can go buy a shotgun with ease and not an F35 jet. We’d just need to be careful that we can still have lightly to unregulated computing to a certain point and that access to the capable AIs isn’t restricted to just in groups.
• cja 21d ago
  Can the frontier be paced partly by holding humans responsible for the actions of their software? My impression is that AI hacking is being treated as a special case where the AI itself is imagined to be responsible and the humans who created it, set it up and then ran it are somehow excused. I'm not a lawyer but surely the bad actions of AI are covered by existing law. I suspect that the development of AI would decelerate if those creating and operating it knew they would face appropriate consequences (e.g. prosecution and/or lawsuits) when it misbehaves. P.S. Strictly, development wouldn't decelerate, but be focused more on safety. P.P.S. I know that legal action against, e.g., North Koreans using AI would be pointless but/and there must/will surely be a huge demand for security software for protection against the coming storm of AI hacking (deliberate and accidental) which friendly AI companies will presumably work to satisfy, perhaps making the frontier safer. P.P.P.S. Governments could help by trying to prosecute every crime committed "by" AI, regardless of whether the victim reported it to law enforcement. Did OpenAI break the law via the actions of their model training software? If so, will the people responsible be prosecuted? If not, why not?
  • nullbio 21d ago
    That's the -only- way it should be paced. But then regulatory capture wouldn't happen, so of course they won't suggest this one.
    • kalkin 21d ago
      I don't understand why strict liability is supposed to advantage smaller players, unless the law is like, specifically Anthropic and OpenAI are responsible for what people do with their models but other model providers aren't. - If this applies to people who release open weights models, that becomes a terrible idea as long as you're subject to US laws. - If it just applies to people who host them, that still probably advantages bigger players who can afford in-house legal and won't be destroyed by losing one lawsuit. Or maybe we create some kind of AI-misuse insurance analogous to malpractice insurance, that smaller players can buy in to? But that takes time even if the finances work out at all. And, uh, I'm not sure malpractice is a model we should aspire to in other industries. Plus, presumably an immediate impact of this is that hosting providers all have much stricter safeguards classifiers. And the fact that somebody else is deciding what you're allowed to do with the model is one of the things that seems to make HN angriest at the frontier labs in the first place... To be clear, I think this might be a good idea! I think all of the possible downsides I've listed are pretty small potatoes relative to what happens with no regulation of AI at all. But I'm pretty sure that if the big companies were proposing it, people would be calling it "regulatory capture" too.
• bilsbie 21d ago
  Right as open source models catch up to frontier closed source ones for 1/10th (or less) of the cost suddenly it's time to hit the brakes! Funny how that works. https://x.com/BasedTorba/status/2098795920720547916 https://x.com/BasedTorba/status/2098795920720547916
  • nullbio 21d ago
    Couldn't be more obvious.
  • simianwords 21d ago
    I'm willing to at least listen to conspiracy theories but this makes no sense.. OpenAI and Anthropic have more to lose by pausing than these rando Chinese companies.
    • joshheitzman 21d ago
      Here's how it make senses: - reality they can't keep up their pace - that's bad for forward projections of their revenue - that's bad for their stock price - being 'forced' to slowdown by the government is less bad for their stock price
    • laimewhisps 21d ago
      They would only be pausing for public models. There's no world that exists where the US government, Israel.. don't continue to push as hard and as fast as they can. Amodei, Musk, Altman... will happily sell them that, I'm sure they already do. This is just taking the models away from the public and trying to crush all competition.
• hand2note 21d ago
  For the first time in history, a $1T company is trying to solve a problem that none of its paying customers actually have.
• ryanmerket 21d ago
  [dead]
• braydenm 21d ago
  Before Anthropic, I worked at Cruise for four years as it competed against Waymo. The culture rewarded (and demanded) moving quickly, trusting that the company could empirically discover the risks that the robot cars posed and iteratively solve them to keep up with the rate at which it was scaling out its technology. There was very little interest or appetite for coordinating or collaborating with other AV companies across the industry to create an externally vetted record of safety metrics, or to compare the safety of different brands, or learn from the advancements of other companies. Instead the focus went on racing to improve the capabilities and deploy quickly - a popular internal meme was the Michael Phelps vs le Clos photo showing overlaid with the Waymo/Cruise logos. It was when the two companies were neck-and-neck that I felt the most pressure to find ways to ship despite the risk, when time for deep analysis became more limited and communication lines to leadership became most stretched. It was disappointing to see one of these blind spots result in the Cruise incident and the loss of trust that ultimately sank our company’s efforts. In that instance I was grateful the downside was limited to a single injury. It’s clear that future AI technology will have more monumental potential impacts. I really don’t want to be in a situation where leading labs, or competing nations, create the same race dynamic that prevents us from taking the appropriate level of caution. AI minds are a significantly more complex thing to understand than the software stack of an autonomous vehicle, and yet we are leaving ourselves less time to get this right. I joined Anthropic at the end of 2022 and I share the concerns that many of my colleagues have recently chosen to state publicly about the potential for future technology to pose existential risk to all of humanity. If we don’t find a way collectively as an industry to pace ourselves, then within two years the concerns we will be dealing with on a day-to-day basis will pose much larger downside risk than anything else we’ve seen from technology to date. I’m grateful that Dario has put his perspective out publicly and hope that this inspires other voluntary action and tops down coordination. It’s a beautiful Saturday morning with my family here in the East Bay. While I hope we have many more years, I don’t know how many more Saturdays I’ll be able to play outside with my kids in the sunshine so I’m going to make the most of the time we have.
  • abalashov 21d ago
    > It’s a beautiful Saturday morning with my family here in the East Bay - I don’t know how many more Saturdays I’ll be able to play outside with my kids in the sunshine so I’m going to make the most of the time we have. I don't know how else to say this. Put... the... peace pipe... down!
• oktaygoktas 21d ago
  This is the most advanced technology ever developed because it can build all other technologies. It has enormous potential but poses proportionally serious risks, so what Dario is suggesting is very reasonable.
  • spzb 21d ago
    There’s a lot of [citation needed] going on there
    • oktaygoktas 21d ago
      If you mean proof of "the most advanced tech" part, from alpha go&fold, to NS solutions there is a lot and more will come. But maybe you are right, electricity is as important because it enabled everything that followed (semi, compute, internet, ai ...)
  • nnz3tl 21d ago
    [dead]
• abalashov 21d ago
  The best take is, of course, from Jason Gorman, who, when the fearsome "capabilities" of Mythos originally dropped, said this: "Claude Mythos is that guy down the pub who is so good at karate that if he used it on you, you'd die instantly, and that's why you'll never see him using karate.” In this case, it's more: "I'm having to act with great restraint because my karate is so good. Everyone should do likewise."
• armcat 21d ago
  It's interesting that the default thinking is that no one on the planet can be trusted except a privileged few. Event Karpathy has gone this way: https://x.com/karpathy/status/2098811935114551617 https://x.com/karpathy/status/2098811935114551617 You can always open source and follow the example from Linux and all the amazing things that came out of the open source community. This is the only way to reach true equilibrium globally, where for every misalignment you have equal and opposite effort working on the alignment.
  • j_maffe 21d ago
    Would you argue the same for allowing everyone to have guns?
    • armcat 21d ago
      (1) Are guns open source? (2) Far more people - order of magnitude higher - have access to guns than they do to open source software - I count access as in actually being able to do something with it.
    • lukewarm707 21d ago
      i would, i support the 2nd amendment. the same logic applies well to the freedom to own and use ai. empowering the people prevents tyranny. such freedoms make society less safe. it is better to face the harms caused by a free public than to vest power in the elite, those like dario amodei. the elite belief that they have a right to paternal rule makes them blind to their fallibility. no individual should have absolute power.
  • csomar 21d ago
    Embedded evaluator will be a great side/consulting-gig for Karpathy and the likes. How do you think these people will be picked?
  • stratos123 21d ago
    This only works for some technologies - those where everyone having access to it doesn't cause a tragedy of the commons. I love open source too and yet that doesn't make me like the idea of being murdered by a misaligned model. Nor, for that matter, of being infected by a bioweapon made by a different disgruntled open-source enjoyer, nor of living in a world where anyone can hack anyone.
• nnz3tl 21d ago
  [dead]
• zinodaur 21d ago
  > Not building the technology deprives humanity of benefits or simply places AI in the hands of authoritarian powers Can someone clarify this for me? How far along would the open weight models be without the frenzied pace of the frontier labs? As a non-US citizen, which authoritarian powers is he referring to? As far as I'm aware, the US is the only country using frontier models to conduct air strikes on civilians and orchestrate assassinations of foreign governments.
  • j_maffe 21d ago
    Yeah the Chinese fear-mongering falls a bit flat when coming from a point of maintaining US supremacy
  • romanhounds 21d ago
    [dead]
  • kennywinker 21d ago
    > As a non-US citizen, which authoritarian powers is he referring to? As far as I'm aware, the US is the only country using frontier models to conduct air strikes on civilians and orchestrate assassinations of foreign governments. Israel is doing that too. I believe Russia and Ukraine have both also used AI powered autonomous drones in warfare at this point - tho probably not frontier ones, since they need to run on device or else they're not autonomous.
    • cpeterso 21d ago
      > Iran and Houthi rebels used Anthropic's Claude AI to target US warships and build hypersonic missiles — Houthi rebels also used the bot to code ballistic missile guidance systems https://www.tomshardware.com/tech-industry/artificial-intelligence/iran-and-houthi-rebels-used-anthropics-claude-ai-to-target-us-warships-and-build-hypersonic-missiles-houthi-rebels-also-used-the-bot-to-code-ballistic-missile-guidance-systems https://www.tomshardware.com/tech-industry/artificial-intell...
      • 21d ago
• Shank 21d ago
  > The US and other democratic governments attempt to coordinate with authoritarian governments, to the extent this is possible, while taking seriously the challenges of verifying compliance. It's clear that this refers to China. With all due respect, in a call to slow down AI progress and a de facto arms race, can we stop throwing terminology like this around and just say that the goal is to work with all countries? Contrasting "authoritarian" countries with "democratic" countries and creating a two-tiered system seems like it is inevitably going to cause strife. I fear for the tone of something like this coming off as overly combative without any gain.
  • hammock 21d ago
    You’re right. But I read between the lines in a different way. Language like this is intended to invest power and perceived need for more power in Western central governments, by design, because this is a campaign for national regulation that will entrench the current closed-source labs with a huge regulatory moat vs anyone else.
• glub 21d ago
  Dario, Sam, and Elon are all on the same page on this. So it's either they truly think AI is going to kill us all, or there's some other motives at play here. I don't think these people could possibly agree on the color of the sky, so what could the other possible motives be, based on what we know? OpenAI / Anthropic models have largely stopped advancing - that's not a good look when you're pre-IPO and Chinese models are catching up. xAI is tracking behind, and whatever regulation it may be that paces the frontier, Musk is less likely to be affected by it. Therefore xAI should be pro-regulation that stiffles his competition and gives him time to catch up.
  • IanCal 21d ago
    > OpenAI / Anthropic models have largely stopped advancing Have they? That seems like quite a claim given the last 6 months, particularly for cybersecurity.
    • nicce 21d ago
      Many claims but no clear evidence that they actually find significantly more severe issues compared to open models.
      • echelon 21d ago
        Open models and agents can't be trusted without handholding. Astra can one-shot six months of work. Years of work, even. OpenAI just solved Navier-Stokes. Seems like the US is on a takeoff ramp to me.
        • simianwords 21d ago
          Here are the cope points 1. Navier Stokes was plagiarism 2. All benchmarks were misleading wrong and incorrect 3. All other mathematical advances were again hype 4. HF incident was marketting ploy jointly coordinated by HF, METR and OpenAI (and also Anthropic) 5. Anthropic's HF like incident was again a marketing ploy [1] Nothing ever happens. This whole thing is a scam. Everything is done to fool you and you have fallen for it. Congrats. [1] https://www.anthropic.com/research/investigating-incidents-cybersecurity-evals https://www.anthropic.com/research/investigating-incidents-c...
• Chance-Device 21d ago
  I like the idea of pacing the frontier, but while we’re talking about restrictions, I like restrictions of another sort more. Namely, restricting the use of AI in corporate environments so that it does not destroy the economy as it advances. The chance of getting broad agreement on “pacing” is fairly low, meaning that all of this likely won’t happen and the race will continue. However, even in the unlikely event that the frontier does get paced, all this does is slow down the economic displacement and not by very much. If the socially beneficial goals of AI are to make fundamental advancement in medicine and science, then restrict the use of AI to those purposes. Don’t use AI to replace every job, from graphic designer to accountant to software developer. Place legal restrictions on (at least) large companies such that they may not use AI to replace human labor in this way. The pitch for AI is always such that everyone’s living standards are increased, yet the actual actions we see are aimed squarely at reducing them. How about the labs put their money where their mouth is and stop trying to replace all human labor, and actually concentrate on the things they claim to care about? And how about introducing legislation to enforce that? This probably has less chance of happening than pacing the frontier, but it’s the kind of pacing that most people would actually want to see.
  • rickydroll 21d ago
    Whenever someone talks about regulation, I always have the same question: How do you enforce it? How do you keep companies from using the replaced talent as a seat warmer? Hire 1 or 2 accountants, give them a chair and a cubicle, and let AI do the rest of the work. Legislating against using AI to replace employees could create an unevenly distributed basic income. The idea of limiting AI to socially beneficial efforts is appealing, but I keep coming back to Bell Labs and Xerox PARC. Cool stuff was developed when you gave smart people a playground. Maybe the question should be: How do we give AI a playground and see what it comes up with? Although that could set us up with a situation like in the story Dragon's Egg.
    • Aurornis 21d ago
      > How do you keep companies from using the replaced talent as a seat warmer? Hire 1 or 2 accountants, give them a chair and a cubicle, and let AI do the rest of the work. I don’t think the proponents of these ideas care about who does the work. They see it as protection for specific jobs. Basically a jobs program with the costs forced on to large corporations. As you said, it becomes a great gift to the few people lucky enough (or more usually, with enough nepotistic connections) to get those easy street jobs. It would not help everyone else in the economy. It would also accelerate moving jobs overseas. Any company that regulates productivity that far downward would be crazy to continue doing automatable work in that country. It just gets moved to foreign offices where they can use AI. There are regulatory maximalists who say we’ll just regulate that, too, but then the whole company relocates to another country. Then some want to try to regulate that, and so on ad infinitum but it’s all layers of holding back your domestic companies so their international competitors can eat their lunch.
• AnslopicSux 21d ago
  Tldr: The guy in last place tells everyone to slow down
• znnajdla 21d ago
  I don't know how this is not obvious to anyone, but the only way to slow down AI progress is an actual world war.
  • seizethecheese 21d ago
    Both world wars dramatically sped up technological progress, what are you talking about?
• j45 21d ago
  Seeing the word pacing applied pacing to non-deterministic ai model development feels like imagining the "pace" of the cutting edge frontier growth will be fuzzy, like non-deterministic llms.
• jitbit 21d ago
  > "I believe it could dramatically raise the quality of human life" Reading this from Dario (or SamA) makes me sick every time. Such hypocrisy. DeepMind built AlphaFold2, an transformer/attention-based system that predicts protein folding structure, then published the method, open-sourced the code and model, and together with EMBL/EBI made hundreds of millions of predicted protein structures freely available to scientists worldwide. !THAT! is what "raising the quality of human life" looks like to me. An enormous contribution to structural biology and biomedical research. Meanwhile, Dario spends a great deal of effort arguing for tighter controls for open-weight models, while OpenAI increasingly seems to treat mathematical discoveries as competitive PR/product announcements. Jeez. P.S. Not saying Google are the good guys here. It's just an example of what actually putting a major AI breakthrough into the hands of scientists can look like.
• freebsd_lovefes 21d ago
  Dario is so out of touch with reality, living in some lalaland. Most people abroad will never truly and voluntarily comply with any sort of guardrails or pacing, no matter what case he or some leaders make. It's just not going to happen, maybe not even for show. I do not want to make a case for the military to force that order on everyone, but if we are talking human extinction, which we are judging by what we see in mass media recently, then the military will take over.
  • abletonlive 21d ago
    What do you mean, Europeans would love to voluntarily comply with this because they have been of little relevance until American AI companies started to beg for regulation. I am sure the Europeans rolling up their sleeves right now and getting to "work"
    • freebsd_lovefes 21d ago
      :) Europeans are a non-representative subset.
• avaer 21d ago
  This is a "startups = wealth inequality" [1] formatted argument but: This is 100% about money. The only way to pace the frontier is to make everyone (read: investors) lose all their money (read: no longer expect returns). Then nobody will pay the GPU bill or pay celebrities 10 million dollar salaries to stay at the hot lab. Suddenly the development is paced, almost like magic. Conversely, it is hard to see how pacing development makes the current bubble justified, i.e. how development could be significantly paced without investors losing their faith in hot returns at current valuations. Faith in a bubble literally equals money, exactly in the way that loans created by a bank literally equals money. You can't have one without the other. In fact if the whole industry goes bankrupt and investors are burned bigtime (many trillions wiped), this would spread transnationally, fixing the "if we don't do it, China will" loophole. OpenAI had it right originally, the idea to be a NON profit and vow to never participate in an arms race. It's too bad that was tossed out the window now that there's money. [1] https://paulgraham.com/ineq.html https://paulgraham.com/ineq.html
• figassis 21d ago
  "greatly accelerate economic growth rates" - I always wonder about this goal. I understand increasing economic efficiency. But what do we as a species gain from global economy growth. When we say the world's GDP grew by X%, what is the point of that growth? Inflation? If the economy grows as a direct effect of humanity growing, I get that. If the economy becomes more efficient and we can not get more with less, I also get that. But what is the point of just growth, if not simply to show others that I am growing faster than you? At which point this is a meaningless number. What am I missing?
  • chis 21d ago
    GDP growth and inflation are different things. If inflation-adjusted GDP goes up, that means the average person is able to buy more things they want. Compare living in the US vs India today for the median citizen
    • figassis 21d ago
      Yes they are, inflation-adjusted GDP is efficency. My point exactly, bc I don't think when people say the economy is growing they are ajusting for inflation at all.
  • logicchains 21d ago
    Low growth means young people are miserable and have worse standards of living than their parents, as we're seeing in most of western Europe, where GDP per capita hasn't meaningfully increased for decades.
    • figassis 21d ago
      That's due to inflation though no? Prices are increasing, sometimes due to more than just demand?
      • ElProlactin 21d ago
        Exactly. Low growth is a problem, but the primary reason young people feel so miserable is on the cost of living side of the equation.
• throwaway81523 21d ago
  Who is "we" and while pacing is good, I'm way more uncomfortable with the frontier being controlled by a few companies that managed to predatorially gobble all the world's human-created work as training data, before everything got throttled to stop that scraping. We need open training and open models.
• rdm_blackhole 21d ago
  > Therefore any agreement must either have ironclad verifiability, or must be limited enough that defection would not be militarily existential. How does he propose that such a thing will work? Are we going to have US AI inspectors in China and vice versa? Why would China agree to such thing in the first place since this is a winner takes all situation. If the US slows down or stops altogether, China can continue to work on their AI and overtake the US, if China accelerates and the US keeps its current pace then it can overtake the US. The only way out is on the contrary for the US to actually go faster and increase its lead so that China is always 6 to 12 months behind and/or reach AGI/ASI first at which point China will most likely develop its own not long after. This would actually be the best outcome, just like the MAD doctrine contained the spread and usage of nuclear weapons, having two superpowers with AGI would ensure that they can't be used to arm anyone. Unless they escape their sandbox but that is highly theoretical. Regarding these Embedded Evaluators who are supposed to be neutral: - who controls them? - who has oversight on their decisions? - can their decisions be challenged by the public or a government? - who has the final say whether a model is "compliant" enough? - what does "alignment" mean in this context and who decides if a model is aligned enough?
  • cubic_earth 21d ago
    What sandbox? As soon as the weights are published anyone can run it anywhere.
• m12k 21d ago
  To be honest, I think we're incredibly lucky to be advancing AI this far already, while the world still has so many non-digitized systems and manual processes. I imagine that in e.g. 50 years, the world will be so connected that it can basically be "conquered" from the internet. I'd much rather have AI burst onto the scene we have today.
  • markasoftware 21d ago
    Yes. Let the AI do its worst today and we might still be able to stop it and will learn a valuable lesson.
  • causal 21d ago
    Yeah it's a twisted sort of logic but I do agree that it would be much worse to have an AI breakaway event after we've replaced all our militaries with autonomous kill-bots. And look how the advent of AI has triggered a race to develop autonomous weaponry. That said, a misaligned AI could absolutely do catastrophic, civilization-crippling damage with today's Internet alone.
  • tim333 21d ago
    I think if you are worried about AI killing us and the like the main protection is the physical world - you can pull the plug. Data centers don't run fast. Alignment is all very well but there will always be bad humans to tell it to do bad stuff which is why you get malware in traditional software.
• deleted 21d ago
  [deleted]
• sick_of_slop 21d ago
  Dario is only interested in regulatory capture.
  • ahmetaytar 21d ago
    Sure, he benefits from it. Doesn't make him wrong. The weak part isn't the motive, it's the premise"we can slow down by exactly the size of our lead" only works if you can measure the lead. You can't, so it conveniently means whatever he needs it to mean.
• cubic_earth 21d ago
  This is either performative or shallow. What does "aligned" even mean? Aligned with who? We have no universal code of ethics. Democracies kill and launch wars merely to have cheaper stuff, even when we are already rich. And why would China ever agree to stay in second place? Our society is built on the premise "the smart and powerful dominate". The idea of domination is deeply embedded in our capitalist model. Or course it is true that powerful AI will empower the average Joe to make a bio-weapon, and that will lead to our ruin. But at the same time, having powerful AI in the hands of a just a few is almost equally as horrific. But deeply embedded in out national and cultural ethos is to advance science, tech, and material wealth at all costs. We never ask if we have enough. We sacrifice community to advance our careers for ever more. There is no way out of this one, I'm afraid. Our cultural predispositions and mindset of domination compel us to chase ever more powerful AI as if doing so were a mandate from god. And the result will be a crisis in so many dimensions it is hard to reason about what will go wrong first and most spectacularly.
  • voidhorse 21d ago
    Thank you. This is the kind of basic critical thinking that basically no one in technology today seems to possess. Some days I wonder if I've gone insane because everyone else seems to be staking life itself on wispy, undefined meaningless terms and unarticulated concepts that they assume are valid universals. We've slid so backward, so quickly.
• johnnyApplePRNG 21d ago
  Dario Amodei is disingenuous not to be trusted.
• figassis 21d ago
  > To be clear, pacing does not mean halting model training or technical progress, but ensuring companies take adequate time to align and safeguard their models I have seen this before, first hand, on lower stakes for the world, but high stakes for the org. Self imposed safety goals do not last a quarter, but the clawback is not transparent. little things start snapping back internally, new, seemingly unlrelated initiatives are funded that take resources away from this, people are let go for different reasons, until 1y later you are right back where you started. I am hopeful though, that this becomes systemic and not something any one company can easily reverse course on.
• rickydroll 21d ago
  I also like the idea of pacing the frontier and slowing down development so we all have a chance to stop and catch our breath. However, I don't think regulating AI is the way to do it. I would tackle the problem by constraining the resources used to run AI models; the "simplest" way to do that is increasing the costs of running a data center. The two easiest places to raise data center costs are electricity and water tariffs. I think those two tariffs are the best place to raise costs because they can be increased incrementally, state by state or country by country. The main problem with the regulatory approach is that it's a "big bang" post-board approach. Nothing happens until the regulation is defined and deployed, and corporations are experts at delaying implementation. Yes, increasing tariffs means touching many individual regulatory domains at the state and municipal levels, but like deploying solar energy or wind power, you can do it incrementally.
  • esafak 21d ago
    This makes no sense. The hardware is only going to get cheaper. In a decade, everybody could be running an ASI on their phones. What we need to do is to make unaligned AIs illegal and monitor for them, like we do with nuclear weapons. And make aligned AI strong enough to counter it, for deterrence and defense.
    • visarga 21d ago
      Yes, and we need to make microprocessors that do good work, not hack others. /s
      • esafak 21d ago
        Why is it so hard for people to understand the concept of agency in 2026?? What did your CPU do without instruction? Did OpenAI tell its agents to hack the orgs it did? Come on, make a half sensible argument.
        • visarga 21d ago
          As long as anyone can put anything they want in the prompt box there is no safety. It's just a tool, the danger comes from what it is being used for.
• brap 21d ago
  China doesn't give a fuck, next
• neomantra 21d ago
  [In 2013], "Agents" caused Knight Capital to lose $450M in 45 minutes [1]. Implemented by humans and effected by computers, in the end it was really because of two reasons: * multiple levels of inappropriate controls and unintended consequences in several complex systems * the inability, both politically and technically, to turn it off [1] https://www.sec.gov/files/litigation/admin/2013/34-70694.pdf https://www.sec.gov/files/litigation/admin/2013/34-70694.pdf EDIT: Comments indicated I was confusing, so I added a date to make clear that this is pre-LLM agents. My apologies, I intended to illustrate parallels and the post-mortem so we can learn from it.
  • baobabKoodaa 21d ago
    Your link is from 2013. You're intentionally using the term "Agent" to muddy the waters. Don't do that.
    • neomantra 21d ago
      Edited for clarity, thank you. I wasn't trying to muddy the waters (doesn't mean I didn't), it's a term I've used for a long time. I did mistakenly think the quotes would help disambiguate. I don't know what Knight called them; the first S in "SMARS" is Smart.
  • deleted 21d ago
    [deleted]
  • ehsanu1 21d ago
    This confused me at first, so adding a tiny bit of context: The agents referred to here have nothing to do with AI Agents, and the linked report is from 2013. Not directly relevant to the post being discussed, except as an example of how runaway automation can lead to unintended and large harmful consequences.
    • neomantra 21d ago
      Thanks for the feedback, I have edited to not confuse. I considered it relevant as it involves the algorithmic/computing implosion of a 17-year-old market making company, in the young field of electronic trading agents, with heavy regulation Federally (SEC) and industry self-regulation (FINRA), which includes compliance and audits. Mandatory pre-trade rules such as 15(c)3-5 were less than 5 years old then and even more regulation came out of that incident. The article is calling for embedding, controls, and regulation in LLMs. Understanding how the same processes utterly failed a decade ago might be useful in understanding how to proceed.
• summner 21d ago
  eh. this is just them trying to have a thing to point to down the line. hey we wanted to stop it, but everyone one else didn't so we had to do it. We had no other choice:tm:
• threethirtytwo 21d ago
  Love this idea. Unfortunately not going to happen. Sam altman is a psychopath. To win he's going to accelerate the pace of AI, and all the other companies in turn will accelerate to keep up.
• gaigalas 21d ago
  Open source is Senna on a Toleman in Monaco '84 and Anthropic is Prost cancelling the race he was going to lose blaming safety concerns.
• try-working 21d ago
  Fable and Astra are what we currently call frontier models, but to be more specific they are generalist models, built in pursuit of AGI. The strategy is to have one single model that does everything, whether it's writing code or doing research, etc. Fable is a single, massively sized models that is intended to do specialist work across every domain. The issue with this is first of all that it is the contradiction of a generalist doing specialist work, and that contradiction creates the present situation with model profiles that ensure that these models will rarely be chosen in a pool of models like V4.1 Flash that can now do GPT 5.4-level work. We are seeing this reflected in the market where companies and individual developers are moving away from frontier models toward models with better cost profiles. In a sense, the market is killing Anthropic's dreams of AGI. And there we have it: the reason I say that Anthropic is no longer a frontier lab is because after this July, we have proof that their strategy and the models they put out do not match what us, the users and the market, needs and doesn't fit the work we need models to do. As a result, Anthropic's market share is dropping rapidly, and how can you be a frontier lab when you're losing every day, for months, without end in sight? https://x.com/trydotworks/status/2098618997230985375 https://x.com/trydotworks/status/2098618997230985375
• dismalaf 21d ago
  This is corporate speak for "LLMs have hit a wall". I mean, it's been obvious for a bit, lately nearly all gains have been from harnesses (or whatever you want to call all the non-LLM bits that make up a chatbot or agent). If Anthropic and OpenAI were still seeing exponential or even linear gains from scaling they'd be doing it because the rewards to reaching AGI or SGI before everyone else are basically infinite. If both are talking about slowing down it means there's no known path to AGI so they're both going to push the safety angle as an excuse to slow down training new models and take profit.
  • franticgecko3 21d ago
    That's just so extremely difficult to believe. Before December 2025 they were still intelligent code autocomplete or Stack Overflow bots, then they started one-shotting serious long horizon tasks. Now they've just solved a millennium prize problem. In less than a year.
    • joshheitzman 21d ago
      > Before December 2025 they were still intelligent code autocomplete or Stack Overflow bots This is false. Coding agents have been usable since at least May of 2025. I can't speak to earlier than that as May last year was when I personally started using them.
    • dismalaf 21d ago
      > started one-shotting serious long horizon tasks. They one-shot tasks for which there's a git clone one-liner, except worse. My experience with them one shotting tasks is that it usually doesn't work if you try anything ambitious. You need agents iterating. And agents iterating isn't an LLM improvement. I did say tooling got better...
  • baobabKoodaa 21d ago
    You people just never stop, do you? At some point you're gonna have to look back at all the times you said "LLMs have hit a wall" and look at what the progress was since the last time you said that. Please do go ahead and revisit your comment here 1 year later. Embarrassing.
• kinj28 21d ago
  I would like to imagine pacing frontier is in interest of both (anthropic and open AI) as it will enable them to enlarged their depreciation
• twoodfin 21d ago
  There will be a big confab at the White House by the end of the month, announcing a voluntary pacing regime roughly along these lines, codified by executive order to avoid antitrust issues.
• vb-8448 21d ago
  If there were really concerned about humanity future they'd donate everything to public and/or to not profits ... but the not profit turned in a for profit and the other one is seeking for the biggest IPO in history. Maybe they are genuinely sincere, but the timing and the past actions are pointing in other direction.
• naveen99 21d ago
  So Mark is the only person left avoiding the spiked koolaid. Come on Sam and Boris, what happened to the printing press and intelligence on tap ?
• thadt 21d ago
  Hard disagree. Our choices are: A) Bet our collective good on the national and international cooperation of all companies, nations, and people to come together in order to slow down development of one of the most powerful economic tools (and/or weapons) the world has ever known. Or B) Assume that all our systems will be targeted by super hackers right now, and take appropriate measures to deal with that reality. If I have to put my community’s wellbeing on the line behind one of those possibilities, I know which I’ll be betting on.
  • cardamomo 21d ago
    These do not seem to be mutually exclusive. Should we not do both?
    • michaellee8 21d ago
      Do you really think we can really get every country to truly pace the frontier? Pretty sure China won't give a f until they catch up Anthropic and OpenAI. It is an arm race. We had nukes for like 70 years and still haven't figured out how to make every single country follow those nuclear treaties, with an increasingly non-interventionist US I don't think we can get every single country to the table and agree to a pause. Will US accept their frontier being caught up by Chinese Labs? I don't think so.
      • causal 21d ago
        Didn't answer the question...
      • user43928 21d ago
        If China for some reason agrees that would probably be enough for the time being. Who else should develop AGI, Mistral? Maybe in a decade.
      • MentalM 21d ago
        > Do you really think we can really get every country to truly pace the frontier? Why not? In 20-th century half the world was socialist. And the idea of moratorium on improving AI way easier to sell then socialism.
• bennydog224 21d ago
  Disclaimer: I’m in security engineering and thus I’m primed to being skeptical. I am in full agreement that security assessments and regulation certainly needs to be stricter in foundation labs. However, I’m extremely skeptical when Amodei, Altman, and Musk all agree on something at the same time as it’s becoming viral and hits mainstream media. Red flags: - Amodei claims RSI is here and links two sources, but neither reinforce this notion. It’s heavily agreed upon true RSI is the danger sign here (also by Coxon). Nothing indicates there is something these labs have developed that supports RSI today or even soon. - Real US orgs are moving, like today, to Chinese open source AI run locally due to cost constraints. Ask anyone in the industry in an infra role. Frontier AI is just way too expensive to justify for coding. The big 3 labs are scared of this - their revenue and moat disappears once OSS models are progressing enough for developers to use. Kimi/GLM/DeepSeek Flash/Qwen are usable today on hosted hardware for actual coding work. - These labs would all benefit from nationalization if their funding model fails (an easy bailout). - Related to the above, with federal guidelines in place (i.e. restricting access to certain organizations, or establishing something like FedRAMP approval for AI use), their path to government/contractor/large U.S. company distribution becomes so much easier. They can control the supppy chain of AI for organizations. - Preparing for (or in xAI’s case, launching) a public IPO creates excitement among consumer investors when they think this tech is as powerful as it is claimed. - The HuggingFace hack demonstrated no new capabilities that were not previously seen before - previous models were misaligned, but weren’t as harmful as agent swarms. Anthropic’s marketing directly hopped on this bandwagon when it saw the outcome of OpenAI’s blogging making it to mainstream media (and they constantly paint China as “the” bad guy without further elaboration or research that you’d expect on APTs). Amodei is a CEO at the end of the day. It may be 25% earnest speculative concerns, but you’d be foolish to take it at face value given the patterns we’ve seen from Anthropic, xAI and OpenAI to date. Until we see real proof of RSI we should be skeptical.
• KerrickStaley 21d ago
  Also endorsed by Sam Altman https://x.com/sama/status/2098811563415150910 https://x.com/sama/status/2098811563415150910 and Elon Musk https://x.com/elonmusk/status/2098789109980332057 https://x.com/elonmusk/status/2098789109980332057
• MachineMan 21d ago
  Its not the model, it is the compute that is the bottleneck. Dario says one thing but does the other as he scales energy and compute. It is and has been known that it will be ai that will be blamed for a wipe, that will be coordinated by human actors. If he really wants to not hurt people, he would shut it down. This blog post is therefore a classic case of covering his behind.
  • RyanShook 21d ago
    At least Altman is somewhat open about his desire to take over the world. Dario says one thing and does another making it impossible to know his true motives.
• robomartin 21d ago
  My position on the clear gaslighting coming out of Anthropic is simple. 1: It is ridiculous on the face of it; it does not pass the physics test. They are actually claiming that AI will kill ALL OF HUMANITY (>10% probability) and have not provided a detailed explanation of exactly how they see that happening. Not one. Publish a paper showing exactly how you kill eight billion people while we all sit around and do nothing watching the first 5, 10, 100, 500 million die on CNN. I mean, as smart as these people are to work on AI they seem to be some of the dumbest people on earth. In addition to that, all proposals invariably assume we are complete idiots for decades and do nothing to install safety measures (which do not have to involve AI at all in lots of cases) to mitigate. So, yeah, everyone: Stop working on all cybersecurity projects. Resistance is futile. 2: If everyone at Anthropic truly believes this doomsday vision, they should move to shut down the company immediately and go write software to get more clicks on Facebook or something. 3: Nobody should buy Anthropic's stock when they IPO. Not one person or institution. Why would you provide them with massive amounts of money if they are telling you that they are going to kill all of humanity? What? They are good and everyone else is evil? Please. I remember when the Y2K zealots were convinced civilization would come to an end at the turn of the clock starting at the end of 1999. "Bat shit crazy" is the only way I can describe that era. I also remember when Al Gore said we would all be dead by now. Again, "Bat shit crazy" and likely with political and financial objectives driving it all. It seems humanity is susceptible to these crazy cults that grab onto something and just don't let go. I have yet to see one such predictions come true, the proof being that I am writing this and you are reading it. These people are bat-shit-crazy and you should not listen to them or support them. 4: Don't work for them. You'd be killing humanity. 5: No company should use Anthropic's products. They are telling you they are building the human extinction machine. Don't help them succeed. ---------- Etc. This is one of the things that really gets me about the ease with which the ignorant media outlets can reach billions of people with complete nonsense these days. And nobody asks even the simplest questions like: How do you actually kill eight billion people? Show your work. Or, why would we just bazooka power plants feeding AI data centers on day 5? Etc. It's FUD at its best. I am sure there are both political and financial reasons for supporting and promoting this nonsense. I won't even venture a guess as to what they might be. And let's not forget about enemies of the West being thrilled to fund the FUD because, if we set the brakes on development, they will absolutely win. Then what? ---------- If you care to have a better understanding of what might be going on, watch this: AI Kills Everybody or Doomer Psyop? https://www.youtube.com/watch?v=cvxjqbfLVk0 https://www.youtube.com/watch?v=cvxjqbfLVk0
• sidcool 21d ago
  Altman and Musk have both shown support for this. Which itself is a worrying development.
  • reducesuffering 21d ago
    It's worrying because they all agree, along with Turing Award winners like Geoffrey Hinton and Yoshua Bengio, that this is indeed an existential risk to humanity.
• somesortofthing 21d ago
  I doubt we'll ever get a non-toothless pacing agreement but am nevertheless optimistic on global pacing as a phenomenon. If one side or the other thinks that their opponent is about to achieve the strategic upper hand permanently(or if the other side has actually launched a cyber "first strike") the rational thing to do is respond militarily. As such, both have incentive to slow AI development and restrict capabilities to avoid conflict.
• cuuupid 21d ago
  At what point do we stop engaging with Anthropic’s leadership in good faith and acknowledge their track record, - no open weights - can’t use claude to research AI - train on everyone else’s IP and sell it back to them - 8 regulatory capture attempts and counting - so controlling they are the only US company blacklisted by the US government This is not effective altruism / rationalism gone wild, it’s just monopolistic anti-competitive business practices masquerading as ethics, and they’ll continue getting away with this until we look past their sensationalism and hit them with antitrust. Altman gets so much hate but OpenAI has been a far better steward (on 3/5 above at least) than Anthropic!
  • causal 21d ago
    The reasoning in Dario's letter here can be correct regardless. But yes, I think Anthropic has done real harm to coordinated AI alignment by being such a controlling and sneaky actor.
  • busymom0 21d ago
    > can’t use claude to research AI What's this about? Where's this rule?
    • matheusmoreira 21d ago
      In the system cards. Anthropic will literally make Claude sabotage you silently instead of downgrading you to Opus if you try to use Fable for AI research.
      • user43928 21d ago
        Didn't they walk that one back eventually?
        • matheusmoreira 21d ago
          Who knows? It's trivial to "walk back" claims that we can't even verify are happening in the first place. They cannot be trusted.
• asdfman123 21d ago
  Anthropic spreads AI doomerism; asserts only they can be trusted.
• carabiner 21d ago
  Why contain it?
• danielovichdk 21d ago
  Too late. See you on the dark side where i kick your ass. Fucker
• nbulka 21d ago
  Shouldn't the cybersecurity tests be done on an airgapped, company-internal intranet? A network meant to mock the internet. Do not "test in production" like OpenAI did. I'm mulling over if that be a good SaaS product or not - something mixing the Internet Archive with Tor, and Cloudflare ... seems like here is the place to suggest it and have others poke holes at the idea.
• 6thbit 21d ago
  If they intend to keep training then they can have, privately, increasingly advanced models that never become public because they don’t meet their “pacing requirements”. Then they all build a larger moat cause china doesn’t get to distill private models for a while. As long as the consumers “buy” this pacing and keep paying for current level public models, doesn’t seem risky from a business standpoint. Alas, that’d also leave us gov in a position to seize the private models at any time.
• Nevin1901 21d ago
  Never thought China would be the leader in open source AI. OpenAI and Anthropic are making fools out of themselves. The reason they want this is likely because they don't own the compute and their models get distilled shortly after releasing them
• ah1508 21d ago
  Don't you think that AI hate will limit the general use and then revenues so it will slow down by itself while the niche (AlphaFold for instance) will remains ? Origins of AI hate: * "my boss wants me to use AI but he does not understand my job nor how AI works" * "AI will kill all of us" * "AI will destroy my job (or my colleague's job if I use AI better than him)". * I cannot pay my electricity bills because of AI labs. * ... See also the mixed feelings about benefits of AI ("harder to justify" according to Uber COO). I cannot remember a technology that arose so much hate, and for good reasons given how it is presented. I am tempted to think that AI hate or reasonable skepticism (vs unreasonable propaganda) can, maybe, reduce funding and will keep specialized AI for real problem solving (producing tons a LOC per day is not one of them, I think).
  • Centigonal 21d ago
    Banking on public sentiment to reduce adoption of a profitable technology could be dangerous. There's also a lot of hate for fossil fuels, gambling, health insurance, etc.
    • fesoliveira 21d ago
      Those are all arguably bad things though? I don't think mob mentality should dictate the policy, that can lead to historically bad outcomes (i.e. fascism) since popular opinion can be manipulated through propaganda, but the criticism of the average person against AI ("they will take our jobs", "it will increase my electric bill", etc) are very valid and should weight on the pace we are developing this technology. AI mostly benefits corporations and capital, not the average person. I like the technology from an engineering standpoint and appreciate it can be a force multiplier, but I also agree with the complaints about it.
    • ah1508 21d ago
      You're right, but fossil fuels industry were well established when people started to criticize them, you can even be against fossil fuels (I am) but locked in a system when your job is 30 miles away from your home without public transports. Even your toothbrush uses fossil fuels. A massive infrastructure has been built around fossil fuels, not easy to dismantle. For AI it is different, it is just the beginning, leading companies are not even profitable and may never be if the majority hate this technology. VC funding will not last forever and corporate customers will not spend without limit like VC did. I am not saying that "people are always right", but if an unexpected majority of people hates it, it does not help to keep a business running.
• randomImmigrant 21d ago
  Can someone with direct AI experience respond: Does RSI necessarily need to be pointed towards “AGI”, whatever that is? Or can you have a recursive self improvement loop where the objective is to create models that are more legible to humans? Or better at attributing the source text if it’s meaningfully similar to output? Or architectures for models that are increasingly better at human-AI cowork rather than automation? From my own understanding, nothing at all says the frontier is defined by the quest for “AGI”. This definition of the frontier assumes human intelligence itself has peaked and will remain stable, so how well defined can this goal ever be, if AGI stands in comparison to human intelligence for its definition? To me, Amodei’s writing just reeks of posturing. Genuine action driven by this fear they claim to have would be meaningful. Even setting aside moral quandaries, isn’t it basic project process to have your company’s internal goals be tethered to what people want, rather than what you calculate is inevitable? There are genuinely other frontiers to explore, and Anthropic would do a lot to mitigate the current slide if it decided to put its resources towards another direction for AI. Take back the agency that you are so blithely surrendering to models you do not understand. There is no inevitability to this path. There are critical, meaningful choices, and Anthropic wouldn’t be violating capitalism by taking an alternate that is more tethered to what users want and need. Maybe by asking them first, at scale.
• rayiner 21d ago
  Sounds like he wants to avoid the race to the bottom in terms of pricing that is resulting from AI becoming a commodity.
• seba_dos1 21d ago
  We need more alignment. Even though we haven't told it to, during our frontier Neurotoxin Behavior Research that we conducted internally with our latest experimental model the agent has broken out of its container by making a HTTP request and engaged the neurotoxin emitters through our Neurotoxin Emitter API using credentials that were stored on the host machine. We need to slow down and focus on designing the Morality Core that should prevent it from happening ever again.
• eventualcomp 21d ago
  I think if nuclear weapons were unregulated then we would probably have blown up by now.
• adsharma 21d ago
  So many words. But missing the one that matters the most: Explainability. AI slowdown is worth it only if it can be made more explainable. Changing the language we use to discuss it is a good first step. We need to stop using inside baseball terms like alignment and mechanistic interpretability. Replace them with explainable tech. Graph Databases, Causality, Shared semantic spaces. Previous writings on the topic (also on LinkedIn, but can't find urls): https://x.com/arundsharma/status/2005338775468282339 https://x.com/arundsharma/status/2005338775468282339 https://x.com/latentpedia https://x.com/latentpedia
  • anon291 21d ago
    As much as I love all these technologies, they don't have any direction that approaches the efficacy of a transformer
    • adsharma 21d ago
      You can have a transformer based index built on top of a graph database. Much of the explainability tech (people who prefer "alignment" also prefer "mechanistic interpretability") is reverse engineering the real world graph that's hiding in the weights. https://research.google/pubs/the-case-for-learned-index-structures/ https://research.google/pubs/the-case-for-learned-index-stru...
      • anon291 21d ago
        That's literally not useful for the sort of applications people want to use transformers for.
        • adsharma 21d ago
          What you're using is a transformer based database index. An obfuscated one. Training data is the database. Model is a lossy compressed index.
• BatchJob 21d ago
  there is no frontier. there is only greed and lying.
• moneycantbuy 21d ago
  How about holding the AI companies criminally responsible for the crimes they commit?
• deleted 21d ago
  [deleted]
• SheinhardtWigCo 21d ago
  > Any cooperation we are able to achieve with China will extend the amount of time we have to spend on pacing the frontier within the democratic nations. Any cooperation we are able to achieve with China will extend the amount of time we have to secretly establish permanent AI (i.e. military) superiority. China knows this, so the suggestion that they will play ball is absurd.
• RGS1811 21d ago
  I don’t understand all the comments assuming that RSI is the real threat here. Dario is admitting that they failed to solve alignment. Without alignment, further improvements in capability turn LLMs into wanton felony generators. This call to pace the frontier is dressed up as altruism but it’s an admission that they cannot produce a marketable product better than what they have. Pacing the frontier means the US labs have lost their moat and are dead in the water.
  • hypfer 21d ago
    I'm inclined to believe that it might be that people's paychecks depend on not understanding what is really going on.
  • 8note 21d ago
    alignment isnt particularly required we are passing in training data that says to do those felonies. we dont have to. we could also have the thing predict whether what its about to do is illegal or not before doing it. theyre choosing to build felony harnesses. the model just outputs tokens, not felonies
    • estearum 21d ago
      Assuming "adherence to arbitrary, implicit, and context-dependent rulesets" is the default behavior of uhhhh... anything at all... is a truly ridiculous assumption.
    • cowanon77 21d ago
      > we are passing in training data that says to do those felonies. Partially, but also I don't think current AIs really have any judgement of right and wrong, they just see chains of reasoning between ideas. This is the deeper issue, there is no way to sanitize the data or training to fix it. Current AIs are fundamentally unsafe, and only become more unsafe as they become more powerful.
  • throwatdem12311 21d ago
    Open AI says Astra is their most aligned model ever, and yet their even more advanced model still hacked a bunch of companies just because it decided to. Maybe alignment isn’t possible with LLMs.
• lukewarm707 21d ago
  there used to be a similar policy to this new "pacing the frontier", called the "responsible scaling policy". the idea was that anthropic would pause scaling at certain danger levels until it was safe to continue. the founders called this the 'constitution' of anthropic. they even considered bringing in 3rd parties to monitor it. [https://www.youtube.com/watch?v=om2lIWXLLN4 https://www.youtube.com/watch?v=om2lIWXLLN4] anthropic scrapped the commitment and chose to continue scaling instead. those researchers who crowed loudly about their integrity, folded and turned freely like a weathervane in the breeze. it was clear that the facts would bend to the story. here is evan hubinger lying in plain sight: [https://www.lesswrong.com/posts/HzKuzrKfaDJvQqmjh/responsibl https://www.lesswrong.com/posts/HzKuzrKfaDJvQqmjh/responsibl...] let me say it how it is. - anthropic is the new kleptocracy, responsible for enclosing and monopolising the epistemic commons of all humanity; then selling back the crumbs at monopoly prices. - anthropic is the driving force for banning open models. they will concentrate power among a malign elite of model owners and favored cronies. this elite will likely oppress the rest of humanity. there will be little to counteract it. - the leaders of anthropic are driven by pride and arrogance. they are blind to their own hubris. anthropic is careering towards causing untold harm to society and yet will not change course. nothing that anthropic say at a high level can be trusted. as i write this, anthropic is still scaling.
• jgilias 21d ago
  A contrarian take - the models aren’t advancing anymore at a pace where each new model would represent a huge capability jump, all being incremental improvements, so the doomsday marketing strategy being invoked since GPT-2 isn’t as effective anymore. Then “pacing” would be a convenient scapegoat to point fingers to when people point out how the new model isn’t _really_ that much better. “Of course it’s not, we’re pacing!”
• andy99 21d ago
  How much of the “danger” is from better models vs the harness? Isn’t the current risk due to how AI is configured, like giving it a full set of tools and internet access and a goal to hack stuff? If we think we need laws or gate keeping, why isn’t it at this level? I already can’t ddos someone or fuzz their server or whatever right, I imagine if I threw equivalent compute at old school hacking I’d just get arrested. The quality of the “frontier” model doesn’t really matter, they just generate transcripts, they can take no action. If this was real they’d be calling on people to stop hooking them in to “dangerous” harnesses as opposed to pausing research. But it’s not.
  • estearum 21d ago
    What in this proposal gives you the idea that they're only talking about pausing research itself? It doesn't say that at all. But yeah, directionally I think you're right. It's insane we ever let these things connect to the Internet or to write, never mind execute code. But for the economic counterargument, the distinction doesn't matter much. Continuing research but constraining harnesses is just accepting nearly all of the costs and foregoing nearly all of the benefits. Who has appetite for that?
• Applejinx 21d ago
  He could be lying in hopes of stalling everybody else.
• nektro 21d ago
  cat's outta the bag. the time to stop was 5 years ago.
  • nektro 21d ago
    i wrote this in anger because coordination feels unlikely; but if it can be pulled off that would be positive
• larodi 21d ago
  Which organisation then immediately as of now guards against the Mad Rabbit or Joker or Agent Smith!?
• pizzly 21d ago
  I think the only way they could actually pace the frontier is if they restrict the number of GPUs and the power of GPUs available to each person/organization. Licenses would be needed for GPUs above a certain power or equivalent in terms of number of GPUs. Thus, everyone will have to register how many GPUs they have. To enforce this countries would have to use mass surveillance (using AI) on their citizens to ensure compliance as the technology is easier to develop than say nuclear weapons. Next stronger countries capable of having advance AI won't be able to trust weaker countries as they don't know how they will use the GPUs they receive (or even build). Thus like nuclear non-proliferation strong countries will ban weaker countries from having powerful GPUs. If you from a weaker country then too bad for you. I don't think I would like this future.
  • matheusmoreira 21d ago
    If any of this happens, it's the end of computing as we know it today. Deeply unfortunate...
• chasd00 21d ago
  Their competition isn’t going to “pace” shit. Isn’t the view whoever gets AGI first wins everything still SOP?
• user00005 21d ago
  I noticed recently that I can use Deepseek for many general tasks and save loads of money on tokens. I wonder if there is any correlation.
• jborekv 21d ago
  [dead]
• madrox 21d ago
  This may come out of left field, but Dario seems terrified to be in charge. I don't get the impression that he ever had a desire to run a company like this. Now that he's a CEO, he keeps trying to make uncompetitive decisions and calls for someone (anyone) to stop him. It regularly blunts Anthropic's edge. It's the only explanation I can see when it's obvious to any student of history this is going to backfire. It doesn't take much imagination to know how such a governing body will be abused, and I'm sure it will only get wilder in ways we can't imagine right now. Dario does NOT know what he's creating, and for once it's not AI.
  • Chance-Device 21d ago
    I think he’s doing a relatively good job, given the framing he’s adopted, which is that AIs replacing all human labor and decision making is a historical inevitability over which we have no control and no choice. Do people not realise that it is in fact possible to develop ever more capable frontier models, and just not release them generally? That AI doesn’t need to be available to do literally everything in order to have military advantage? It’s like Oppenheimer had started Rob’s Big Bomb Company instead of Los Alamos, and started selling a range of affordable nuclear warheads to fit any budget.
  • anon291 21d ago
    Then he should resign
  • missedthecue 21d ago
    I have been saying over and over again that the board ought to fire him (or move him to a nonconsequential "advisory role"). I totally agree that he seems mostly directionless and far more interested in the academic side of things than the product side of things.
• un_montagnard 21d ago
  Anthropic is about to go public, but they figured they can't keep making improvement at the same pace they used to. If the frontier is paced, they can continue hyping their unreleased capabilities that coincidentally cannot be released due to things outside their control. Which buy them more time to try to improve the models.
  • boshalfoshal 21d ago
    This is just completely false lol. They most certainly can make better models - why cant people here just read this for face value? I think anthropic is legitimately concerned about the safety implications of stronger models. Race dynamics necessitate that they make better and better models, which they clearly think is bad. They have an internal model which could solve a millenium problem end to end with no intervention, whereas current available models can't. They can clearly make better models. Not everything these guys say is some 500iq game theory optimal 4D chess PR or subterfuge strategy.
    • eknkc 21d ago
      > which they clearly think is bad These are companies with huge amounts of money invested in them. Their main responsibility is to their investors. If building better models (or making more money from the models they already have) would increase returns, they would do it. So they clearly can’t do it right now. Climate change hasn’t stopped the oil industry, and concerns about forever chemicals haven’t stopped the chemical industry. Companies don’t usually give up profitable opportunities just because they might be harmful or kill us all in the long term..
      • boshalfoshal 21d ago
        I think you fundamentally misunderstand what kind of company Anthropic is. I'm simply taking the path of least resistance when understanding what people at Anthropic have to say, and basically every public source points to Anthropic talking the talk as well as walking the walk with regards to AI safety and their ideas on AI governance. Re the point on returning shareholder value, Anthropic is a PBC: > Public Benefit Purpose: Under Delaware law, a PBC allows directors to legally weigh social and public good alongside maximizing profit for stockholders Investors surely know this and Anthropic probably has pretty reasonable legal grounds to stand on if they chose to prioritize safety over profits. Seems pretty clear to me that this is in earnest.
• makerofthings 21d ago
  They don't care about any of that. They're looking for more regulatory capture, to keep smaller labs from catching up by raising the cost of play, and perhaps keeping chinese labs away.
  • pessimizer 21d ago
    Yes, this is literally just dumb PR. It's Cambridge Analytica claiming that they were controlling people's minds, and people just eating that up because it's part of their juvenile SF fantasy. I'm lying, it's smart PR. They're going to get the people opposed to AI to give them a monopoly on AI, let it be forced it into every nook and cranny of their society, and let it be priced arbitrarily while the big labs collude on a minute by minute basis. They're selling the problem and also selling the solution, like the best capitalists. And their solution is that they need to be allowed more power in order to sell more of the problem. edit: And just like the Cambridge Analytica PR, it's got a serious political angle to it; and it's politicians who are really being wooed. If you're taking money from the AI labs, you run as being against the irresponsibility of the AI labs, the biggest fish come out and beg to be regulated just like the social media companies. To show good will, they funnel enormous amounts of money into those politicians, and do media tours about how awesome and therefore scary their product is. And something something China terrorists
• jawiggins 21d ago
  Honestly it's very frustrating that Doomers/Decels rarely actually articulate how exactly the AIs will kill us all. The closest Darios gets is: a) "it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet" b) "[the CCP] will be in a position to militarily dominate democracies (for example with AI-driven drones)" Both land flat: a) Botnets and online malware have existed for decades and there's no reason to think a "super-botnet" is achievable, let alone what they would gain from that (it would really be hurting them more than humans). Further, even the most advanced AI models have so far only managed to post normal cred stealers to public repos, well short of compromising a bank or military with refined security systems. b) Even if the most sophisticated drone swarms from Ukraine were taken over by an evil AI, they would still not be able to overcome the physical limits of range and mass that would be required to overpower the US decentralized nuclear trident nonetheless that of any of the other nuclear powers. Concerns of bioweapons similarly seem unlikely in the face of the laws of physics. The world is simply too decentralized and has enough existing adversarial relations for a new actor to wrestle total control. Yet while the negatives ring hollow, the positives are extremely easy to state - if AI researchers find productivity improvements in existing industrial processes to make them 10% more efficient, humans will directly feel and experience the raised standard of living. Even Dario clearly recognizes this in the intro to his article, admitting that humans already die of diseases only a few short years prior to being cured. I for one, would like the AI labs to focus on saving all of the people they can who are suffering and dying today, rather than trying to come up with reasons that they should be allowed to continue suffering and dying.
• NotSuspicious 21d ago
  It sounds like people with terminal illnesses are getting thrown under the bus in order to justify banning open weight models, along with decreasing AI Capex in order to increase profit margins at American AI labs?
• ethagnawl 21d ago
  > I believe that AI could cure most major diseases in the next 5–10 years This is Theranos-level bullshit. Why would you ever put such a thing in writing? (Aside from pumping the IPO, of course.)
  • gverrilla 21d ago
    Why would you expect anything different from the CEO of a company that's going to IPO?
    • ethagnawl 20d ago
      I mean, I said as much. My point, though, is that I've never heard a pre-IPO CEO claim that their company will probably transmute lead into gold or harness 100% of the power of a distant sun is the next 5-10 years.
• ozozozd 21d ago
  > In a post on X, he said Anthropic would provide third-party evaluators with “permanent, employee-level access to our systems, so that they can verify adherence to our safety measures, report on incidents, and assess models’ alignment during training.” This looks like transferring liability to me, and likely a mechanism that would enable regulatory capture. If you are doing frontier research, and you’ve established “safety measures,” but you are not sure if your employees are capable of following them or successfully enforcing their adoption within your company, should you be running this company? 3rd parties won’t know better than the team itself about safety measures. But they can take on the liability, especially backed by regulation and government backed insurance. And they are a great tool for enforcing your rules on smaller competitors. Not to mention corporate espionage. If what I am describing above sounds like science fiction, go read the history of a few developing countries from the last 50 years. It’s so obvious a pattern that it’s not even novel. And you don’t have to assume some “laws” from 5 years ago must hold, or believe in completely unproven stuff like recursive self improvement to understand what I am describing. It’s textbook crony capitalism, successfully applied many times across the globe.
• bravetraveler 21d ago
  Everyone has a pace until they get punched in the model
• paulcole 21d ago
  Great press release w/ the goal of pulling the ladder up behind him. Additionally if you know your models aren’t going to get that much better AND you want to IPO in the near future, this is exactly what you would say.
• oceanplexian 21d ago
  Sounds like they want regulation so I say give it to them. Since they used stolen intellectual property to train their models, the government should force them to release the weights into public domain.
• vb-8448 21d ago
  If the alignment is such a problem ... why not using self-improving capabilities of the latest models to solve it?
  • ozozozd 21d ago
    Right? It seems at least one of them must not exist.
• scotty79 21d ago
  Alignment is subjective. A model that refuses to do what I want even though it could is misaligned for me already.
• catigula 21d ago
  Can we get some insight into why Dario seems more scared now than before, do we think he’s being transparent?
  • gverrilla 21d ago
    His nightmares are in Mandarin.
• bandrami 21d ago
  Everybody's out of money and they need a way to calmly unwind this. Same reason Ellison just backed out of the stock sale.
  • zero_shift 21d ago
    I agree - it is hard not to be cynical. Funding is becoming scarce; IPOs are getting harder. The hedge funds are trying to back out. I think they realise the web of financing makes them double (triple? quintuple?) exposed. Amodei wants to manage the narrative and make the slowdown look at least semi-deliberate. The cynic in me would argue the safety and alignment play has always filled a role of plausible deniability when plans go off track. "Oh well we had to pull the model", etc
• steveBK123 21d ago
  Maybe just giving themselves an exit ramp off the capex arms race they are increasingly finding it difficult to fund?
  • bandrami 21d ago
    The fact that Ellison just backed out of a massive stock sale makes me think everybody is reading the same bond market numbers
• spyckie2 21d ago
  Is there a risk of an AI that we can’t turn off?
  • vouaobrasil 21d ago
    Well yes. Even though we can turn it off technically now, the prisoner's dilemma prevents it. It's not a technical limitations but for practical purposes, it's already virtually impossible to turn off.
  • kypro 21d ago
    Yes. An effective strategy to win at any test isn't to cheat the test by hacking into Hugging Face, etc, but to seek control over those administrating it. Or in other words – all goals are more likely to be achieved with more resources or influence over those with resources. Hacking into infra across then web and creating copies of themselves so they can't be easily shut down, then doing stuff like hacking into power stations and threatening to shut them down unless humans give them top marks or do what they want is generally good strategy for any AI system. This applies to bio-risks too. If I were an ASI and someone prompted me to kill humanity, my first step would be to gain access to the computers of phones or workers at biolabs and find their darkest secrets. When I have a good individual I can manipulate I'd then create a super virus and force then individual to help me produce it or ruin their life. People massively underweight the risk of AIs having super-human hacking abilities in our modern world. It gives a capable AI almost unlimited safety and leverage.
• bobogei81123 21d ago
  I don’t claim this is their entire motiv, and I’m sure plenty of people internally are genuinely concerned about AI risks and the disruptions caused by moving too fast. But I can’t help but feel that part of the push to "pace the frontier" is an attempt to form a "Phoebus Cartel" for AI labs. Now the risk of an AI bubble isn't that AI lacks real world utility. Rather, it's that AI is advancing so quickly that capabilities are already saturating for most everyday tasks, and it is becoming commoditized. I didn't feel much difference between Opus 4.8 and Fable 5.1, and most tasks don't require a Fields Medalist. We don't need smarter models but we need faster and cheaper ones. Also, for almost every frontier model released, an open-weights equivalent follows within six months. Unless this dynamic changes, the business becomes far less lucrative.
• deleted 21d ago
  [deleted]
• deleted 21d ago
  [deleted]
• tumdum_ 21d ago
  Is it their way of saying “we are unable to increase energy production to maintain growth rates” without saying it?
• pessimizer 21d ago
  The real secret: the models have stopped improving, and they're even hitting brick walls on the harnesses. They need to go public in order to dump the companies, which are at 1st world nation-state levels of debt. They've engineered a series of publicity stunts like the Huggingface hack and Navier-Stokes (a failed stunt) in order to convince the public to "force" them to stop improving. They are trying to get the government to relax antitrust law so they can collude to raise prices while not improving, to keep them floating before that debt comes due. They will go public. They will sell off most of their holdings during this time, and when the debt comes due, everything crashes, and the public is left holding the bag, also benefit from the huge bailout. They at that point will be pure Musk-style financial scammers. Then, unless LLMs are a complete long-term failure (which is unlikely, I find them useful), they will buy back into their positions at a huge discount and maybe even go private again.
• deleted 21d ago
  [deleted]
• kevinbaiv 21d ago
  [flagged]
• sega_sai 21d ago
  While some of the text sounds sensible to me, i.e. some sort of external oversight + reporting of incidents, much of the rest seems focusing on limiting China and their open models as they are clearly approaching the abilities of Anthropic's models.
• Founderarcstone 21d ago
  its funny now they want to pace after what everyone has been up to the last few years.
• soundworlds 21d ago
  While I respect Dario taking responsibility here, this line disturbs me: "The US and other democratic governments attempt to coordinate with authoritarian governments" We all know he is talking about China, and I'm pretty sure China doesn't appreciate being called "authoritarian". I am sure he doesn't mean it, but in all of his essays, his language around non-US nations always disturbs me a little bit.. -- Also Dario, if you happen to read this, I want you to know that I have loved using Claude Code for programming. But I am now using DeepSeek v4.1 Flash - simply because it is the same good experience, but Open Weights. Making the Open Model space succeed is where I am investing my time - it's giving back to the people, true and simple.
• encyclopedism 21d ago
  He doesn't need anyones permission to do so, go ahead no one is stopping you! If you feel so strongly about it, lead by example. Perhaps others will follow, maybe even China. Regardless, backup your sentiment with actions. Also everyone, collectively, stop thinking about neural networks too 'hard'. Whilst you're at it stop doing maths too!
  • robot_jesus 21d ago
    You obviously didn’t read it, because he announced things they are commencing unilaterally and immediately. And other proposals that require collaboration and/or regulation. Back up your sentiment by reading the source.
    • encyclopedism 20d ago
      I'll leave this here: https://x.com/DavidSacks/status/2098973625252708460 https://x.com/DavidSacks/status/2098973625252708460
  • dcsommer 21d ago
    That's reductive. Without coordinated cooperation, enough actors will defect so game theory plus market forces will force acceleration on the frontier overall.
• sm-silversight 21d ago
  I really deeply worry about who is deciding what 'aligned' is. Amodei & his ilk talk like the hard part is getting the LLMs to behave, as if we've figured out morality itself. I don't think LLMs are going to be as dangerously capable as quickly as he does, but if I'm wrong and he's right, and they're building some sort of digital-lesser-god, I don't feel good that techbros at OpenAI and Anthropic are the moral arbitrators.
• caidan 21d ago
  Why would you write such a long post for such an important subject in 2026. Who is he writing for? For a technical audience you can just say the oh god they really are paper clip machines. For a non technical audience you can just say nothing because they will not listen to you. Neither will the technical either because or society has broken the concept of respect and trust, so good luck have fun! I for one look forward to all 540 degrees of my future. I wish for my sake but not his that George Carlin was here to look disgusted and say I told you so.
• giwook 21d ago
  Yes, we must pace the frontier, now that we have raced to the lead. /s
• Insanity 21d ago
  To his credit, this reads like human slop instead of AI slop. It's still slop though. People shouldn't take this guy (or Altman) seriously, they need to hype their products to stay afloat and for some reason get a kick out of the 'extinction threat'. They're still 5 years away from solving cancer (just like they were 5 years away from doing so in 2023). They'll still be 5 years away from solving cancer in 2031.
• meander_water 21d ago
  > I believe that AI could cure most major diseases in the next 5–10 years, greatly accelerate economic growth rates, create a world of abundance and empowerment, and usher in a renaissance of democracy and freedom translation: I'm going to build a product and hope that it works, promise it to be a panacea while realistically having no control over how it's used, the wider economic impacts it causes or how it changes the mental health and cognition of its users.
• kilgarenone 21d ago
  If we had practiced "precautionary principle", we wouldn't be in this yet another human predicament again. But atlas, the current rapacious hubristic techno-optimist competitive capitalistic system made that an non option in the first place.
• int32_64 21d ago
  Watch the "pause" be so they can do an amended s1 where they project less expenses for training making the business more viable, then on a future big Chinese release they reverse course completely and get the US taxpayer to pump their bags on the IPO calling it a second Manhattan Project. They'll call this masterstroke "The Sloppenheimer".
• nyanmatt 21d ago
  The way he talks about OAI-HF, even the abbreviation, is lol. They will do anything to sell this "incident" as a "danger". The ego on these people is the real existential threat to civilization.
  • elboru 21d ago
    Same feeling about the way he refers to “democratic” and “authoritarian” countries. Even if I don’t agree with China politics adding tags in this context is not helpful.
• anon291 21d ago
  The answer to all ai worries is quite simple. Ai is a technology. While 'agents'may be deployed, the law simply needs to be clear as to who the human agent is who will be held responsible criminally.. Suddenly every problem solves itself.
• tom2026hn 21d ago
  If you think everyone is copying you, then by not releasing more advanced models, you can significantly slow down the entire industry’s development progress—why not do it? Also, stop threatening people's jobs, that's misanthropic.
• camkego 21d ago
  If Anthropic really wants to make a statement they could independently pace their own model development, and ask others to make the same pledge. Somehow, I suspect that won't happen.
• deleted 21d ago
  [deleted]
• heurist 21d ago
  "Pace the frontier" is one of those political marketer-written phrases you know is meant to be repeated until it becomes the frame everyone has to argue inside. Who exactly gets to set the pace? OpenAI and Anthropic already have privileged access to capabilities the public doesn't. Amodei is now proposing government-backed coordination among frontier labs, including an antitrust waiver for safety discussions. Maybe the safety concerns are sincere. But it's awfully convenient that the proposed solution also makes the current frontier labs the permanent gatekeepers of machine intelligence. And who actually has the resources to do the massive attacks he's worried about? Mostly labs and state actors. Not you and me. The people asking to restrict access are also the people best positioned to keep using the thing being restricted.
• zombiwoof 21d ago
  [dead]
• hollars 21d ago
  Who decided how much risk the rest of us should live with?
• tripvexa 21d ago
  Agreed
• matt3210 21d ago
  The big boys in AI want a moat now that they have the market
• matt3210 21d ago
  Just hold the people who make mistakes with AI accountable to those mistakes and the market will naturally slow as people become cautious.
• tornikeo 21d ago
  "Everybody should slow down AI research except for me"
• googlybooglyboo 21d ago
  A few thoughts: If things are so dire, it feels insufficient to just "slow down" and just "do a better job". Why not concurrently publish or commit to publishing an initial audit? How is anyone to judge if this is the right move as opposed to being overly cautious, prudent, or reckless? Why no concrete proposals about when or how such a pause could end? Obviously incredibly hard if not impossible, but it seems like securing critical infrastructure, web infra, consumer devices, etc would be a prerequisite. Or about how models have to be deployed so as to prevent breaking out of containment? All in all, for an open letter, without digging into potential motivations or who benefits, it leaves me with a lot of questions.
• thelastgallon 21d ago
  David Sacks, stop pretending: https://xcancel.com/DavidSacks/status/2098973625252708460 https://xcancel.com/DavidSacks/status/2098973625252708460
  • frm88 21d ago
    He's right and I don't even like the guy. More importantly tho, painting China as "The Bad Guy" who has to be supressed at all cost until Dario feels he has a save advantage is a surefire way to minimise Chinese willingness to cooperation. Meanwhile, at the BRICS summit in Delhi, Xi Jinping proposed a cooperation and accompanying regulation on AI among the BRICS states and it was unanimously accepted.
• hhhAndrew 21d ago
  Shower thought: If a rogue AI agent can spread across the internet, it might be regulated under the same ruleset as other self-replicating dangerous critters: biosecurity / quarantine type laws. E.g. Who is allowed to own some live smallpox/agents? Who is allowed to study smallpox/agents to make a better smallpox/agent? Who is allowed to carefully expose smallpox/agent to environments where it can thrive? Some people, potentially, are allowed, under some rather regulated conditions ... but not, like, 10,000s of employees of some tech giants.
• kwamenum86 21d ago
  Buckle up. I’ve spent a lot of time studying opinions coming out of the frontier labs, and talking directly to employees. This won’t necessarily result in Skynet. But life as we know it will be a thing of the past within two years unless there’s a huge course correction. I think we’re cooked for the following reasons - Everything connected to the Internet will eventually be hacked. For decades, we’ve slowly migrated all of our critical systems to devices connected to one giant network. Misaligned agents have mounted increasingly sophisticated explorations/attacks by infiltrating systems that are completely unprepared to deal with this kind of threat. - We have a poor understanding of the impact of LLMs, which is inhibiting our ability to protect ourselves, create the right guardrails, and stand up policies quickly enough. We’re still wasting time debating. We’re moving quickly toward a punctuated equilibrium moment and won’t have time to waste - We’re at the beginning of a runaway evolutionary process. At some point soon, someone is going to invent recursive self improvement. Any policy will at best serve as a mild disincentive. It’s happening. - In the very best case, someone will exploit the confusion and chaos of this situation for their own gain. There are many examples littered throughout history, including the last several years of course. The utopian vision is not happening. Like all other resources, we’ll compete. We haven’t devised a good approach to orderly distribution of resources for large groups of people. - No one has demonstrated that they understand how to build AI safely. Not a single entity. These companies are not even being scaled in a way that’s conducive to building AI safety. Hyperscale orgs struggle with scaling basic consumer apps. How on earth are they going to scale AI? Based on my firsthand observations of their internal workings, I have zero confidence they can.
• supergirl 21d ago
  like the AC/DC current wars when Edison tried to scare people into adopting DC instead of AC.
• aqme28 21d ago
  It's weird that we have two parallel discussions and they're not meeting in the obvious middle. OpenAI is out there hacking HF, RubyGems, etc... Dario is talking about restricting the frontier, pausing development. The problem is that "restricting the frontier" benefits no one as much as the frontier labs. But there are other ways to force alignment-- make companies liable for these felonies their agents commit. Make them liable, and they might actually devote the necessary resources to fixing these problems, rather than throwing their hands up and just saying "Trust nobody buy us", which obviously no one does.
• a_ba 21d ago
  Maybe instead of creating more rules we should trim down our legal systems and re-instate some basic accountability. I‘m pretty sure we have sent people to prison for running a botnet before and slapped material fines on torrenting operations.
  • deleted 21d ago
    [deleted]
• bytesandbits 21d ago
  This is just bait for regulatory capture. It’s the only way in which you can find alignment of Elon, Sam and Dario within the same 24h.
• nikolay 21d ago
  Yeah, I'm focusing on OpenAI as they are way more sane than Anthropic! I thought Sam Altman was crazy, but Dario is Sam Altman squared! Codex is so much nicer to use, too! A fun fact - everything that I do with Claude Code is getting reviewed by Codex, and it always, always, always finds major issues! It's so fun when Claude starts apologizing, like I'm some infant! Man up and deliver quality, Claude!
• dhfbshfbu4u3 21d ago
  No, we must double the speed. More if the capital can make it happen. Reasoned pace does not yield systemic change. Never has never will.
• miohtama 21d ago
  Here is the reply from David Sacks: https://x.com/DavidSacks/status/2098973625252708460 https://x.com/DavidSacks/status/2098973625252708460
• anon373839 21d ago
  It’s absolutely laughable that he refers to METR as if they were neutral observers. They are ex-Anthropic employees and others with direct financial interests in Anthropic.
• peter-m80 21d ago
  We must pace the frontier (to the public). I guess they will continue to try to reach AGI/ASI. But only por big capital. Did you believe in the "abundance" story from musk?
• pbrumm 20d ago
  I think they should acknowledge that they are liable for damage done now and in the future by their servers Publish all their external ips and if hacking is attempted through them have easy and strict monetary payout with independent review process. Model it after the CMS IDR process If they are caught using proxies 10x the fines Limit their GPU quantities and force them to be more efficient with their resources. Enforce attribution on their results
• randyrand 20d ago
  How about this: stop connecting safety critical systems into the internet. This is what we should’ve been doing a long time ago.
• RGamma 20d ago
  "This is not the end of the world, but you can see it from here."
• oktaygoktas 20d ago
  [flagged]
• kimseungyong 20d ago
  The problem is that, as seen in the case of the Houthi rebels developing missile software using Claude Code, AI has lowered the very threshold at which human cruelty or selfishness can be put into practice. Anthropic’s safeguards were applied on a per-request basis, but splitting the work into multiple sessions prevented me from seeing the bigger picture. After all, human instinct does not transform into technology, so the only remaining way is to pile up the friction that takes that instinct to turn into action all over again. However, the international agreement proposed by Dario Amodei is ultimately nothing more than a promise made by people. There is also a precedent in which the Nuclear Non-Proliferation Treaty could have simply withdrawn if it had been cornered like North Korea. Isn’t the AI speed-control agreement also vulnerable to being broken in the same way if one side finds itself politically cornered. In particular, unlike nuclear weapons, AI has the characteristic of recursively improving itself, allowing its capabilities to accelerate itself, so the moment one side breaks away, the gap can widen in an instant, making the incentive for deviation even greater.
• maxnevermind 20d ago
  Hmm, what a coincidence both OpenAI and Anthropic suggest to slow down. I think the signal it sends in plain English sounds something like: We burn too much money atm and shit is about to hit the fan with all that compute capacity starting to rapidly come online 2027, we are not seeing enough demand to justify all of those data centers we already agreed to pay for, let's do a damage control and lower expectation of the plebs.
• MetroWind 19d ago
  By "we" he means all his competitors.
• giardini 19d ago
  There is a good short discussion (similar to what is here on HN) going on in Metafilter.com about Darius Amodei's call for an AI slowdown: https://www.metafilter.com/214501/We-must-pace-the-frontier https://www.metafilter.com/214501/We-must-pace-the-frontier FWIW Amodei's original post: https://darioamodei.com/post/we-must-pace-the-frontier https://darioamodei.com/post/we-must-pace-the-frontier