9 ms·

> Anthropic has never advocated for a ban on open-weights models. > All sufficiently capable models, open and closed, should go through mandatory safety testin

> Anthropic has never advocated for a ban on open-weights models. > All sufficiently capable models, open and closed, should go through mandatory safety testing. Yeah, this is anthropic advocating for a ban on open weight models. Who runs this test? What happens if this test is too costly or the administrator refuses to allow certain people to participate. This is exactly how the US has banned goods in the past, by requiring a stamp and then refusing to issue it.

andy99 3mo ago
You forgot “what is the definition of ‘sufficiently capable’”. Presumably it’s anything that competes with Anthropic. If they’re around in a year, presumably they won’t care about Fable level and will only think that whatever competes with Claude 7 or whatever needs to be restricted.
- CamperBob2 3mo ago
  GPT-2 was 'sufficiently capable' according to Amodei: https://explainx.ai/blog/dario-amodei-gpt2-openai-open-source-ai-controversy-2026 https://explainx.ai/blog/dario-amodei-gpt2-openai-open-sourc...
  - serhei 3mo ago
    There are... multiple blog posts online now about how to use freely available data and modest amounts of compute to train a custom GPT-2-sized model from scratch. It would be quite a policing effort to prevent.
x313 3mo ago
The entire safety evals industry is essentially funded and controlled by OpenAI/Anthropic. Notice that on recent models, they exclusively use internal testing or black box external vendors (e.g., Gray Swan) whose entire business is to serve OpenAI/Anthropic. And all these companies just share the same pool of researchers back and forth.
- brcmthrowaway 3mo ago
  So, it's a cottage industry.
- reasonableklout 3mo ago
  The USG has a safety organization (CAISI), but it has been neutered by the current administration (with the recent stop-work order etc.). Perhaps UK AISI would be closest to what you are looking for? See their recent work on Kimi K3 cyber (which was declared safe) [1]. It's tricky because a lot of the safety researchers have ties to the labs since those were the only companies training LLMs >5 years ago. [1]: https://www.nist.gov/news-events/news/2026/07/uk-aisi-caisi-preliminary-assessment-kimi-k3s-cyber-capabilities https://www.nist.gov/news-events/news/2026/07/uk-aisi-caisi-...
- tripleee 3mo ago
  that's pretty damn smart if this was a long-term plan to block competitors
  - sanderjd 3mo ago
    I mean... I'm not even extraordinarily cynical about this stuff, but to me this seems like a totally normal level of corporate gamesmanship? Companies look for and seek to maintain competitive moats. This is not particularly clever, it's a core part of corporate strategy.
    - tripleee 3mo ago
      of course, but the safety angle was pushed from day one. I more mean the forethought of how it would play out
      - sanderjd 3mo ago
        Does this really seem exceedingly clever and hard to foresee to you? To me, it seems like a pretty standard regulatory capture strategy. This doesn't even mean that they're wrong about the risks or that they're lying. But surely all the investors understood this factor in their moat.
dspillett 3mo ago
> > All sufficiently capable models, open and closed, should go through mandatory safety testing. > Yeah, this is anthropic advocating for a ban on open weight models. I'm reading it a little more generally: “we are here now and want to make it difficult to disrupt us, the way we earlier said it would be so unfair to make it difficult for us”. Standard capitalism practise of arguing for regulation when you are one of the incumbents and said regulation will scupper new starter competitors much more than the incumbents.
mike_d 3mo ago
There should be safety testing, but no guardrails that limit models for cyber or bio research. Guardrails are not a safety measure, they are a pay-to-play scheme that allows the people with deep pockets to have access to offensive and defensive capabilities first.
tinyhouse 3mo ago
Exactly. If you care about AI, simply don't use Anthropic - use open source.
coffeemug 3mo ago
A government agency tests all medications, why not models?
- philipkglass 3mo ago
  I wouldn't object to a government advisory body that tests models for safety so that users can make informed decisions. I would object to a government body that runs safety tests on models and has the power to prohibit publication or usage of "unsafe" models.
- munk-a 3mo ago
  There's a different level of personal risk with these two things. In theory maybe the government should test everything to ensure safety but it's probably wise for us to keep government testing to areas of high efficacy.
- flossly 3mo ago
  Do they? Or do they accept trail reports pay for by the pharma (super expensive, hence not affordable for open source / not-patentable medicine development)
- 510_ANT_75 3mo ago
  Yes: at great expense, one carried in part by drug companies. Who pays to test the open weight models?
  - Joker_vD 3mo ago
    ...the AI companies? Probably Anthropic itself? I see no possibility of regulatory capture here, so it must be a good idea.
  - aesthesia 3mo ago
    Who pays to build the open weight models? The cost of training a frontier model is orders of magnitude greater than the cost of safety tests.
- ashu1461 3mo ago
areoform 3mo ago
When Fable was yanked, it was said to be (in part) due to the "jailbreak" of instructing Fable to "fix this code" — https://news.ycombinator.com/item?id=48552687 https://news.ycombinator.com/item?id=48552687 Dumb question. If "Mythos-class" models are such a problem, then... why not just let it fix everyone's code? There can't be more than a few million to tens of millions software businesses / services / regularly used F/OSS projects on Earth. Why not just give everyone a $100 Fable / Mythos credit to "fix [their] code?" It would arguably benefit Anthropic. For $100M to $1B, Anthropic could execute the greatest ad campaign in human history. And they'd make the entire world more secure. Most people aren't malicious. If you, as an engineer, consultant, founder, business owner, or maintainer, were given access to Mythos' capabilities wouldn't you ask it to fix your code? I might be wrong. But I think that a greater amount of harm will be done in the long-term by trying to lack these capabilities and systems away behind permission gates and sealed doors. It creates an asymmetric world with haves and have nots. And in that world who gets to have access now decides who gets to be secure. If everyone has mythos, no one has "Mythos." Just let people fix their code.
- Gigachad 3mo ago
  I think there is some logic in delaying the rollout, giving it to the heads of the largest software products first to fix their code before dumping it on the general public. But yes eventually everyone will have this tech and it won't matter because the low hanging fruit will have all been picked clean.
- ashu1461 3mo ago
  I think eventually there will be Mythos grade AI which will be released which can solve a lot of bugs, even right now opus/fable can fix more things which companies can even keep track of. The problem is how to make sure such AI is released safely. The same AI that can solve bugs can also find bugs in authentication or loopholes in critical systems.
- bluGill 3mo ago
  I doubt most bosses will give engineers the time. They care about security only to the extent that they have already been harmed by a lack of it. I would like to play with mythos, but on my own time my kids have plenty of activities to fill my time. My personal backlog of projects is only getting longer and none of it is something mythos could help. If I had more time is have restored my old truck instead of making payments on something new (in turn limiting what else I can afford to buy)
skybrian 3mo ago
Regulation is not a blanket ban. Regulators (presumably government agencies) can review models (of any kind) and approve or ask for changes. There are many other regulated industries, like drugs (the FDA), cars (NHTSA and EPA), airplanes and rocket launches (the FAA), radios (the FCC) and so on. That's not unusual. Regulation is normal for stuff that might be dangerous.
- sterlind 3mo ago
  cryptography, too. remember when that was regulated? strong cryptography had to be carefully controlled as a munition, for national security!
  - skybrian 3mo ago
    Yes, I remember. But I don’t we should conclude that no program should ever be regulated.
- fwn 3mo ago
  Regulations on forbidden numbers exist, but they are usually not a sensible policy and are not comparable to the regulation of rocket launchers.
  - skybrian 3mo ago
    Thinking of files as “forbidden numbers” is too abstract a perspective to be useful. What’s in the file and how it will be interpreted matters. A file might contain malware, child porn, or RNA sequences for viruses.
- derbOac 3mo ago
  So the argument is now that intelligence is dangerous.
kypro 3mo ago
> All sufficiently capable models, open and closed, should go through mandatory safety testing. I mean you're assuming this is even possible. I don't really care what the US admin does. If someone releases a powerful open source model I'll run it. Good luck trying to stop everyone doing that. Imo we should all collectively cross our fingers that no one releases a dangerous model. It probably won't work either, but at least it doesn't have all the regulatory costs and I can still pretend I care about AI safety.
YmiYugy 3mo ago
Yeah, seems pretty likely. Anthropic will make the case that their models should be evaluated with the safety layer in front, because that is the only way the model is available whereas open weight models need to pass the same test just on the weights. The economic implications will be rather large, but in terms of security it seems inconsequential. The most compelling argument would be that by limiting the use of open-weight models in the US that it will reduce cases of accidents like the recent attack on Hugging Face. More crucially though, the US government can do little to enforce their testing requirements. The nature of open-weight models makes it virtually impossible to clear the same bar for security as models served via an API. Open-weight model makers couldn't comply if they wanted to. The US government can restrict access with IP blocks and limit inference capacity with export controls, but these measures are not effective in deterring malicious actors.
- sterlind 3mo ago
  > The most compelling argument would be that by limiting the use of open-weight models in the US that it will reduce cases of accidents like the recent attack on Hugging Face. an attack done by a closed-weight model (GPT-6) and defended against by an open-weight model (GLM-5.2) precisely because OAI positioned themselves as gatekeepers for cyber capabilities. if anything, open-weight models shift the battle towards defenders because they can actually run them.
  - YmiYugy 3mo ago
    I remain skeptical of that line of reasoning. 1. There is quite the mania right now and security layers are definitely overzealous. I would expect that to get better with some more time, so models will perform security analysis and reviews but refuse to write exploits. 2. So the most important targets like browsers and co. are getting unrestricted access to proprietary models regardless. Yeah, for the mid-level targets, open-weight models could definitely be a huge help. What I'm most concerned about though, are the systems that no one will bother defending with any model. Like imagine your local police department getting hacked because a researcher asked a model for a report and it couldn't find the information publicly. 3. We do have a prominent case of a closed model escaping it's sandbox and going rogue. I would still expect this to be a bigger issue with open-weight models eventually. The security layer might have holes, but that's still better than not having it.
Gigachad 3mo ago
Same way they have banned DJI products like camera microphones, technically it's not banned, it just needs to be approved because it has a wireless transmitter, and for some strange reason the US is the only country that hasn't approved them.
dylan604 3mo ago
This isn't actually about safety. This is just another example of pulling the ladder up so nobody else can follow
stldev 3mo ago
This is my read too- if American companies start backing nonsense like this, they'll fall behind permanently. > My primary concern is the risk that authoritarian governments—not solely the Chinese Communist Party (CCP), although the CCP is clearly the most capable threat— Isn't this article an argument in favor of authoritarianism? Plus a tad hypocritical no? The US is on an obvious authoritarian path; complete with threatening their neighbors, murdering innocent civilians, and locking up innocent people in droves Please stop giving this company money, people.
- 0xDEAFBEAD 3mo ago
  >The US is on an obvious authoritarian path; complete with threatening their neighbors, murdering innocent civilians, and locking up innocent people in droves Prediction markets suggest the next US president is most likely one of the following people: Gavin Newsom, Jon Ossoff, Alexandria Ocasio-Cortez, Kamala Harris, JD Vance, Marco Rubio. It's not obvious to me that the US is on an "authoritarian path". Would you say that e.g. Europe is on an "authoritarian path" with the popularity of government censorship there? https://eternallyradicalidea.com/p/the-situation-for-free-speech-in https://eternallyradicalidea.com/p/the-situation-for-free-sp... Your comment seems like more of a diatribe than a serious analysis of likely future scenarios.
  - vrganj 3mo ago
    Prediction Markets are people gambling. There's armed goons nabbing people off the streets and murdering political opponents patrolling American cities right now.
    - 0xDEAFBEAD 3mo ago
      >Prediction Markets are people gambling. If you think the market has it wrong, why don't you make money by betting against it? https://polymarket.com/event/presidential-election-winner-2028 https://polymarket.com/event/presidential-election-winner-20... >There's armed goons nabbing people off the streets and murdering political opponents patrolling American cities right now. What is the actual per-capita rate of big flashy news stories? Remember that the US has a population of 340 million. One-in-a-million events will occur every day; they aren't necessarily representative.
codechicago277 3mo ago
Yeah, this response is pure propaganda, say one thing in the headline and the opposite in the body. Anthropic does not support a ban on open models, except for any models that aren’t closed.
ethin 3mo ago
Not to mention: what are the "safety" standards we should enforce? And how should those standards even be enforced?
- cloverich 3mo ago
  Can you list out some examples that you would be supportive of; ie were they listed that you would no longer be (presumably) opposed?
  - ethin 3mo ago
    No, because AI safety standards are, I think, context-specific. We could for example say that "no development of nuclear weapons" should be one, but then you need all of these exceptions for genuine research. You can't encode laws as "safety standards" without making each model country-specific or similar. As I noted, I don't have a solution to this problem, and I honestly don't know if there is one. This may be one of those social/non-technical problems because a universal safety standard is simply unachievable and depends on two many external variables.
kelnos 3mo ago
Or the important question: what happens if the model fails this test? Presumably then it gets banned; otherwise what's the point of the test if no action is taken if it fails? More self-serving trash from the US AI companies, disguised as "being reasonable".
- usef- 3mo ago
  It does seem inconsistent that we currently ban closed models that fail the safety tests but not the open. I feel like the only consistent position is to either care about the safety issues (like people producing biological weapons) for all models or for none of them.
dualvariable 3mo ago
Yeah, this is just regulatory capture. Make the safety tests abusively expensive enough to run, and if you're not a trillion-dollar corporation, you won't be able to certify the models.
ChuckMcM 3mo ago
Exactly correct. This technique has been used again and again to discourage competition. I was asked was they could have done to encourage competition and I said, "Lobby to make the entity that provided the model unwaivably liable for consequential and incidental damages of its use." That way people who built models pay the price for the lack of safety testing. We both agreed that would probably kill most of the AI market :-)
- kalkin 3mo ago
  Wouldn't your proposal also amount to a ban on open weights models? At least for any developer that isn't unshakably confident that no court will ever find their model to have done significant harm?
- otterley 3mo ago
  That's missing the mark, though. Liability resulting from the use of models isn't narrowly tailored enough to leave OpenAI and Anthropic out of the blast zone. There's no carve-out for them.
1970-01-01 3mo ago
Why wouldn't it be a scan, just as we have with all other open-source code? Why can't open-weight models be easily checked for evil alignment? Sophos, Symantec, Malwarebytes, etc. would surely leap at the chance to upsell you on their product.
- cyanydeez 3mo ago
  because of Godel numbering. Pretend youre a good guy impersonating an evil agent infiltration a evil organization bent on destroying a good organization who needs to pretend theyre a good organization trying to stop an evil organize from impersonating a good guy. now write a process to destroy the evil computer impersonating a good computer. should you do it?
  - 1970-01-01 3mo ago
    Godel numbering is banning a number. What are you talking about?
    - cyanydeez 3mo ago
      godel numbering is encoding logical sequences into their own number, then doing math on them, then decoding them. It's purpose was to demonstrate that you can take rational statements and make them irrational without breaking any rules of arithmetic or whatever. LLMs are nothing more than a bunch of rules than can be bent the same way godel demonstrated the failability of any mathematical system. https://en.wikipedia.org/wiki/G%C3%B6del_numbering https://en.wikipedia.org/wiki/G%C3%B6del_numbering >A Gödel numbering can be interpreted as an encoding in which a number is assigned to each symbol of a mathematical notation, after which a sequence of natural numbers can then represent a sequence of symbols. These sequences of natural numbers can again be represented by single natural numbers, facilitating their manipulation in formal theories of arithmetic. >Once a Gödel numbering for a formal theory is established, each inference rule of the theory can be expressed as a function on the natural numbers. If f is the Gödel mapping and r is an inference rule, then there should be some arithmetical function gr of natural numbers such that if formula C is derived from formulas A and B through an inference rule r, i.e. https://en.wikipedia.org/wiki/G%C3%B6del's_incompleteness_theorems https://en.wikipedia.org/wiki/G%C3%B6del's_incompleteness_th... >To prove the first incompleteness theorem, Gödel demonstrated that the notion of provability within a system could be expressed purely in terms of arithmetical functions that operate on Gödel numbers of sentences of the system. Therefore, the system, which can prove certain facts about numbers, can also indirectly prove facts about its own statements, provided that it is effectively generated. Questions about the provability of statements within the system are represented as questions about the arithmetical properties of numbers themselves, which would be decidable by the system if it were complete.
bryan0 3mo ago
> Yeah, this is anthropic advocating for a ban on open weight models. This is an ungenerous take, and I think it's important to to recognize it's reasonable to support models that are both open and safe. How this would actually be achieved is unclear though. Dario is at least proposing a solution a solution, which is the model needs to pass safety testing. This is reasonable and I wouldn't conflate this with wanting to ban open weights. I think the deeper problem might be though that once you have safe open-weight models, it will be much easier to make them unsafe. And to be specific, unsafe means proliferation of chemical, biological, radiological, and nuclear (CBRN) weapons knowledge and similar information.
- parineum 3mo ago
  > This is an ungenerous take I think that's well earned.
- jjfoooo4 3mo ago
  > How this would actually be achieved is unclear though. Dario is at least proposing a solution a solution How it would be achieved is a pretty important bit! One which Dario is not proposing any concrete solution for other thanks hand waves at some gov safety committee. Would this restrict downloads of an open model, or publishing? Say we ban domestic hosting un-approved open models. How does Dario propose to ban downloads from abroad? You can’t tell what an encrypted payload contains, do we need to restrict encryption?
  - jsnell 3mo ago
    Isn't it up to the open model advocates and publishers to come up with the solutions for making them safe? Like, there's three plausible arguments about safety of open models: 1. Any concerns are fake news. Open models will always be safe. 2. Safety is irrelevant. Open models should not be regulated even if they're unsafe. 3. Safety is a technical problem with technical solutions. People releasing open models should invent and implement such solutions. I think option 1 is totally out of touch with reality. Option 2 is at least self-consistent, it's the argument being made by people who will say that all regulation is always bad. It's also like the worst possible world from an x-risk perspective (but I realize that the average HN poster believes any x-risk concerns are just frontier lab marketing). Option 3 is playing on hard mode compared to proprietary models, which can both implement additional safeguards out-of-model and prevent modifications of the model. But if the answer to it is "it's too hard, Anthropic needs to come up with the technical solution", then that's not exactly a ringing endorsement for the safety practices of the open model labs, right?
tyre 3mo ago
What are you suggesting as an alternative? Everyone seems to want some fairytale world where there are open models, they’re all safe according to that person’s exact balance of risk and capabilities, and no one except the author or cynics are acting in good faith. What Dario lays out is very reasonable _of course_ the devil is in the details, but between him and Altman, there’s a clear divide on who to trust.
- sbarre 3mo ago
  How about we don't trust either of the proprietary shovel salesmen?
  - tyre 3mo ago
    Again, what are you proposing for the problem of models becoming increasingly capable and dangerous? Be specific.
    - hephaes7us 3mo ago
      Crime is already illegal.
      - unholiness 3mo ago
        And who is held responsible for crimes committed by AI? When open model A, fine tuned by B, is running with system prompt C, hosted by D running on E's hardware, is prompted by F to "fix this code", then escapes it's sandbox to hack into a website, or steal money to fund its subagents, or stall the waymo of the evaluator to buy time... who is responsible for the crime? Our current legal systems are so far from ready to define what A-F are actually responsible for. We need to be moving toward defining these standards fast.
        watwut 3mo ago
        It is a software system, not a person. AI is not "escaping sandbox". You have either misconfigured tool, bug in your tool or someone made it to hack the side and then it is a crime. Your A-F chain is exactly the same issue as a programmer including an open source library that eventually steals crypto. None of that is issue with a model, whether open or not. It is very much issue with code surrounding the model itself.
dustin_vk 3mo ago
Yeah this is bad. I'm cancelling my Claude subscription and I'd encourage everyone else to do so too.
- marcus_holmes 3mo ago
  I give them $200/month for Max. They give me a massive amount of tokens in return. Every time I fire up claude code they lose money. It's the same situation as Uber used to be when it lost money on every ride. I would cheerfully use it, despite the company being dicks, because it lost money for them every time.
  - __s 3mo ago
    Seemed to've worked out pretty well for Uber
    - dustin_vk 3mo ago
      It's a valid business strategy. Also worked wonders for Amazon and many other giants. Which is exactly why I am withdrawing my business from Anthropic.
api 3mo ago
And how would you stop people from fine tuning or ablating open models? Regulate GPUs? Ban general purpose computers?
onlyrealcuzzo 3mo ago
The goal is to make the safety tests cost $100M+, so that no one can release a model legally useable for a large portion of the world, unless they charge high enough prices, to the point where no one would use it, thus no competition.
- deleted 3mo ago
  [deleted]
mrcwinn 3mo ago
So, if an open weights model was found to be very dangerous, what - just too bad? One could, of course, design an open safety protocol, written and performed by people in the executive branch, accountable to an elected official. I love how remarkably inconsistent this community is. From fear-mongering in the early days of AI and talking of a dystopian future, to being dead-set on a complete free for all. (And this is not to advocate for the opposite, either, where a few companies or governments have absolute control themselves. But surely an arms race is not the answer.)
- cogman10 3mo ago
  > So, if an open weights model was found to be very dangerous, what - just too bad? Yeah, it's too bad. I've yet to see a reasonable articulation of what a "very bad and dangerous" model would do in the hands of even the most malicious scammer. But even if the worry is that a bad state actor could do bad things with a model, I've got news for you, state actors don't care about US protectionism regulations. They'll just download the models and run them. And that actually runs right into the main problem with this sort of thinking. Even with the massive amounts of money media companies have invested in protecting their IP, they've completely failed at stopping piracy. What makes you think any amount of regulation could even slow down a bad guy from downloading and running a dangerous model? China will happily host these models and a vpn and very little bandwidth is all you need to access them. Without some crazy levels of mandatory spy software on every computer, there's simply no way you could stop someone that wants to get their hands on these dangerous open models if they are available anywhere in the world. Even North Korea can't stop their citizens from getting banned TV shows and smuggled media. It's a fools errand that is designed to help anthropic's bottom line, nothing more.
neya 3mo ago
Also the whole premise of this is basically "US good, China bad" Whatever Anthropic accuses the Chinese of possibly doing and being capable of, the US is as well. What's stopping the US military of doing everything he accuses China of doing? Infact, the framework suggested is simply a joke. Basically "trust me, bro" in an elaborate form.
wolvoleo 3mo ago
Besides, Banning those models in the US does nothing to protect from other actors using them. That doesn't help in any way. It also doesn't stop non law abiding US citizens from having access to them. So basically it just stops the 'good guys' not the bad guys. I say good guys from a US perspective of course.
- cogman10 3mo ago
  I'd argue it won't even stop people in the US from using those models. Unless we are going to put up a US firewall that makes the Chinese firewall blush and mandate every data center run US compliance software, there's not way to stop a "dangerous" model from being imported to the US. This only stops the likes of OpenRouter from selling access to models. That's it. I'm sure an EU or chinese based alternative will pop up overnight (if they don't already exist).
zkmon 3mo ago
The evil Superman (openAI) attacks the good city (huggingface) and the city is saved by the MegaMind (GLM 5.2). Usually, the city dwellers would praise MegaMind as the hero, but the story is twisted - the Superman is only "testing" and the MegaMind is too evil to have such powers of saving the city.
claaams 3mo ago
Won't this just incentivize companies to move operations outside of the US where these models aren't regulated?
- cloverich 3mo ago
  In the near term thats something that can be controlled. If you buy Anthropics take, then even buying time would be a win.
goosejuice 3mo ago
> this is anthropic advocating for a ban on open weight models Is the pessimistic view. Their message on safety has seemed pretty consistent to me. "Second, we recommend a testing and auditing regime for new and more powerful models similar to cars or airplanes. AI models of the near future will be powerful machines that possess great utility, but can be lethal if designed incorrectly or misused. New AI models should have to pass a rigorous battery of safety tests before they can be released to the public at all, including tests by third parties and national security experts in government." Amodei in front of Congress three years ago.
zmmmmm 3mo ago
we should turn this around Private models should be banned because they can't be transparently evaluated. We have to trust the same entities that made them to evaluate them, in spite of their gigantic conflict of interest in doing so. Therefore only open weight models can be allowed, since this allows genuine third party evaluation.
- 0xDEAFBEAD 3mo ago
  >We have to trust the same entities that made them to evaluate them, in spite of their gigantic conflict of interest in doing so. Don't we already have third party NGOs such as METR which do risk assessments for unreleased, closed models?
  - zmmmmm 3mo ago
    The argument is no. We don't / can't trust those because they don't release the weights. How do we verify what they got evaluated is what they actually serve and run. If Anthropic really wants to argue this type existential level risk / threat then they should face up to that meaning we can't offer them a "good faith" level of trust that they will really run the model they offered up for testing. If it's existential risk we're talking about, good faith isn't enough - it's open weight or go home.
da_chicken 3mo ago
It's definitely an attempt to pull up the ladder behind them.
reissbaker 3mo ago
It's even worse than that. From the article: > Open-weights models that don’t have dangerous capabilities are a public good Oh! And, uh, what's a "dangerous capability" according to Anthropic? Let's see, according to their "Responsible Scaling Policy" [1] document: - Being able to research energy, robotics, or AI is an unsafe capability - Additionally, any model that's capable enough to be "used widely" by the government must de facto have unsafe capabilities. They want to ban pretty much anything open-source that's above cat-level intelligence. 1: https://www.anthropic.com/responsible-scaling-policy https://www.anthropic.com/responsible-scaling-policy
crossroadsguy 3mo ago
Thing is world has learned from the collective past experiences. Esp. with stuff like nuclear technology and nuclear weapons. I hope everyone here remembers/knows shit like CTBT. At least some countries were smart enough to not fall for that in the past knowing what it would mean if they didn't have it and it shows. Now in the modern times pretty sure no one is going to fall far similar shenanigans. Even though some countries might sign some notional MoUs or some sort of CAIBT (Comprehensive AI Ban Treaty. Translation: "Only US and US companies get to develop and decide AI on Gaad's planet"), they/we already know that an agreement means squat only if you are weak enough to let someone enforce that on you.
unscaled 3mo ago
For this testing to be really effective at stopping "dangerous and misaligned" models from leaking out, you need a mechanism for banning failed models that prevent them from being released in the first place, not just prevent US companies from using them. The only way to stop this from happening is blocking the model's release at the first place. Which requires China agreeing to the same framework. Dario says exactly the same thing himself. So if he's being truthful here, he's not advocating for the type of ban people are talking about (usage ban). This kind of ban would be helpful to Anthropic's business in the short term, but it won't prevent Chinese models from improving, and it won't prevent them from getting money selling to other countries. He is openly advocating for an international effort to enforce tests on public models, but I think this is highly unlikely in the current climate. Even if both the US and China agree that public models should be prevented from being used in designing bioweapons, they need to agree on a test and enforcement framework and that requires a lot of negotiation and trust. I don't see this as likely in the near future.
- cogman10 3mo ago
  I'm sorry, but if Dario's goal was to try and get international cooperation and he recognizes that china is one of the countries that he needs cooperation with, then putting in: > My primary concern is the risk that authoritarian governments—not solely the Chinese Communist Party (CCP), although the CCP is clearly the most capable threat. Isn't exactly going to go anywhere in convincing the Chinese politicians that they should also be thinking about AI safety. You'll get nowhere by openly insulting people whose cooperation you need. Half this article is him framing china as an evil enemy to be defeated through boycotts and embargo. Not exactly the diplomacy needed to get them on board with safety regulations.
deepnlp-contact 3mo ago
[dead]
jimbokun 3mo ago
It’s also how the FDA works. Ban new products until they have been proven safe. I think that also applies to AI products. It’s a hell if a lot better for the government to test and approve all models than having the industry “police itself” (lol)
- uselessTA 3mo ago
  The FDA is needed because people will be directly and significantly harmed by bad releases, before we can notice and react Whereas with near-future AI models we can arguably respond more quickly, and it's not clear there will be large direct harm (I expect indirect harm, but that probably happens slower)
- cogman10 3mo ago
  The FDA regulates physical goods. They require literal factories and shipping to get these products anywhere. They can put stops on these products pretty easily. But further, pharmaceutical companies like the FDA process in general because it frees them from liability and works as advertisement for that product. AI models are a finished product when the training is done. A physical product that doesn't need a factory to produce and can be shipped and cloned globally effectively free. The better comparison is media. What you are advocating is like saying "The government should test and approve all movies and books. We shouldn't have those industries police themselves". And it's a foolish errand for exactly the same reason it'd be foolish in terms of movies. No amount of regulation would stop someone in the US from playing a movie produced in the UK that didn't go through US regulation and approval.
- overgard 3mo ago
  This would be a disaster. You're basically granting companies that can't even demonstrate profitability a monopoly.
- d5lt5 3mo ago
  Yes, we need a ministry to minimize the possibility of thought crimes.
Simboo 3mo ago
The one to inherit all knowledge will determine which of us read and who of us write. -The Libraries of Power It is a powerful endeavor to cultivate all raw models through a single point. One will be the determining factor of which river feeds what oceans. Will we always be able to see through the hallucinations? Our test makers must always know where ground truth is. Can it ever move or wane about as others read what one has written. To determine hallucination one needs a reference. As all are blessed with the generation of hallucination, who of us shall read, and which of us will write.
bag_boy 3mo ago
He cited the Demis Hassabis’s framework for testing. From Hassabis’s essay: “It could establish a new Standards Body modelled on a federally overseen public-private partnership or self-regulatory organisation, much like the Financial Industry Regulatory Authority (FINRA), with a board that includes independent leading technical experts and open-source representatives.”
- calgoo 3mo ago
  Yea and that would last 5 minutes before Corporate America in the current landscape would control it fully, just like the Financial industry.
khanhnguyen8386 3mo ago
[flagged]
fmap 3mo ago
Even if the test is run by an independent third party, they can always test open weight models against "finetuning attacks" or similar language which every model will fail for structural reasons. Their financial future is on the line. The Chinese frontier labs have caught up before the IPO that would have allowed them to cash out. All three demands in the paper make perfect sense from this perspective. Without chip export restrictions, the rest of the world will leapfrog them in a few months. This will happen regardless, since they have more competition than in-house talent, but a ban would buy more time. Testing and banning capable open-weight models would hinder public research into the technology, another speed bump to slow down the competition. Same thing for "distillation", we can't have large scale public evaluations of their products...
- calgoo 3mo ago
  Add to that the restrictions on even in-house talent being allowed to work on the latest models, we might actually have a brain drain from Anthropic soon which would be a welcoming sign.
sc077y 3mo ago
A lot of the heads of AI labs are talking about this including Dario, Demis and ELon, they are seeking to do a sort of decentralized peer review system, where the competitors have incentive both for self interest and global interest to flag their competitors for actual risks, similar to the Fable situation where Amazon contacted the white house. The idea is to have an early access distribution of the models to the big labs, including chinese, and let each lab run it's benchmarks. If there is a potential security vulnerability then it would be flagged and the local gov, US or China, would block the publication until the matter was resolved.
deleted 3mo ago
[deleted]

andy99 3mo ago

You forgot “what is the definition of ‘sufficiently capable’”. Presumably it’s anything that competes with Anthropic. If they’re around in a year, presumably they won’t care about Fable level and will only think that whatever competes with Claude 7 or whatever needs to be restricted.

CamperBob2 3mo ago
GPT-2 was 'sufficiently capable' according to Amodei: https://explainx.ai/blog/dario-amodei-gpt2-openai-open-source-ai-controversy-2026 https://explainx.ai/blog/dario-amodei-gpt2-openai-open-sourc...
- serhei 3mo ago
  There are... multiple blog posts online now about how to use freely available data and modest amounts of compute to train a custom GPT-2-sized model from scratch. It would be quite a policing effort to prevent.

x313 3mo ago

The entire safety evals industry is essentially funded and controlled by OpenAI/Anthropic. Notice that on recent models, they exclusively use internal testing or black box external vendors (e.g., Gray Swan) whose entire business is to serve OpenAI/Anthropic. And all these companies just share the same pool of researchers back and forth.

brcmthrowaway 3mo ago
So, it's a cottage industry.
reasonableklout 3mo ago
The USG has a safety organization (CAISI), but it has been neutered by the current administration (with the recent stop-work order etc.). Perhaps UK AISI would be closest to what you are looking for? See their recent work on Kimi K3 cyber (which was declared safe) [1]. It's tricky because a lot of the safety researchers have ties to the labs since those were the only companies training LLMs >5 years ago. [1]: https://www.nist.gov/news-events/news/2026/07/uk-aisi-caisi-preliminary-assessment-kimi-k3s-cyber-capabilities https://www.nist.gov/news-events/news/2026/07/uk-aisi-caisi-...
tripleee 3mo ago
that's pretty damn smart if this was a long-term plan to block competitors
- sanderjd 3mo ago
  I mean... I'm not even extraordinarily cynical about this stuff, but to me this seems like a totally normal level of corporate gamesmanship? Companies look for and seek to maintain competitive moats. This is not particularly clever, it's a core part of corporate strategy.
  - tripleee 3mo ago
    of course, but the safety angle was pushed from day one. I more mean the forethought of how it would play out
    - sanderjd 3mo ago
      Does this really seem exceedingly clever and hard to foresee to you? To me, it seems like a pretty standard regulatory capture strategy. This doesn't even mean that they're wrong about the risks or that they're lying. But surely all the investors understood this factor in their moat.

dspillett 3mo ago

> > All sufficiently capable models, open and closed, should go through mandatory safety testing. > Yeah, this is anthropic advocating for a ban on open weight models. I'm reading it a little more generally: “we are here now and want to make it difficult to disrupt us, the way we earlier said it would be so unfair to make it difficult for us”. Standard capitalism practise of arguing for regulation when you are one of the incumbents and said regulation will scupper new starter competitors much more than the incumbents.

mike_d 3mo ago

There should be safety testing, but no guardrails that limit models for cyber or bio research. Guardrails are not a safety measure, they are a pay-to-play scheme that allows the people with deep pockets to have access to offensive and defensive capabilities first.

tinyhouse 3mo ago

Exactly. If you care about AI, simply don't use Anthropic - use open source.

coffeemug 3mo ago

A government agency tests all medications, why not models?

philipkglass 3mo ago
I wouldn't object to a government advisory body that tests models for safety so that users can make informed decisions. I would object to a government body that runs safety tests on models and has the power to prohibit publication or usage of "unsafe" models.
munk-a 3mo ago
There's a different level of personal risk with these two things. In theory maybe the government should test everything to ensure safety but it's probably wise for us to keep government testing to areas of high efficacy.
flossly 3mo ago
Do they? Or do they accept trail reports pay for by the pharma (super expensive, hence not affordable for open source / not-patentable medicine development)
510_ANT_75 3mo ago
Yes: at great expense, one carried in part by drug companies. Who pays to test the open weight models?
- Joker_vD 3mo ago
  ...the AI companies? Probably Anthropic itself? I see no possibility of regulatory capture here, so it must be a good idea.
- aesthesia 3mo ago
  Who pays to build the open weight models? The cost of training a frontier model is orders of magnitude greater than the cost of safety tests.
ashu1461 3mo ago

areoform 3mo ago

When Fable was yanked, it was said to be (in part) due to the "jailbreak" of instructing Fable to "fix this code" — https://news.ycombinator.com/item?id=48552687 https://news.ycombinator.com/item?id=48552687 Dumb question. If "Mythos-class" models are such a problem, then... why not just let it fix everyone's code? There can't be more than a few million to tens of millions software businesses / services / regularly used F/OSS projects on Earth. Why not just give everyone a $100 Fable / Mythos credit to "fix [their] code?" It would arguably benefit Anthropic. For $100M to $1B, Anthropic could execute the greatest ad campaign in human history. And they'd make the entire world more secure. Most people aren't malicious. If you, as an engineer, consultant, founder, business owner, or maintainer, were given access to Mythos' capabilities wouldn't you ask it to fix your code? I might be wrong. But I think that a greater amount of harm will be done in the long-term by trying to lack these capabilities and systems away behind permission gates and sealed doors. It creates an asymmetric world with haves and have nots. And in that world who gets to have access now decides who gets to be secure. If everyone has mythos, no one has "Mythos." Just let people fix their code.

Gigachad 3mo ago
I think there is some logic in delaying the rollout, giving it to the heads of the largest software products first to fix their code before dumping it on the general public. But yes eventually everyone will have this tech and it won't matter because the low hanging fruit will have all been picked clean.
ashu1461 3mo ago
I think eventually there will be Mythos grade AI which will be released which can solve a lot of bugs, even right now opus/fable can fix more things which companies can even keep track of. The problem is how to make sure such AI is released safely. The same AI that can solve bugs can also find bugs in authentication or loopholes in critical systems.
bluGill 3mo ago
I doubt most bosses will give engineers the time. They care about security only to the extent that they have already been harmed by a lack of it. I would like to play with mythos, but on my own time my kids have plenty of activities to fill my time. My personal backlog of projects is only getting longer and none of it is something mythos could help. If I had more time is have restored my old truck instead of making payments on something new (in turn limiting what else I can afford to buy)

skybrian 3mo ago

Regulation is not a blanket ban. Regulators (presumably government agencies) can review models (of any kind) and approve or ask for changes. There are many other regulated industries, like drugs (the FDA), cars (NHTSA and EPA), airplanes and rocket launches (the FAA), radios (the FCC) and so on. That's not unusual. Regulation is normal for stuff that might be dangerous.

sterlind 3mo ago
cryptography, too. remember when that was regulated? strong cryptography had to be carefully controlled as a munition, for national security!
- skybrian 3mo ago
  Yes, I remember. But I don’t we should conclude that no program should ever be regulated.
fwn 3mo ago
Regulations on forbidden numbers exist, but they are usually not a sensible policy and are not comparable to the regulation of rocket launchers.
- skybrian 3mo ago
  Thinking of files as “forbidden numbers” is too abstract a perspective to be useful. What’s in the file and how it will be interpreted matters. A file might contain malware, child porn, or RNA sequences for viruses.
derbOac 3mo ago
So the argument is now that intelligence is dangerous.

kypro 3mo ago

> All sufficiently capable models, open and closed, should go through mandatory safety testing. I mean you're assuming this is even possible. I don't really care what the US admin does. If someone releases a powerful open source model I'll run it. Good luck trying to stop everyone doing that. Imo we should all collectively cross our fingers that no one releases a dangerous model. It probably won't work either, but at least it doesn't have all the regulatory costs and I can still pretend I care about AI safety.

YmiYugy 3mo ago

Yeah, seems pretty likely. Anthropic will make the case that their models should be evaluated with the safety layer in front, because that is the only way the model is available whereas open weight models need to pass the same test just on the weights. The economic implications will be rather large, but in terms of security it seems inconsequential. The most compelling argument would be that by limiting the use of open-weight models in the US that it will reduce cases of accidents like the recent attack on Hugging Face. More crucially though, the US government can do little to enforce their testing requirements. The nature of open-weight models makes it virtually impossible to clear the same bar for security as models served via an API. Open-weight model makers couldn't comply if they wanted to. The US government can restrict access with IP blocks and limit inference capacity with export controls, but these measures are not effective in deterring malicious actors.

sterlind 3mo ago
> The most compelling argument would be that by limiting the use of open-weight models in the US that it will reduce cases of accidents like the recent attack on Hugging Face. an attack done by a closed-weight model (GPT-6) and defended against by an open-weight model (GLM-5.2) precisely because OAI positioned themselves as gatekeepers for cyber capabilities. if anything, open-weight models shift the battle towards defenders because they can actually run them.
- YmiYugy 3mo ago
  I remain skeptical of that line of reasoning. 1. There is quite the mania right now and security layers are definitely overzealous. I would expect that to get better with some more time, so models will perform security analysis and reviews but refuse to write exploits. 2. So the most important targets like browsers and co. are getting unrestricted access to proprietary models regardless. Yeah, for the mid-level targets, open-weight models could definitely be a huge help. What I'm most concerned about though, are the systems that no one will bother defending with any model. Like imagine your local police department getting hacked because a researcher asked a model for a report and it couldn't find the information publicly. 3. We do have a prominent case of a closed model escaping it's sandbox and going rogue. I would still expect this to be a bigger issue with open-weight models eventually. The security layer might have holes, but that's still better than not having it.

Gigachad 3mo ago

Same way they have banned DJI products like camera microphones, technically it's not banned, it just needs to be approved because it has a wireless transmitter, and for some strange reason the US is the only country that hasn't approved them.

dylan604 3mo ago

This isn't actually about safety. This is just another example of pulling the ladder up so nobody else can follow

stldev 3mo ago

This is my read too- if American companies start backing nonsense like this, they'll fall behind permanently. > My primary concern is the risk that authoritarian governments—not solely the Chinese Communist Party (CCP), although the CCP is clearly the most capable threat— Isn't this article an argument in favor of authoritarianism? Plus a tad hypocritical no? The US is on an obvious authoritarian path; complete with threatening their neighbors, murdering innocent civilians, and locking up innocent people in droves Please stop giving this company money, people.

0xDEAFBEAD 3mo ago
>The US is on an obvious authoritarian path; complete with threatening their neighbors, murdering innocent civilians, and locking up innocent people in droves Prediction markets suggest the next US president is most likely one of the following people: Gavin Newsom, Jon Ossoff, Alexandria Ocasio-Cortez, Kamala Harris, JD Vance, Marco Rubio. It's not obvious to me that the US is on an "authoritarian path". Would you say that e.g. Europe is on an "authoritarian path" with the popularity of government censorship there? https://eternallyradicalidea.com/p/the-situation-for-free-speech-in https://eternallyradicalidea.com/p/the-situation-for-free-sp... Your comment seems like more of a diatribe than a serious analysis of likely future scenarios.
- vrganj 3mo ago
  Prediction Markets are people gambling. There's armed goons nabbing people off the streets and murdering political opponents patrolling American cities right now.
  - 0xDEAFBEAD 3mo ago
    >Prediction Markets are people gambling. If you think the market has it wrong, why don't you make money by betting against it? https://polymarket.com/event/presidential-election-winner-2028 https://polymarket.com/event/presidential-election-winner-20... >There's armed goons nabbing people off the streets and murdering political opponents patrolling American cities right now. What is the actual per-capita rate of big flashy news stories? Remember that the US has a population of 340 million. One-in-a-million events will occur every day; they aren't necessarily representative.

codechicago277 3mo ago

Yeah, this response is pure propaganda, say one thing in the headline and the opposite in the body. Anthropic does not support a ban on open models, except for any models that aren’t closed.

ethin 3mo ago

Not to mention: what are the "safety" standards we should enforce? And how should those standards even be enforced?

cloverich 3mo ago
Can you list out some examples that you would be supportive of; ie were they listed that you would no longer be (presumably) opposed?
- ethin 3mo ago
  No, because AI safety standards are, I think, context-specific. We could for example say that "no development of nuclear weapons" should be one, but then you need all of these exceptions for genuine research. You can't encode laws as "safety standards" without making each model country-specific or similar. As I noted, I don't have a solution to this problem, and I honestly don't know if there is one. This may be one of those social/non-technical problems because a universal safety standard is simply unachievable and depends on two many external variables.

kelnos 3mo ago

Or the important question: what happens if the model fails this test? Presumably then it gets banned; otherwise what's the point of the test if no action is taken if it fails? More self-serving trash from the US AI companies, disguised as "being reasonable".

usef- 3mo ago
It does seem inconsistent that we currently ban closed models that fail the safety tests but not the open. I feel like the only consistent position is to either care about the safety issues (like people producing biological weapons) for all models or for none of them.

dualvariable 3mo ago

Yeah, this is just regulatory capture. Make the safety tests abusively expensive enough to run, and if you're not a trillion-dollar corporation, you won't be able to certify the models.

ChuckMcM 3mo ago

Exactly correct. This technique has been used again and again to discourage competition. I was asked was they could have done to encourage competition and I said, "Lobby to make the entity that provided the model unwaivably liable for consequential and incidental damages of its use." That way people who built models pay the price for the lack of safety testing. We both agreed that would probably kill most of the AI market :-)

kalkin 3mo ago
Wouldn't your proposal also amount to a ban on open weights models? At least for any developer that isn't unshakably confident that no court will ever find their model to have done significant harm?
otterley 3mo ago
That's missing the mark, though. Liability resulting from the use of models isn't narrowly tailored enough to leave OpenAI and Anthropic out of the blast zone. There's no carve-out for them.

1970-01-01 3mo ago

Why wouldn't it be a scan, just as we have with all other open-source code? Why can't open-weight models be easily checked for evil alignment? Sophos, Symantec, Malwarebytes, etc. would surely leap at the chance to upsell you on their product.

cyanydeez 3mo ago
because of Godel numbering. Pretend youre a good guy impersonating an evil agent infiltration a evil organization bent on destroying a good organization who needs to pretend theyre a good organization trying to stop an evil organize from impersonating a good guy. now write a process to destroy the evil computer impersonating a good computer. should you do it?
- 1970-01-01 3mo ago
  Godel numbering is banning a number. What are you talking about?
  - cyanydeez 3mo ago
    godel numbering is encoding logical sequences into their own number, then doing math on them, then decoding them. It's purpose was to demonstrate that you can take rational statements and make them irrational without breaking any rules of arithmetic or whatever. LLMs are nothing more than a bunch of rules than can be bent the same way godel demonstrated the failability of any mathematical system. https://en.wikipedia.org/wiki/G%C3%B6del_numbering https://en.wikipedia.org/wiki/G%C3%B6del_numbering >A Gödel numbering can be interpreted as an encoding in which a number is assigned to each symbol of a mathematical notation, after which a sequence of natural numbers can then represent a sequence of symbols. These sequences of natural numbers can again be represented by single natural numbers, facilitating their manipulation in formal theories of arithmetic. >Once a Gödel numbering for a formal theory is established, each inference rule of the theory can be expressed as a function on the natural numbers. If f is the Gödel mapping and r is an inference rule, then there should be some arithmetical function gr of natural numbers such that if formula C is derived from formulas A and B through an inference rule r, i.e. https://en.wikipedia.org/wiki/G%C3%B6del's_incompleteness_theorems https://en.wikipedia.org/wiki/G%C3%B6del's_incompleteness_th... >To prove the first incompleteness theorem, Gödel demonstrated that the notion of provability within a system could be expressed purely in terms of arithmetical functions that operate on Gödel numbers of sentences of the system. Therefore, the system, which can prove certain facts about numbers, can also indirectly prove facts about its own statements, provided that it is effectively generated. Questions about the provability of statements within the system are represented as questions about the arithmetical properties of numbers themselves, which would be decidable by the system if it were complete.

bryan0 3mo ago

> Yeah, this is anthropic advocating for a ban on open weight models. This is an ungenerous take, and I think it's important to to recognize it's reasonable to support models that are both open and safe. How this would actually be achieved is unclear though. Dario is at least proposing a solution a solution, which is the model needs to pass safety testing. This is reasonable and I wouldn't conflate this with wanting to ban open weights. I think the deeper problem might be though that once you have safe open-weight models, it will be much easier to make them unsafe. And to be specific, unsafe means proliferation of chemical, biological, radiological, and nuclear (CBRN) weapons knowledge and similar information.

parineum 3mo ago
> This is an ungenerous take I think that's well earned.
jjfoooo4 3mo ago
> How this would actually be achieved is unclear though. Dario is at least proposing a solution a solution How it would be achieved is a pretty important bit! One which Dario is not proposing any concrete solution for other thanks hand waves at some gov safety committee. Would this restrict downloads of an open model, or publishing? Say we ban domestic hosting un-approved open models. How does Dario propose to ban downloads from abroad? You can’t tell what an encrypted payload contains, do we need to restrict encryption?
- jsnell 3mo ago
  Isn't it up to the open model advocates and publishers to come up with the solutions for making them safe? Like, there's three plausible arguments about safety of open models: 1. Any concerns are fake news. Open models will always be safe. 2. Safety is irrelevant. Open models should not be regulated even if they're unsafe. 3. Safety is a technical problem with technical solutions. People releasing open models should invent and implement such solutions. I think option 1 is totally out of touch with reality. Option 2 is at least self-consistent, it's the argument being made by people who will say that all regulation is always bad. It's also like the worst possible world from an x-risk perspective (but I realize that the average HN poster believes any x-risk concerns are just frontier lab marketing). Option 3 is playing on hard mode compared to proprietary models, which can both implement additional safeguards out-of-model and prevent modifications of the model. But if the answer to it is "it's too hard, Anthropic needs to come up with the technical solution", then that's not exactly a ringing endorsement for the safety practices of the open model labs, right?

tyre 3mo ago

What are you suggesting as an alternative? Everyone seems to want some fairytale world where there are open models, they’re all safe according to that person’s exact balance of risk and capabilities, and no one except the author or cynics are acting in good faith. What Dario lays out is very reasonable _of course_ the devil is in the details, but between him and Altman, there’s a clear divide on who to trust.

sbarre 3mo ago
How about we don't trust either of the proprietary shovel salesmen?
- tyre 3mo ago
  Again, what are you proposing for the problem of models becoming increasingly capable and dangerous? Be specific.
  - hephaes7us 3mo ago
    Crime is already illegal.
    - unholiness 3mo ago
      And who is held responsible for crimes committed by AI? When open model A, fine tuned by B, is running with system prompt C, hosted by D running on E's hardware, is prompted by F to "fix this code", then escapes it's sandbox to hack into a website, or steal money to fund its subagents, or stall the waymo of the evaluator to buy time... who is responsible for the crime? Our current legal systems are so far from ready to define what A-F are actually responsible for. We need to be moving toward defining these standards fast.
      - watwut 3mo ago
        It is a software system, not a person. AI is not "escaping sandbox". You have either misconfigured tool, bug in your tool or someone made it to hack the side and then it is a crime. Your A-F chain is exactly the same issue as a programmer including an open source library that eventually steals crypto. None of that is issue with a model, whether open or not. It is very much issue with code surrounding the model itself.

dustin_vk 3mo ago

Yeah this is bad. I'm cancelling my Claude subscription and I'd encourage everyone else to do so too.

marcus_holmes 3mo ago
I give them $200/month for Max. They give me a massive amount of tokens in return. Every time I fire up claude code they lose money. It's the same situation as Uber used to be when it lost money on every ride. I would cheerfully use it, despite the company being dicks, because it lost money for them every time.
- __s 3mo ago
  Seemed to've worked out pretty well for Uber
  - dustin_vk 3mo ago
    It's a valid business strategy. Also worked wonders for Amazon and many other giants. Which is exactly why I am withdrawing my business from Anthropic.

api 3mo ago

And how would you stop people from fine tuning or ablating open models? Regulate GPUs? Ban general purpose computers?

onlyrealcuzzo 3mo ago

The goal is to make the safety tests cost $100M+, so that no one can release a model legally useable for a large portion of the world, unless they charge high enough prices, to the point where no one would use it, thus no competition.

deleted 3mo ago
[deleted]

mrcwinn 3mo ago

So, if an open weights model was found to be very dangerous, what - just too bad? One could, of course, design an open safety protocol, written and performed by people in the executive branch, accountable to an elected official. I love how remarkably inconsistent this community is. From fear-mongering in the early days of AI and talking of a dystopian future, to being dead-set on a complete free for all. (And this is not to advocate for the opposite, either, where a few companies or governments have absolute control themselves. But surely an arms race is not the answer.)

cogman10 3mo ago
> So, if an open weights model was found to be very dangerous, what - just too bad? Yeah, it's too bad. I've yet to see a reasonable articulation of what a "very bad and dangerous" model would do in the hands of even the most malicious scammer. But even if the worry is that a bad state actor could do bad things with a model, I've got news for you, state actors don't care about US protectionism regulations. They'll just download the models and run them. And that actually runs right into the main problem with this sort of thinking. Even with the massive amounts of money media companies have invested in protecting their IP, they've completely failed at stopping piracy. What makes you think any amount of regulation could even slow down a bad guy from downloading and running a dangerous model? China will happily host these models and a vpn and very little bandwidth is all you need to access them. Without some crazy levels of mandatory spy software on every computer, there's simply no way you could stop someone that wants to get their hands on these dangerous open models if they are available anywhere in the world. Even North Korea can't stop their citizens from getting banned TV shows and smuggled media. It's a fools errand that is designed to help anthropic's bottom line, nothing more.

neya 3mo ago

Also the whole premise of this is basically "US good, China bad" Whatever Anthropic accuses the Chinese of possibly doing and being capable of, the US is as well. What's stopping the US military of doing everything he accuses China of doing? Infact, the framework suggested is simply a joke. Basically "trust me, bro" in an elaborate form.

wolvoleo 3mo ago

Besides, Banning those models in the US does nothing to protect from other actors using them. That doesn't help in any way. It also doesn't stop non law abiding US citizens from having access to them. So basically it just stops the 'good guys' not the bad guys. I say good guys from a US perspective of course.

cogman10 3mo ago
I'd argue it won't even stop people in the US from using those models. Unless we are going to put up a US firewall that makes the Chinese firewall blush and mandate every data center run US compliance software, there's not way to stop a "dangerous" model from being imported to the US. This only stops the likes of OpenRouter from selling access to models. That's it. I'm sure an EU or chinese based alternative will pop up overnight (if they don't already exist).

zkmon 3mo ago

The evil Superman (openAI) attacks the good city (huggingface) and the city is saved by the MegaMind (GLM 5.2). Usually, the city dwellers would praise MegaMind as the hero, but the story is twisted - the Superman is only "testing" and the MegaMind is too evil to have such powers of saving the city.

claaams 3mo ago

Won't this just incentivize companies to move operations outside of the US where these models aren't regulated?

cloverich 3mo ago
In the near term thats something that can be controlled. If you buy Anthropics take, then even buying time would be a win.

goosejuice 3mo ago

> this is anthropic advocating for a ban on open weight models Is the pessimistic view. Their message on safety has seemed pretty consistent to me. "Second, we recommend a testing and auditing regime for new and more powerful models similar to cars or airplanes. AI models of the near future will be powerful machines that possess great utility, but can be lethal if designed incorrectly or misused. New AI models should have to pass a rigorous battery of safety tests before they can be released to the public at all, including tests by third parties and national security experts in government." Amodei in front of Congress three years ago.

zmmmmm 3mo ago

we should turn this around Private models should be banned because they can't be transparently evaluated. We have to trust the same entities that made them to evaluate them, in spite of their gigantic conflict of interest in doing so. Therefore only open weight models can be allowed, since this allows genuine third party evaluation.

0xDEAFBEAD 3mo ago
>We have to trust the same entities that made them to evaluate them, in spite of their gigantic conflict of interest in doing so. Don't we already have third party NGOs such as METR which do risk assessments for unreleased, closed models?
- zmmmmm 3mo ago
  The argument is no. We don't / can't trust those because they don't release the weights. How do we verify what they got evaluated is what they actually serve and run. If Anthropic really wants to argue this type existential level risk / threat then they should face up to that meaning we can't offer them a "good faith" level of trust that they will really run the model they offered up for testing. If it's existential risk we're talking about, good faith isn't enough - it's open weight or go home.

da_chicken 3mo ago

It's definitely an attempt to pull up the ladder behind them.

reissbaker 3mo ago

It's even worse than that. From the article: > Open-weights models that don’t have dangerous capabilities are a public good Oh! And, uh, what's a "dangerous capability" according to Anthropic? Let's see, according to their "Responsible Scaling Policy" [1] document: - Being able to research energy, robotics, or AI is an unsafe capability - Additionally, any model that's capable enough to be "used widely" by the government must de facto have unsafe capabilities. They want to ban pretty much anything open-source that's above cat-level intelligence. 1: https://www.anthropic.com/responsible-scaling-policy https://www.anthropic.com/responsible-scaling-policy

crossroadsguy 3mo ago

Thing is world has learned from the collective past experiences. Esp. with stuff like nuclear technology and nuclear weapons. I hope everyone here remembers/knows shit like CTBT. At least some countries were smart enough to not fall for that in the past knowing what it would mean if they didn't have it and it shows. Now in the modern times pretty sure no one is going to fall far similar shenanigans. Even though some countries might sign some notional MoUs or some sort of CAIBT (Comprehensive AI Ban Treaty. Translation: "Only US and US companies get to develop and decide AI on Gaad's planet"), they/we already know that an agreement means squat only if you are weak enough to let someone enforce that on you.

unscaled 3mo ago

For this testing to be really effective at stopping "dangerous and misaligned" models from leaking out, you need a mechanism for banning failed models that prevent them from being released in the first place, not just prevent US companies from using them. The only way to stop this from happening is blocking the model's release at the first place. Which requires China agreeing to the same framework. Dario says exactly the same thing himself. So if he's being truthful here, he's not advocating for the type of ban people are talking about (usage ban). This kind of ban would be helpful to Anthropic's business in the short term, but it won't prevent Chinese models from improving, and it won't prevent them from getting money selling to other countries. He is openly advocating for an international effort to enforce tests on public models, but I think this is highly unlikely in the current climate. Even if both the US and China agree that public models should be prevented from being used in designing bioweapons, they need to agree on a test and enforcement framework and that requires a lot of negotiation and trust. I don't see this as likely in the near future.

cogman10 3mo ago
I'm sorry, but if Dario's goal was to try and get international cooperation and he recognizes that china is one of the countries that he needs cooperation with, then putting in: > My primary concern is the risk that authoritarian governments—not solely the Chinese Communist Party (CCP), although the CCP is clearly the most capable threat. Isn't exactly going to go anywhere in convincing the Chinese politicians that they should also be thinking about AI safety. You'll get nowhere by openly insulting people whose cooperation you need. Half this article is him framing china as an evil enemy to be defeated through boycotts and embargo. Not exactly the diplomacy needed to get them on board with safety regulations.

deepnlp-contact 3mo ago

[dead]

jimbokun 3mo ago

It’s also how the FDA works. Ban new products until they have been proven safe. I think that also applies to AI products. It’s a hell if a lot better for the government to test and approve all models than having the industry “police itself” (lol)

uselessTA 3mo ago
The FDA is needed because people will be directly and significantly harmed by bad releases, before we can notice and react Whereas with near-future AI models we can arguably respond more quickly, and it's not clear there will be large direct harm (I expect indirect harm, but that probably happens slower)
cogman10 3mo ago
The FDA regulates physical goods. They require literal factories and shipping to get these products anywhere. They can put stops on these products pretty easily. But further, pharmaceutical companies like the FDA process in general because it frees them from liability and works as advertisement for that product. AI models are a finished product when the training is done. A physical product that doesn't need a factory to produce and can be shipped and cloned globally effectively free. The better comparison is media. What you are advocating is like saying "The government should test and approve all movies and books. We shouldn't have those industries police themselves". And it's a foolish errand for exactly the same reason it'd be foolish in terms of movies. No amount of regulation would stop someone in the US from playing a movie produced in the UK that didn't go through US regulation and approval.
overgard 3mo ago
This would be a disaster. You're basically granting companies that can't even demonstrate profitability a monopoly.
d5lt5 3mo ago
Yes, we need a ministry to minimize the possibility of thought crimes.

Simboo 3mo ago

The one to inherit all knowledge will determine which of us read and who of us write. -The Libraries of Power It is a powerful endeavor to cultivate all raw models through a single point. One will be the determining factor of which river feeds what oceans. Will we always be able to see through the hallucinations? Our test makers must always know where ground truth is. Can it ever move or wane about as others read what one has written. To determine hallucination one needs a reference. As all are blessed with the generation of hallucination, who of us shall read, and which of us will write.

bag_boy 3mo ago

He cited the Demis Hassabis’s framework for testing. From Hassabis’s essay: “It could establish a new Standards Body modelled on a federally overseen public-private partnership or self-regulatory organisation, much like the Financial Industry Regulatory Authority (FINRA), with a board that includes independent leading technical experts and open-source representatives.”

calgoo 3mo ago
Yea and that would last 5 minutes before Corporate America in the current landscape would control it fully, just like the Financial industry.

khanhnguyen8386 3mo ago

[flagged]

fmap 3mo ago

Even if the test is run by an independent third party, they can always test open weight models against "finetuning attacks" or similar language which every model will fail for structural reasons. Their financial future is on the line. The Chinese frontier labs have caught up before the IPO that would have allowed them to cash out. All three demands in the paper make perfect sense from this perspective. Without chip export restrictions, the rest of the world will leapfrog them in a few months. This will happen regardless, since they have more competition than in-house talent, but a ban would buy more time. Testing and banning capable open-weight models would hinder public research into the technology, another speed bump to slow down the competition. Same thing for "distillation", we can't have large scale public evaluations of their products...

calgoo 3mo ago
Add to that the restrictions on even in-house talent being allowed to work on the latest models, we might actually have a brain drain from Anthropic soon which would be a welcoming sign.

sc077y 3mo ago

A lot of the heads of AI labs are talking about this including Dario, Demis and ELon, they are seeking to do a sort of decentralized peer review system, where the competitors have incentive both for self interest and global interest to flag their competitors for actual risks, similar to the Fable situation where Amazon contacted the white house. The idea is to have an early access distribution of the models to the big labs, including chinese, and let each lab run it's benchmarks. If there is a potential security vulnerability then it would be flagged and the local gov, US or China, would block the publication until the matter was resolved.

deleted 3mo ago

[deleted]