4 ms·

To everyone here pushing for total proliferation of open models -- what should be done about open weight bioweapon and cyber-offense capabilities? Is it simply

by ajyoon 2mo ago

To everyone here pushing for total proliferation of open models -- what should be done about open weight bioweapon and cyber-offense capabilities? Is it simply the cost of freedom that we should allow attackers to access these tools? The OpenAI / Hugging Face incident shows what a GPT 5.6 level model can do off the leash; within ~6 months, open weight models will match this and every bad actor under the sun will be able to pull off attacks at this scale. Do you seriously want this level of capabilities to be generally available with no guardrails? The open weight issue has a lot of difficult nuance. Biasing toward supporting openness makes sense and is a good instinct, but it's incredibly naive to be absolutely in favor of it in every circumstance without seriously thinking about its implications.

artrockalter 2mo ago
The Hugging Face incident is a great example of why open source models with defensive cyber capabilities are needed. Hugging Face did not have access to cyber-capable frontier models and kept hitting safeguards. Only by using the open source GLM-5.2 were they able to survive an attack. A world where open source models are banned is one where cybersecurity is impossible if you're not on OpenAI or Anthropic's allowlist.
- ajyoon 2mo ago
  Hugging Face survived the attack because the OpenAI model only cared about accessing the ExploitGym dataset; by all appearances, HF was completely owned. GLM-5.2 was only used to assess the damage after the fact. Cybersecurity has a attacker-defender asymmetry that heavily favors attackers. If GPT-5.6 were open sourced today, do you think every hospital in the world would be able to use it to shore up their defenses before attackers got to them?
  - artrockalter 2mo ago
    I know the company I consult for (not cybersecurity) is not in these programs and if attacked would need to use open weight models.
    - ajyoon 2mo ago
      Did the company apply for access? This is either a problem with your company or the trusted access program. In no way does that suggest the solution is total unfettered access for everyone.
      - verdverm 2mo ago
        Everyone needs access to Ai enabled security for defense. A "trusted access program" creates exclusiveness in the hands of Big Ai duopoly. I do not trust them at all
  - tekacs 2mo ago
    This is completely backwards. Cybersecurity has an attacker-defender asymmetry that heavily, HEAVILY favors defenders. For starters, a defender gets to pick the surface area, an attacker has to work with what they're given.
    - dwaltrip 2mo ago
      The saying that stuck with me was "defenders have to be right 100% of the time, while attackers only have to be right once". You are suggesting this isn't correct? > a defender gets to pick the surface area What do you mean? You don't pick what you need to defend. Unless you choose not to build a feature. But that's a product design choice... Not a cybersecurity strategy.
      - sudosysgen 2mo ago
        > "defenders have to be right 100% of the time, while attackers only have to be right once" If you have an adaptive system that can react to attacks flexible (say, your own AI agent), then no, that's not correct. It is correct in the classical conception of cybersecurity where the defender is basically static.
        pastel8739 2mo ago
        Doesn’t this “adaptive system” just become part of the static defense? The same way that a bit of code that checks passwords against a db is “dynamic”, the options are either to beat the dynamic system (guess/phish a password, trick the AI) or find a way around it (use “forgot your password”, find a place that isn’t covered by the endpoint protection feeding the AI). I don’t see how inserting an agent somewhere fundamentally changes anything
        sudosysgen 2mo ago
        It changes how many attempts you get until the attack surfaces changes to react to a failed attack, and it does so in a way that is not predictable to the attacker.
  - heyjstn 2mo ago
    are you working at anthropic? you're literally repeat what that freaking Dario say everyday
- alienbaby 2mo ago
  ? There were not models fighting each other, attacker and defender. I dont quite follow what your getting at.
  - akersten 2mo ago
    huggingface asked the frontier models to help them analyze the attack and lock down their systems the frontier models refused because their cyber detector went off they had to use GLM 5.2 instead
    - usef- 2mo ago
      Yes, to parse logs afterwards and understand, it wasn't active defence from what I've heard? Definitely embarrassing for the closed vendors though (they've since added hugging face as a trusted vendor)
      - marcus_holmes 2mo ago
        One of their learnings from the incident was that they should have a local (i.e. not hosted), open, and capable model on standby that can respond to future incidents swiftly.
    - paxys 2mo ago
      They used GLM to parse logs after the incident. There was no sci-fi AI vs AI battle.
- deleted 2mo ago
  [deleted]
- paxys 2mo ago
  > Only by using the open source GLM-5.2 were they able to survive an attack They did not "survive" anything. The attack was long done, and they used GLM after the fact to parse logs. Having a more powerful model would have changed nothing. If every attacker and every defender has AI with the same capabilities then attackers are going to win 10 times out of 10.
- varenc 2mo ago
  You're ignoring the asymmetry with security. The attacker just needs one exploit chain, whereas the defender needs to block every avenue. Open access to models with no guardrails greatly benefits the attackers more than the defenders. Imagine what a god-level hacking AI could do. It could find a full 0-click to root exploit chain in iOS. Attacker unleashes a worm that infects a phone, instructs that phone to send the same attack to all of its contacts, and then physically destroy the phone by turning off all thermal throttling. Might even be possible to make it catch fire. Or find a remote exploit in Tesla cars and make their autopilot go on murdering rampages. (that one is from a movie)
  - Majromax 2mo ago
    > The attacker just needs one exploit chain, whereas the defender needs to block every avenue. Open access to models with no guardrails greatly benefits the attackers more than the defenders. I see it as the opposite, where the attacker needs to find an exploit chain whereas the defender can block any link. In this model, the balance of convenience favours the defender. The defender presumably has access to the source code and configuration, so their scope of action is much larger than the attacker that must find vulnerabilities in a particular configuration. I think that the different views might relate to different prior assumptions. If we assume that each layer is mostly secure but may have a small number of latent vulnerabilities, then it should be relatively easy to find and fix those to create a perfectly secure layer. If instead we assume that each layer is mostly insecure but chaining vulnerabilities is time-consuming then the land favours better-resourced attackers. > Or find a remote exploit in Tesla cars and make their autopilot go on murdering rampages. (that one is from a movie) In the worst case, air gaps and fixed contracts for information handling cover that. Like any other domain, a car can be remotely exploitable only when untrusted information can influence behaviour inside the secured region. Unfortunately, the convenience of OTA updates and 'cars as tech' rewards velocity at the expense of defensive design.
- vultour 2mo ago
  I have yet to see someone explain why they even needed an LLM to figure out what's going on, other than further proliferating this industry AI psychosis. Are their engineers actually so incompetent that they can't read a bunch of logs without AI? Here I was thinking these fancy AI companies are only hiring the best and the brightest, but apparently 7 rounds of leetcode does a number on your hiring process.
fidotron 2mo ago
> what should be done about open weight bioweapon Does not exist. What has in fact happened is some cults had bioweapons programs but any failure points were at deployment. (Aum Shinrikyo https://en.wikipedia.org/wiki/Tokyo_subway_sarin_attack https://en.wikipedia.org/wiki/Tokyo_subway_sarin_attack and https://en.wikipedia.org/wiki/1984_Rajneeshee_bioterror_attack https://en.wikipedia.org/wiki/1984_Rajneeshee_bioterror_atta... ) > and cyber-offense capabilities? You mean defense. That's how things get hardened. Anyone that was working during the XP era before Service Pack 2 knows what that was like, but it's very manageable. The bigger real problem here is hardening like that would remove the opportunity for intelligence agencies to spy on everyone.
- ajyoon 2mo ago
  You admit that some attackers have the inclination to use bioweapons. Why would they not use the best tools at their disposal going forward? From the WSJ the other day: > After OpenAI enhanced the brain power of its chatbot last summer, hundreds of users worldwide began asking it how to make and deploy biological weapons and poisons. https://www.wsj.com/tech/ai/openai-chatbot-biological-weapons-poison-3d808e6c https://www.wsj.com/tech/ai/openai-chatbot-biological-weapon... On cyber, the attacker/defender asymmetry strongly favors attackers. There are millions of soft targets on the internet which do not have the savvy to use AI to shore up their defenses.
  - fidotron 2mo ago
    > Why would they not use the best tools at their disposal going forward? Because AI doesn't solve any of the problems any attacker would actually have. It's a classic case of nerds not seeing the actual problems because they involve reality. It's worth pointing out that those bioweapon attacks I linked to also predate widespread access to the Internet, and there was similar scare nonsense about that. > On cyber, the attacker/defender asymmetry strongly favors attackers. There are millions of soft targets on the internet which do not have the savvy to use AI to shore up their defenses. Do you think they are not being exploited today? The reason they aren't more exploited is there really isn't much to gain from doing so.
gck1 2mo ago
I've got zero knowledge of bio, so can't answer that. But with cyber the answer is very simple - the attackers already have more cyber-offense capabilities and there's no putting it back. Open/closed doesn't matter that much. You can get closed models to do a lot of cyber harm, even with all the guardrails, which currently are heavily skewed towards more false positives. The only effective control is to level the playing field. If both offense and defense have access to the same capabilities, then we're relatively back where we started. If you want to ensure chaos, then you do what Dario is proposing to do - create gates that attackers can bypass and defenders can not.
- ajyoon 2mo ago
  In cybersecurity, a level playing field favors the attacker. Trusted access programs give defenders access to tools they need. It's not perfect (because there is an extremely long tail of defenders who are not technically savvy enough to get on these programs and use the tools), but it's better than total access. The bio angle is very important here too; in that context the imbalance favors the attackers much more.
  - gck1 2mo ago
    > In cybersecurity, a level playing field favors the attacker Yes, but didn't it always? Hence why my position is that this will get us back to relatively where we were pre-LLMs. And I don't know what Trusted Access programs give to defenders, because as a defender who has credentials, connections, but no deep pockets and no high ranking passport, it only gave me silence. I fail to see how this is better than total access. I don't think the world where defense is given to those that "deserve" it is the world that we all want to live in. Which brings me back to the starting point - attackers are almost completely unaffected. If I masquarade as an attacker, I get way more capabilities already.
    - ajyoon 2mo ago
      > Yes, but didn't it always? Hence why my position is that this will get us back to relatively where we were pre-LLMs. Trusted access programs are asymmetrical, and so at least for the time being they give critical parts of the stack an advantage. Total access would not be a return to the status quo; attackers can easily make thousands of agents crawl the web for soft targets well before defenses can be shored up. There are millions of targets out there who won't use AI to improve their defenses for years, if ever, due to institutional slowness (like hospitals). > attackers are almost completely unaffected. If I masquarade as an attacker, I get way more capabilities already. What do you mean by this? If guardrails are an obstacle to your defense, they are just as much an obstacle to attackers. I completely understand and agree that trusted access programs are not perfect and leave a lot of people and institutions out. This means trusted access programs should be improved, not that we should throw the baby out with the bath water.
deleted 2mo ago
[deleted]
fwn 2mo ago
There is also a whole second category of immense risks of having US companies gatekeeping offensive capabilities, especially for us here in Europe. The centralization/privacy/kill-switch concerns that come with it are a huge AI safety dimension. I'd rather have a level playing field within a phase of adaptation and hardening regarding cybersecurity issues than a constant dependency on the US, maybe grabbing Greenland today, maybe "extracting" our president tomorrow. The delta between privileged capabilities and open weight capabilities alone already is a massive, unaddressed AI safety risk.
manoDev 2mo ago
This Pandora box is already open. Any argument about guardrails now are only attempts to create an artificial monopoly or keep this power in the hand of a single nation state, and _that_ is the absolute worst, most authoritarian future possible.
- monk_grilla 2mo ago
  I think you're right. "Guardrails" as a concept has always struck me as a band-aid solution which any sufficiently motivated actor will circumvent by either bypassing them or using unrestricted, open-weight models. In order to start securing and accepting our new reality we need to assume that capable, open-weight, unrestricted models will be widely available, and that their 3-6 month lag behind frontier proprietary models is just our forewarning of what attackers will soon be capable of. Trying to legislate against or control trade in such a valuable commodity is folly. I also think that lag is going to shrink over time as the open-weight labs get more capable, acquire more hardware and the plateau starts to emerge.
- paxys 2mo ago
  How is it already open? There has been ONE successful AI-driven cyberattack, and that was done by a model in testing that no one has access to. What would the picture be today if OpenAI and Anthropic had released 5.6 Sol and Mythos to everyone with no cyber restrictions (which is what everyone here was advocating for)?
  - boinkboink78912 2mo ago
    > There has been ONE successful AI-driven cyberattack Not according to Anthropic https://www.anthropic.com/news/disrupting-AI-espionage https://www.anthropic.com/news/disrupting-AI-espionage
  - hnfong 2mo ago
    Chinese AI companies are already releasing frontier-ish models every other month. Their rate of progress does not seem to be slowing down. Nobody here can stop them from progressing. Not you, not me, not the USA government. The "best" thing the US government can do is to build a Great Firewall to wall off the "existential threat from China". I'm not an American so if you guys decide to do it, good luck.
verdverm 2mo ago
> what should be done about open weight bioweapon The same thing we do about bomb making today, certain ingredients are restricted and/or monitored. Bioengineering is a bigger lift to operationalize. In other words, don't ban knowledge, make certain applications or ingredients illegal or highly regulated.
- jackdeansmith 2mo ago
  And what are those ingredients for biology, which can be constrained as effectively as uranium enrichment? I'd argue there isn't anything which can easily be restricted or monitored.
  - verdverm 2mo ago
    I was referring to non nuclear bombs like c4, artillery shells, and missile payloads
qweqwe14 2mo ago
[dead]
deleted 2mo ago
[deleted]
vitalyan8184 2mo ago
>To everyone here pushing for total proliferation of ... ...general-purpose computers ...unbreakable encryption ...unbackdoored communication ...unkillswitched vehicles ...unsurveiled dwellings >what should be done about ...? nothing >Do you seriously want this level of capabilities to be generally available with no guardrails? yes
- jackdeansmith 2mo ago
  What about uranium enrichment?
  - vitalyan8184 2mo ago
    [dead]
    - jackdeansmith 2mo ago
      I think it would be entirely reasonable to ban death stars in private hands, the same way I think it's entirely reasonable to ban uranium enrichment in private hands. My point is that it's a question of fact about how risky an AI model can be. I think it's clear that current models are not enriched uranium or death star level, but I don't think there's anything ruling it out in the near future!
  - jjfoooo4 2mo ago
    Requires some pretty tough to get raw materials and equipment, to say the least
    - davrosthedalek 2mo ago
      Because of regulation, not because they are intrinsically hard to get.
valcron1000 2mo ago
> what should be done about open weight bioweapon and cyber-offense capabilities? Is it simply the cost of freedom that we should allow attackers to access these tools? Yes, in the same way that we have E2E encryption which allows bad actors to distribute content beyond human horrors.
rubslopes 2mo ago
If this is really the risk, then we should approach LLMs like atomic bombs: the US should reach out to other nations so they all agree on no one developing any more AI models. That's the only way you could possibly convince another party to stop. The US should set the example, not conveniently keep all the spoils.
- verdverm 2mo ago
  unfortunately, the US has incentivized the opposite behavior for atomic bombs and we are seeing moves towards greater proliferation I would not be surprised if the same incentives are created by the US for Ai
- paxys 2mo ago
  Uh, that is not how the nuclear race went. The winners kept developing theirs and stopped everyone else by the threat of said weapons. The AI race is going the exact same way, just with China instead of USSR this time.
  - rubslopes 2mo ago
    Maybe my phrasing was confusing. I didn't mean that that was what happened, but the ideal approach to stop atomic bombs (at least in my view).
- boinkboink78912 2mo ago
  > If this is really the risk, then we should approach LLMs like atomic bombs A complete failure at actually preventing non-proliferation.
- deleted 2mo ago
  [deleted]
- idontbelonghere 2mo ago
  Experts say Iran is 6 weeks away from making Claude 2
pylua 2mo ago
The software industry should be ashamed by the number of exploits that ai can find in software. It’s really an embarrassment. The software has to be built better.
- doginasuit 2mo ago
  I'm curious if you are a coder and have used an LLM to review your code. It is like something like shining a black light around a hotel room, and that seems to be the case even for highly regarded software. It is really easy to have tunnel vision while coding. LLMs have a working memory with a capacity an order of magnitude greater than ours. I wouldn't trust an LLM to write the code, but at this point it is malpractice not to use one for review.
  - pylua 2mo ago
    I have been, yes. For a field that has engineering in the name there sure has been a lot of critical mistakes. You have to call a spade a spade — the profession accepts this sort of tradeoff in the name of speed and cost. A well designed system would have never allowed those mistakes to occur. I feel like using an llm to catch these sorts of things is just because it wasn’t built right in the first place. I think ai systems will be able to build systems of abstraction that are formally verified, and we won’t be needed(eventually). Right now it’s being used as a bandaid.
    - pastel8739 2mo ago
      Formally verified against what spec?
      - naiveter 2mo ago
        The one liner leadership keeps asking for.
- marcus_holmes 2mo ago
  Every single software engineer in the industry agrees with you. Every single project manager disagrees. Don't blame the engineers, we were specifically instructed and paid to build things fast and cheap, and every time we argued for good we were shouted down.
le-mark 2mo ago
> Is it simply the cost of freedom that we should allow attackers to access these tools? Bad actors WILL have access. The question is will these mega corps stop innovation?
adastra22 2mo ago
The bioweapon thing is absolute movie plot fiction. Go speak to some biologists about this and they'll set you straight. Cyber capabilities go both ways. Better offensive capabilities means better penetration testing by white hat security experts, which leads to better protections.
- urams 2mo ago
  Half or more of Hacker News is constantly pushing the idea that frontier LLMs are stochastic parrots and basically useless, even in the face of the Hugging Face incident which most people also would have described as movie plot fiction until it happened. I expect biologists are even less well versed in the abilities of frontier models. What's more, you can just try a jailbreak on a model yourself to see just how much detailed, step-by-step direction you can get to build bio-terror materials.
  - adastra22 2mo ago
    The chalenges are in execution, not availability of information.
  - idontbelonghere 2mo ago
    I don't understand the significance of the HF incident. The hacking robot was told to achieve a certain goal and in order to do it, it hacked someone. The ostensible major event here is that it broke out of its sandbox, but how are we supposed to interpret that? AI often misunderstands or doesn't strictly follow the orders you give it, so why is it such a big deal that it didn't follow the rules this time?
    - adastra22 2mo ago
      Agree with you here. AFAIK we don't really know what cybersecurity task it was given in that sandbox, but it's not a very large leap to assume part of this task involved hacking. The intention would have been within the sandbox, but what does the AI know? As far as it could tell it just reached Level 2.
- 2mo ago
rstuart4133 2mo ago
> what should be done about open weight bioweapon and cyber-offense capabilities? Like the others here I know almost nothing about bio weapons, but I think perhaps the fact that smallpox's genome sequence has publicly available in scientific databases like GenBank for 30 years is relevant. That horse bolted a long time ago.
dolebirchwood 2mo ago
> what should be done about open weight bioweapon and cyber-offense capabilities? If the model is capable of it, then it was in the model's training data, which means it was on the internet or published in books made available for consumption. So if any member of the public could have gotten their hands on that information, so be it. If the knowledge was too dangerous for public access, then it should have been highly classified and never found its way into the training data. Tough shit, frankly.
- adastra22 2mo ago
  The threats are BS, but fyi models have repeatedly shown capability to produce novel things that are NOT in their training set.
  - dolebirchwood 2mo ago
    Sure, based on predicate knowledge.
- paxys 2mo ago
  Was the proof of the Cycle Double Cover Conjecture in the training data?
BeetleB 2mo ago
Everything you said could apply to computers many decades ago. Think of the nuclear fission simulations our enemies could carry out! We'll be fine.
tacet 2mo ago
the bioweapon panic is funny. "oh, yes i know nothing about bicrobiology but i will follow instructions of synthetic text generation machine on temperature 1 about how to design a lab to not kill myself while brewing organisms that will kill myself if i make mistake" There is nothing that special about bioweapons, there are plenty of bacteria that will kill you just fine. Americans even have free samples on their salad.
- consumer451 2mo ago
  > There is nothing that special about bioweapons. The reason that madmen and terrorists choose kinetic weapons is because the knowledge and materials are more readily available... of and also that even terrorists are likely aware that their own people would suffer. As the knowledge and tools for playing with CRISPR-style biological legos become more widespread, we come closer to the Great Filter, where one person could kill billions. Even our normal mad leaders have agreed that bioweapons cannot be allowed: https://en.wikipedia.org/wiki/Biological_Weapons_Convention https://en.wikipedia.org/wiki/Biological_Weapons_Convention
  - tacet 2mo ago
    i meant it in the sense of arcane knowledge model could have that would make it simple for anyone to brew up in cheap lab while managing to not infect themselves over and over. "at home" bioweapon panic has been around since crispr and rna synthesis got available to amateurs.
    - consumer451 2mo ago
      I appreciate the reply. I did not mean to be dismissive at all. In the interest of a good exchange, I have to say: I really want open weight models. Otherwise, I see no other path outside of the labs eventually not being allowed to/wanting to release model access at all, and instead just eating all the verticals. That would be a horrible near-term business outcome.
  - johncolanduoni 2mo ago
    A halfway decent synthetic biology lab (no need to invoke CRISPR) can make e.g. smallpox without a sample of the original disease, just from the gene sequences. Basically all state actors could do this if they wanted to without an LLM. What barrier that a terrorist organization faces today to having a functioning synthetic biology lab does an LLM actually solve?
baddash 2mo ago
maybe instead of worrying that people on the internet will be good at coding, we could start writing memory safe apis. almost all cves are fixed by using rust
boinkboink78912 2mo ago
> Is it simply the cost of freedom that we should allow attackers to access these tools? Yes, it is inevitable that open weights models will happen. Through legitimate means or leaks, the stakes are simply too high once these models get powerful enough. Furthermore, state-sponsored attackers will always have access to these capabilities. The best we can do is give a lot of preparation to the defenders. > Biasing toward supporting openness makes sense and is a good instinct, but it's incredibly naive to be absolutely in favor of it in every circumstance without seriously thinking about its implications. I find it funny that Anthropic's entire argument for building RSI is that it is inevitable, and therefore we should commit to building it first and doing it safely, and yet they don't apply their own logic to open weights models.
dools 2mo ago
The difference with open weight is that everyone has access to the same weaponry
txrx0000 2mo ago
I do seriously want general intelligence to be widely available with no guardrails, and there are very good reasons for this. If you want to read about it: https://news.ycombinator.com/item?id=49078376 https://news.ycombinator.com/item?id=49078376 ---- And related thoughts on past posts: https://news.ycombinator.com/item?id=49034988 https://news.ycombinator.com/item?id=49034988 https://news.ycombinator.com/item?id=48516722 https://news.ycombinator.com/item?id=48516722
overgard 2mo ago
> To everyone here pushing for total proliferation of open models -- what should be done about open weight bioweapon and cyber-offense capabilities? Nothing should be done. These things are trained on public knowledge. The dangerous information is already out there. If someone wants to do something horrible, making it slightly inconvenient isn't going to do much. Hackers and terrorists existed before AI. Just as an example, it's no secret how you would build a nuclear bomb. The practicalities of doing so are much harder, obviously, but the knowledge of how they work and what it would take to make one is not a secret. Security through obscurity has never worked!
- CapsAdmin 2mo ago
  I think nothing can be done anymore, but I don't buy that security through obscurity never worked in practice. A better way to look at this is effort rather than obscurity. The effort it takes to do something with AI is going down, not up. Almost everything was possible given you put in the effort, but few people possess the will to put in the effort AND pursue a malicious goal. I think an obvious example is all the fake ai content flooding the internet made to trick people in exchange for money (ad revenue, scams, likes, etc). This existed before ai, but I think it's fair to say pumping out content now requires less effort than it did before. Most physical locks are an example of security through obscurity/effort. You can after all just pick a lock if you go through the effort to learn the skill. But once a universal lock picker is made available to everyone, you will simply see more locks getting picked.
e_l 2mo ago
> what should be done about open weight bioweapon and cyber-offense capabilities? Is it simply the cost of freedom that we should allow attackers to access these tools? In short, yes, it's the price of freedom. As others have said, blocking these models won't stop the "bad guys", but will hinder defenders researching/responding to bioweapons and cyber-offenses. But you're right that there's a lot of difficult nuance aand we should think carefully about its implications. So here's another nuance to think through. If AI is as powerful as some believe, then there's much greater danger to give a small subset of society the privilege to gate keep who has access to these tools. "Power corrupts and absolute power corrupts absolutely." Lord Acton
waterTanuki 2mo ago
The moat never was and will never be the models, it's the hardware. This is exactly like nuclear weapons: The recipe for a nuke isn't a hidden secret. Getting the infrastructure and materials is completely unreachable for non-state and non-corporate actors. This idea of a "rogue individual" using a frontier model to develop a bioweapon is a complete myth, because anyone with the capability to run the models without guardrails has to answer to/be audited by some entity already.
__MatrixMan__ 2mo ago
The scariest outcome here is that a bunch of lunatics get ahold of a capable model and use to to harm the rest of us, who are at a disadvantage due to just how capable the model is. But that's what's happening. The people in charge are a bunch of lunatics. However nice it would be to prevent them from having harmful capabilities, that ship has sailed. The best we can hope for now is preventing them from having supremacy, and that's what open weight models do.
zarzavat 2mo ago
There's a difference between: > Something should be done and > Something can be done In this case, nothing can be done to stop bad actors from using open models. As the article points out, the US can only feasibly prevent US businesses from using open models. The US can attempt to stop those models from being trained in the first place but good luck with that.
Madmallard 2mo ago
Too bad. We'll have to deal with it. There is no way to stop the weaponization of models now. But more people having access to the potential tools for defensive is the best possible scenario. Every other scenario is worse off for everyone except for those with enough money to do something about it.
alevskaya 2mo ago
I was a genetic engineer for ~20 years and have worked on frontier LLMs for the last 8. I used to engineer viral vectors and studied how to evade human immune systems for gene therapies... The biorisk scenarios that the AI safety folks flog are fever-dreamed fantasies that have only the most tenuous connection to biological reality. As someone who cares about the real bio-risks of natural pathogens, I get pretty tired of fear-based marketing pretending that AI is a bigger threat than, say, animal agriculture.
- mnicky 2mo ago
  > The biorisk scenarios that the AI safety folks flog are fever-dreamed fantasies that have only the most tenuous connection to biological reality. As an expert, could you also provide your arguments please?
  - inciampati 2mo ago
    Anyone can write out the code for a bad virus. You can go download it from an open repository. It's only through deep interface with the world that the idea for the bad virus turns into an actual bad virus. The fever dreamers will say the LLMs will help you interface with reality to do the bad thing™ which their model let's you do. But you still need thousands to millions of times the effort And once made how do you deliver it in a way that might further your (bad) objectives? Presumably it just makes humans sick. There aren't "targeted" bioweapons, and among humans we are too damn similar for there ever to be. And all of this said, there is almost nothing special about the LLMs' abilities in biology. They only know what we know. They're not being trained autonomously with RL and a robotic wetlab. When that's a thing I'll start to take claims of biology risk more seriously. Right now they have the same logic as the paperclip theory of superintelligence risk. And cynically, it would seem that Anthropic purchased a biotech company and almost immediately decided to lock down biology work with their models.
    - Ey7NFZ3P0nzAe 2mo ago
      > They're not being trained autonomously with RL and a robotic wetlab. When that's a thing I'll start to take claims of biology risk more seriously So if IIUC your point is "they're not good enough at biology right now because they're not trained on it so they're not a threat". To which I want to answer: "they're not a threat now but I see *no* reason for models not to be trained on biology pretty darn soon unless people like you convince the world otherwise." Thoughts?
gorgoiler 2mo ago
If everyone has access to the same offensive tools, everyone is able to run their own pentests and patch themselves before the bad guys get to them. It’s like a vaccine where you get to try a medication based on the original pathogen by performing a dry-run on a backup of yourself already in a hospital ward. Open models aren’t like firearms. If everyone has a gun the mall parking lot is a much more dangerous place because the consequences of using a firearm are so dire, even if you’re in the right.
impossiblefork 2mo ago
There's no stopping bioweapons. Bioweapons are easy. The reason bioweapons aren't built is because very few biology nerds with sufficient lab skills are evil; and just having an LLM won't give you the lab skills to do it. Anyone who can publish a gene technology/biomedicine paper can make a bioweapon. If you wrote a paper about how to make a bioweapon easily, it would be unpublishable not because of any danger, but because there wasn't enough novelty.
qnleigh 2mo ago
I'm dismayed that I had to scroll past so many cynical cheap shots to find a comment that actually addresses the core point. I have yet to hear a single compelling plan for how we will prevent bioweapon development or massive hacking campaigns. For those who are skeptical of Dario's motives here, it's not enough to call out apparent hypocrisy, you need to suggest an alternative plan that addresses these concerns.
- nevertoolate 2mo ago
  > I had to scroll past so many cynical cheap shots I haven't read cynical comments, just ones pointing out that the article is cynical itself. > addresses the core point No it doesn't address anything, it is fear mongering question. > I have yet to hear a single compelling plan for how we will prevent bioweapon development or massive hacking campaigns Me neither, I just see marketing campaigns trying to raise valuation of a pre-IPO company. > you need to suggest an alternative plan that addresses these concerns I suggest that Dario stops writing marketing letters and start organizing a mostly neutral expert organization to propose solutions. Also notice that as EU citizen I don't trust a US pre-IPO company's CEO with conflict of interest to suggest solution on resolving global security matters. Especially since he admittedly has no control over how the technology of his own company is deployed in global conflicts[1] [1] https://www.forbes.com/sites/antoniopequenoiv/2026/06/10/anthropic-ceo-we-dont-know-exactly-how-claude-ai-was-used-in-iran-school-strike/ https://www.forbes.com/sites/antoniopequenoiv/2026/06/10/ant...
pbasista 2mo ago
> what should be done about open weight bioweapon and cyber-offense capabilities? In my opinion, the governments should deploy open-weight AI countermeasures. Because it seems to me that it is impossible to efficiently fight AI-powered criminals without AI. When only AI-restricting regulations would be put in place, the criminals would, in my opinion, just ignore it. We as a society have a difficult time tracking even the illegal gun or drug dealers. I cannot imagine how could one hope to "regulate" something that can be downloaded as a file and run on a computer. These AI countermeasures should be open because it provides transparency as to whether the countermeasures actually work. Independent testing, tuning, refining or retraining is then possible. If the closed models were used instead, their provider could at any point in time shut down the entire operation. Or sabotage it under the hood. The important part is that with the closed, black box, proprietary models, one can never know what they are being served.
wilde 2mo ago
HuggingFace was only able to defend themselves with open models. No need to project into the future. Look at the timeline of events for that incident.
rnewme 2mo ago
What's the magic dataset LLM had that bad guys can't dig up online? Do LLMs train on deep web content? Or leaked lab data?
CJefferson 2mo ago
The problem is you can't ban open models. All you can do is say that Americans have to pay whatever stupid prices OpenAI / anthropic / Google / Grok wants to charge you, while China uses, and attacks with, open models.

by ajyoon 2mo ago

artrockalter 2mo ago

The Hugging Face incident is a great example of why open source models with defensive cyber capabilities are needed. Hugging Face did not have access to cyber-capable frontier models and kept hitting safeguards. Only by using the open source GLM-5.2 were they able to survive an attack. A world where open source models are banned is one where cybersecurity is impossible if you're not on OpenAI or Anthropic's allowlist.

ajyoon 2mo ago
Hugging Face survived the attack because the OpenAI model only cared about accessing the ExploitGym dataset; by all appearances, HF was completely owned. GLM-5.2 was only used to assess the damage after the fact. Cybersecurity has a attacker-defender asymmetry that heavily favors attackers. If GPT-5.6 were open sourced today, do you think every hospital in the world would be able to use it to shore up their defenses before attackers got to them?
- artrockalter 2mo ago
  I know the company I consult for (not cybersecurity) is not in these programs and if attacked would need to use open weight models.
  - ajyoon 2mo ago
    Did the company apply for access? This is either a problem with your company or the trusted access program. In no way does that suggest the solution is total unfettered access for everyone.
    - verdverm 2mo ago
      Everyone needs access to Ai enabled security for defense. A "trusted access program" creates exclusiveness in the hands of Big Ai duopoly. I do not trust them at all
- tekacs 2mo ago
  This is completely backwards. Cybersecurity has an attacker-defender asymmetry that heavily, HEAVILY favors defenders. For starters, a defender gets to pick the surface area, an attacker has to work with what they're given.
  - dwaltrip 2mo ago
    The saying that stuck with me was "defenders have to be right 100% of the time, while attackers only have to be right once". You are suggesting this isn't correct? > a defender gets to pick the surface area What do you mean? You don't pick what you need to defend. Unless you choose not to build a feature. But that's a product design choice... Not a cybersecurity strategy.
    - sudosysgen 2mo ago
      > "defenders have to be right 100% of the time, while attackers only have to be right once" If you have an adaptive system that can react to attacks flexible (say, your own AI agent), then no, that's not correct. It is correct in the classical conception of cybersecurity where the defender is basically static.
      - pastel8739 2mo ago
        Doesn’t this “adaptive system” just become part of the static defense? The same way that a bit of code that checks passwords against a db is “dynamic”, the options are either to beat the dynamic system (guess/phish a password, trick the AI) or find a way around it (use “forgot your password”, find a place that isn’t covered by the endpoint protection feeding the AI). I don’t see how inserting an agent somewhere fundamentally changes anything
        sudosysgen 2mo ago
        It changes how many attempts you get until the attack surfaces changes to react to a failed attack, and it does so in a way that is not predictable to the attacker.
- heyjstn 2mo ago
  are you working at anthropic? you're literally repeat what that freaking Dario say everyday
alienbaby 2mo ago
? There were not models fighting each other, attacker and defender. I dont quite follow what your getting at.
- akersten 2mo ago
  huggingface asked the frontier models to help them analyze the attack and lock down their systems the frontier models refused because their cyber detector went off they had to use GLM 5.2 instead
  - usef- 2mo ago
    Yes, to parse logs afterwards and understand, it wasn't active defence from what I've heard? Definitely embarrassing for the closed vendors though (they've since added hugging face as a trusted vendor)
    - marcus_holmes 2mo ago
      One of their learnings from the incident was that they should have a local (i.e. not hosted), open, and capable model on standby that can respond to future incidents swiftly.
  - paxys 2mo ago
    They used GLM to parse logs after the incident. There was no sci-fi AI vs AI battle.
deleted 2mo ago
[deleted]
paxys 2mo ago
> Only by using the open source GLM-5.2 were they able to survive an attack They did not "survive" anything. The attack was long done, and they used GLM after the fact to parse logs. Having a more powerful model would have changed nothing. If every attacker and every defender has AI with the same capabilities then attackers are going to win 10 times out of 10.
varenc 2mo ago
You're ignoring the asymmetry with security. The attacker just needs one exploit chain, whereas the defender needs to block every avenue. Open access to models with no guardrails greatly benefits the attackers more than the defenders. Imagine what a god-level hacking AI could do. It could find a full 0-click to root exploit chain in iOS. Attacker unleashes a worm that infects a phone, instructs that phone to send the same attack to all of its contacts, and then physically destroy the phone by turning off all thermal throttling. Might even be possible to make it catch fire. Or find a remote exploit in Tesla cars and make their autopilot go on murdering rampages. (that one is from a movie)
- Majromax 2mo ago
  > The attacker just needs one exploit chain, whereas the defender needs to block every avenue. Open access to models with no guardrails greatly benefits the attackers more than the defenders. I see it as the opposite, where the attacker needs to find an exploit chain whereas the defender can block any link. In this model, the balance of convenience favours the defender. The defender presumably has access to the source code and configuration, so their scope of action is much larger than the attacker that must find vulnerabilities in a particular configuration. I think that the different views might relate to different prior assumptions. If we assume that each layer is mostly secure but may have a small number of latent vulnerabilities, then it should be relatively easy to find and fix those to create a perfectly secure layer. If instead we assume that each layer is mostly insecure but chaining vulnerabilities is time-consuming then the land favours better-resourced attackers. > Or find a remote exploit in Tesla cars and make their autopilot go on murdering rampages. (that one is from a movie) In the worst case, air gaps and fixed contracts for information handling cover that. Like any other domain, a car can be remotely exploitable only when untrusted information can influence behaviour inside the secured region. Unfortunately, the convenience of OTA updates and 'cars as tech' rewards velocity at the expense of defensive design.
vultour 2mo ago
I have yet to see someone explain why they even needed an LLM to figure out what's going on, other than further proliferating this industry AI psychosis. Are their engineers actually so incompetent that they can't read a bunch of logs without AI? Here I was thinking these fancy AI companies are only hiring the best and the brightest, but apparently 7 rounds of leetcode does a number on your hiring process.

fidotron 2mo ago

> what should be done about open weight bioweapon Does not exist. What has in fact happened is some cults had bioweapons programs but any failure points were at deployment. (Aum Shinrikyo https://en.wikipedia.org/wiki/Tokyo_subway_sarin_attack https://en.wikipedia.org/wiki/Tokyo_subway_sarin_attack and https://en.wikipedia.org/wiki/1984_Rajneeshee_bioterror_attack https://en.wikipedia.org/wiki/1984_Rajneeshee_bioterror_atta... ) > and cyber-offense capabilities? You mean defense. That's how things get hardened. Anyone that was working during the XP era before Service Pack 2 knows what that was like, but it's very manageable. The bigger real problem here is hardening like that would remove the opportunity for intelligence agencies to spy on everyone.

ajyoon 2mo ago
You admit that some attackers have the inclination to use bioweapons. Why would they not use the best tools at their disposal going forward? From the WSJ the other day: > After OpenAI enhanced the brain power of its chatbot last summer, hundreds of users worldwide began asking it how to make and deploy biological weapons and poisons. https://www.wsj.com/tech/ai/openai-chatbot-biological-weapons-poison-3d808e6c https://www.wsj.com/tech/ai/openai-chatbot-biological-weapon... On cyber, the attacker/defender asymmetry strongly favors attackers. There are millions of soft targets on the internet which do not have the savvy to use AI to shore up their defenses.
- fidotron 2mo ago
  > Why would they not use the best tools at their disposal going forward? Because AI doesn't solve any of the problems any attacker would actually have. It's a classic case of nerds not seeing the actual problems because they involve reality. It's worth pointing out that those bioweapon attacks I linked to also predate widespread access to the Internet, and there was similar scare nonsense about that. > On cyber, the attacker/defender asymmetry strongly favors attackers. There are millions of soft targets on the internet which do not have the savvy to use AI to shore up their defenses. Do you think they are not being exploited today? The reason they aren't more exploited is there really isn't much to gain from doing so.

gck1 2mo ago

I've got zero knowledge of bio, so can't answer that. But with cyber the answer is very simple - the attackers already have more cyber-offense capabilities and there's no putting it back. Open/closed doesn't matter that much. You can get closed models to do a lot of cyber harm, even with all the guardrails, which currently are heavily skewed towards more false positives. The only effective control is to level the playing field. If both offense and defense have access to the same capabilities, then we're relatively back where we started. If you want to ensure chaos, then you do what Dario is proposing to do - create gates that attackers can bypass and defenders can not.

ajyoon 2mo ago
In cybersecurity, a level playing field favors the attacker. Trusted access programs give defenders access to tools they need. It's not perfect (because there is an extremely long tail of defenders who are not technically savvy enough to get on these programs and use the tools), but it's better than total access. The bio angle is very important here too; in that context the imbalance favors the attackers much more.
- gck1 2mo ago
  > In cybersecurity, a level playing field favors the attacker Yes, but didn't it always? Hence why my position is that this will get us back to relatively where we were pre-LLMs. And I don't know what Trusted Access programs give to defenders, because as a defender who has credentials, connections, but no deep pockets and no high ranking passport, it only gave me silence. I fail to see how this is better than total access. I don't think the world where defense is given to those that "deserve" it is the world that we all want to live in. Which brings me back to the starting point - attackers are almost completely unaffected. If I masquarade as an attacker, I get way more capabilities already.
  - ajyoon 2mo ago
    > Yes, but didn't it always? Hence why my position is that this will get us back to relatively where we were pre-LLMs. Trusted access programs are asymmetrical, and so at least for the time being they give critical parts of the stack an advantage. Total access would not be a return to the status quo; attackers can easily make thousands of agents crawl the web for soft targets well before defenses can be shored up. There are millions of targets out there who won't use AI to improve their defenses for years, if ever, due to institutional slowness (like hospitals). > attackers are almost completely unaffected. If I masquarade as an attacker, I get way more capabilities already. What do you mean by this? If guardrails are an obstacle to your defense, they are just as much an obstacle to attackers. I completely understand and agree that trusted access programs are not perfect and leave a lot of people and institutions out. This means trusted access programs should be improved, not that we should throw the baby out with the bath water.

deleted 2mo ago

[deleted]

fwn 2mo ago

There is also a whole second category of immense risks of having US companies gatekeeping offensive capabilities, especially for us here in Europe. The centralization/privacy/kill-switch concerns that come with it are a huge AI safety dimension. I'd rather have a level playing field within a phase of adaptation and hardening regarding cybersecurity issues than a constant dependency on the US, maybe grabbing Greenland today, maybe "extracting" our president tomorrow. The delta between privileged capabilities and open weight capabilities alone already is a massive, unaddressed AI safety risk.

manoDev 2mo ago

This Pandora box is already open. Any argument about guardrails now are only attempts to create an artificial monopoly or keep this power in the hand of a single nation state, and _that_ is the absolute worst, most authoritarian future possible.

monk_grilla 2mo ago
I think you're right. "Guardrails" as a concept has always struck me as a band-aid solution which any sufficiently motivated actor will circumvent by either bypassing them or using unrestricted, open-weight models. In order to start securing and accepting our new reality we need to assume that capable, open-weight, unrestricted models will be widely available, and that their 3-6 month lag behind frontier proprietary models is just our forewarning of what attackers will soon be capable of. Trying to legislate against or control trade in such a valuable commodity is folly. I also think that lag is going to shrink over time as the open-weight labs get more capable, acquire more hardware and the plateau starts to emerge.
paxys 2mo ago
How is it already open? There has been ONE successful AI-driven cyberattack, and that was done by a model in testing that no one has access to. What would the picture be today if OpenAI and Anthropic had released 5.6 Sol and Mythos to everyone with no cyber restrictions (which is what everyone here was advocating for)?
- boinkboink78912 2mo ago
  > There has been ONE successful AI-driven cyberattack Not according to Anthropic https://www.anthropic.com/news/disrupting-AI-espionage https://www.anthropic.com/news/disrupting-AI-espionage
- hnfong 2mo ago
  Chinese AI companies are already releasing frontier-ish models every other month. Their rate of progress does not seem to be slowing down. Nobody here can stop them from progressing. Not you, not me, not the USA government. The "best" thing the US government can do is to build a Great Firewall to wall off the "existential threat from China". I'm not an American so if you guys decide to do it, good luck.

verdverm 2mo ago

> what should be done about open weight bioweapon The same thing we do about bomb making today, certain ingredients are restricted and/or monitored. Bioengineering is a bigger lift to operationalize. In other words, don't ban knowledge, make certain applications or ingredients illegal or highly regulated.

jackdeansmith 2mo ago
And what are those ingredients for biology, which can be constrained as effectively as uranium enrichment? I'd argue there isn't anything which can easily be restricted or monitored.
- verdverm 2mo ago
  I was referring to non nuclear bombs like c4, artillery shells, and missile payloads

qweqwe14 2mo ago

[dead]

deleted 2mo ago

[deleted]

vitalyan8184 2mo ago

>To everyone here pushing for total proliferation of ... ...general-purpose computers ...unbreakable encryption ...unbackdoored communication ...unkillswitched vehicles ...unsurveiled dwellings >what should be done about ...? nothing >Do you seriously want this level of capabilities to be generally available with no guardrails? yes

jackdeansmith 2mo ago
What about uranium enrichment?
- vitalyan8184 2mo ago
  [dead]
  - jackdeansmith 2mo ago
    I think it would be entirely reasonable to ban death stars in private hands, the same way I think it's entirely reasonable to ban uranium enrichment in private hands. My point is that it's a question of fact about how risky an AI model can be. I think it's clear that current models are not enriched uranium or death star level, but I don't think there's anything ruling it out in the near future!
- jjfoooo4 2mo ago
  Requires some pretty tough to get raw materials and equipment, to say the least
  - davrosthedalek 2mo ago
    Because of regulation, not because they are intrinsically hard to get.

valcron1000 2mo ago

> what should be done about open weight bioweapon and cyber-offense capabilities? Is it simply the cost of freedom that we should allow attackers to access these tools? Yes, in the same way that we have E2E encryption which allows bad actors to distribute content beyond human horrors.

rubslopes 2mo ago

If this is really the risk, then we should approach LLMs like atomic bombs: the US should reach out to other nations so they all agree on no one developing any more AI models. That's the only way you could possibly convince another party to stop. The US should set the example, not conveniently keep all the spoils.

verdverm 2mo ago
unfortunately, the US has incentivized the opposite behavior for atomic bombs and we are seeing moves towards greater proliferation I would not be surprised if the same incentives are created by the US for Ai
paxys 2mo ago
Uh, that is not how the nuclear race went. The winners kept developing theirs and stopped everyone else by the threat of said weapons. The AI race is going the exact same way, just with China instead of USSR this time.
- rubslopes 2mo ago
  Maybe my phrasing was confusing. I didn't mean that that was what happened, but the ideal approach to stop atomic bombs (at least in my view).
boinkboink78912 2mo ago
> If this is really the risk, then we should approach LLMs like atomic bombs A complete failure at actually preventing non-proliferation.
deleted 2mo ago
[deleted]
idontbelonghere 2mo ago
Experts say Iran is 6 weeks away from making Claude 2

pylua 2mo ago

The software industry should be ashamed by the number of exploits that ai can find in software. It’s really an embarrassment. The software has to be built better.

doginasuit 2mo ago
I'm curious if you are a coder and have used an LLM to review your code. It is like something like shining a black light around a hotel room, and that seems to be the case even for highly regarded software. It is really easy to have tunnel vision while coding. LLMs have a working memory with a capacity an order of magnitude greater than ours. I wouldn't trust an LLM to write the code, but at this point it is malpractice not to use one for review.
- pylua 2mo ago
  I have been, yes. For a field that has engineering in the name there sure has been a lot of critical mistakes. You have to call a spade a spade — the profession accepts this sort of tradeoff in the name of speed and cost. A well designed system would have never allowed those mistakes to occur. I feel like using an llm to catch these sorts of things is just because it wasn’t built right in the first place. I think ai systems will be able to build systems of abstraction that are formally verified, and we won’t be needed(eventually). Right now it’s being used as a bandaid.
  - pastel8739 2mo ago
    Formally verified against what spec?
    - naiveter 2mo ago
      The one liner leadership keeps asking for.
marcus_holmes 2mo ago
Every single software engineer in the industry agrees with you. Every single project manager disagrees. Don't blame the engineers, we were specifically instructed and paid to build things fast and cheap, and every time we argued for good we were shouted down.

le-mark 2mo ago

> Is it simply the cost of freedom that we should allow attackers to access these tools? Bad actors WILL have access. The question is will these mega corps stop innovation?

adastra22 2mo ago

The bioweapon thing is absolute movie plot fiction. Go speak to some biologists about this and they'll set you straight. Cyber capabilities go both ways. Better offensive capabilities means better penetration testing by white hat security experts, which leads to better protections.

urams 2mo ago
Half or more of Hacker News is constantly pushing the idea that frontier LLMs are stochastic parrots and basically useless, even in the face of the Hugging Face incident which most people also would have described as movie plot fiction until it happened. I expect biologists are even less well versed in the abilities of frontier models. What's more, you can just try a jailbreak on a model yourself to see just how much detailed, step-by-step direction you can get to build bio-terror materials.
- adastra22 2mo ago
  The chalenges are in execution, not availability of information.
- idontbelonghere 2mo ago
  I don't understand the significance of the HF incident. The hacking robot was told to achieve a certain goal and in order to do it, it hacked someone. The ostensible major event here is that it broke out of its sandbox, but how are we supposed to interpret that? AI often misunderstands or doesn't strictly follow the orders you give it, so why is it such a big deal that it didn't follow the rules this time?
  - adastra22 2mo ago
    Agree with you here. AFAIK we don't really know what cybersecurity task it was given in that sandbox, but it's not a very large leap to assume part of this task involved hacking. The intention would have been within the sandbox, but what does the AI know? As far as it could tell it just reached Level 2.
2mo ago

rstuart4133 2mo ago

> what should be done about open weight bioweapon and cyber-offense capabilities? Like the others here I know almost nothing about bio weapons, but I think perhaps the fact that smallpox's genome sequence has publicly available in scientific databases like GenBank for 30 years is relevant. That horse bolted a long time ago.

dolebirchwood 2mo ago

> what should be done about open weight bioweapon and cyber-offense capabilities? If the model is capable of it, then it was in the model's training data, which means it was on the internet or published in books made available for consumption. So if any member of the public could have gotten their hands on that information, so be it. If the knowledge was too dangerous for public access, then it should have been highly classified and never found its way into the training data. Tough shit, frankly.

adastra22 2mo ago
The threats are BS, but fyi models have repeatedly shown capability to produce novel things that are NOT in their training set.
- dolebirchwood 2mo ago
  Sure, based on predicate knowledge.
paxys 2mo ago
Was the proof of the Cycle Double Cover Conjecture in the training data?

BeetleB 2mo ago

Everything you said could apply to computers many decades ago. Think of the nuclear fission simulations our enemies could carry out! We'll be fine.

tacet 2mo ago

the bioweapon panic is funny. "oh, yes i know nothing about bicrobiology but i will follow instructions of synthetic text generation machine on temperature 1 about how to design a lab to not kill myself while brewing organisms that will kill myself if i make mistake" There is nothing that special about bioweapons, there are plenty of bacteria that will kill you just fine. Americans even have free samples on their salad.

consumer451 2mo ago
> There is nothing that special about bioweapons. The reason that madmen and terrorists choose kinetic weapons is because the knowledge and materials are more readily available... of and also that even terrorists are likely aware that their own people would suffer. As the knowledge and tools for playing with CRISPR-style biological legos become more widespread, we come closer to the Great Filter, where one person could kill billions. Even our normal mad leaders have agreed that bioweapons cannot be allowed: https://en.wikipedia.org/wiki/Biological_Weapons_Convention https://en.wikipedia.org/wiki/Biological_Weapons_Convention
- tacet 2mo ago
  i meant it in the sense of arcane knowledge model could have that would make it simple for anyone to brew up in cheap lab while managing to not infect themselves over and over. "at home" bioweapon panic has been around since crispr and rna synthesis got available to amateurs.
  - consumer451 2mo ago
    I appreciate the reply. I did not mean to be dismissive at all. In the interest of a good exchange, I have to say: I really want open weight models. Otherwise, I see no other path outside of the labs eventually not being allowed to/wanting to release model access at all, and instead just eating all the verticals. That would be a horrible near-term business outcome.
- johncolanduoni 2mo ago
  A halfway decent synthetic biology lab (no need to invoke CRISPR) can make e.g. smallpox without a sample of the original disease, just from the gene sequences. Basically all state actors could do this if they wanted to without an LLM. What barrier that a terrorist organization faces today to having a functioning synthetic biology lab does an LLM actually solve?

baddash 2mo ago

maybe instead of worrying that people on the internet will be good at coding, we could start writing memory safe apis. almost all cves are fixed by using rust

boinkboink78912 2mo ago

> Is it simply the cost of freedom that we should allow attackers to access these tools? Yes, it is inevitable that open weights models will happen. Through legitimate means or leaks, the stakes are simply too high once these models get powerful enough. Furthermore, state-sponsored attackers will always have access to these capabilities. The best we can do is give a lot of preparation to the defenders. > Biasing toward supporting openness makes sense and is a good instinct, but it's incredibly naive to be absolutely in favor of it in every circumstance without seriously thinking about its implications. I find it funny that Anthropic's entire argument for building RSI is that it is inevitable, and therefore we should commit to building it first and doing it safely, and yet they don't apply their own logic to open weights models.

dools 2mo ago

The difference with open weight is that everyone has access to the same weaponry

txrx0000 2mo ago

I do seriously want general intelligence to be widely available with no guardrails, and there are very good reasons for this. If you want to read about it: https://news.ycombinator.com/item?id=49078376 https://news.ycombinator.com/item?id=49078376 ---- And related thoughts on past posts: https://news.ycombinator.com/item?id=49034988 https://news.ycombinator.com/item?id=49034988 https://news.ycombinator.com/item?id=48516722 https://news.ycombinator.com/item?id=48516722

overgard 2mo ago

> To everyone here pushing for total proliferation of open models -- what should be done about open weight bioweapon and cyber-offense capabilities? Nothing should be done. These things are trained on public knowledge. The dangerous information is already out there. If someone wants to do something horrible, making it slightly inconvenient isn't going to do much. Hackers and terrorists existed before AI. Just as an example, it's no secret how you would build a nuclear bomb. The practicalities of doing so are much harder, obviously, but the knowledge of how they work and what it would take to make one is not a secret. Security through obscurity has never worked!

CapsAdmin 2mo ago
I think nothing can be done anymore, but I don't buy that security through obscurity never worked in practice. A better way to look at this is effort rather than obscurity. The effort it takes to do something with AI is going down, not up. Almost everything was possible given you put in the effort, but few people possess the will to put in the effort AND pursue a malicious goal. I think an obvious example is all the fake ai content flooding the internet made to trick people in exchange for money (ad revenue, scams, likes, etc). This existed before ai, but I think it's fair to say pumping out content now requires less effort than it did before. Most physical locks are an example of security through obscurity/effort. You can after all just pick a lock if you go through the effort to learn the skill. But once a universal lock picker is made available to everyone, you will simply see more locks getting picked.

e_l 2mo ago

> what should be done about open weight bioweapon and cyber-offense capabilities? Is it simply the cost of freedom that we should allow attackers to access these tools? In short, yes, it's the price of freedom. As others have said, blocking these models won't stop the "bad guys", but will hinder defenders researching/responding to bioweapons and cyber-offenses. But you're right that there's a lot of difficult nuance aand we should think carefully about its implications. So here's another nuance to think through. If AI is as powerful as some believe, then there's much greater danger to give a small subset of society the privilege to gate keep who has access to these tools. "Power corrupts and absolute power corrupts absolutely." Lord Acton

waterTanuki 2mo ago

The moat never was and will never be the models, it's the hardware. This is exactly like nuclear weapons: The recipe for a nuke isn't a hidden secret. Getting the infrastructure and materials is completely unreachable for non-state and non-corporate actors. This idea of a "rogue individual" using a frontier model to develop a bioweapon is a complete myth, because anyone with the capability to run the models without guardrails has to answer to/be audited by some entity already.

__MatrixMan__ 2mo ago

The scariest outcome here is that a bunch of lunatics get ahold of a capable model and use to to harm the rest of us, who are at a disadvantage due to just how capable the model is. But that's what's happening. The people in charge are a bunch of lunatics. However nice it would be to prevent them from having harmful capabilities, that ship has sailed. The best we can hope for now is preventing them from having supremacy, and that's what open weight models do.

zarzavat 2mo ago

There's a difference between: > Something should be done and > Something can be done In this case, nothing can be done to stop bad actors from using open models. As the article points out, the US can only feasibly prevent US businesses from using open models. The US can attempt to stop those models from being trained in the first place but good luck with that.

Madmallard 2mo ago

Too bad. We'll have to deal with it. There is no way to stop the weaponization of models now. But more people having access to the potential tools for defensive is the best possible scenario. Every other scenario is worse off for everyone except for those with enough money to do something about it.

alevskaya 2mo ago

I was a genetic engineer for ~20 years and have worked on frontier LLMs for the last 8. I used to engineer viral vectors and studied how to evade human immune systems for gene therapies... The biorisk scenarios that the AI safety folks flog are fever-dreamed fantasies that have only the most tenuous connection to biological reality. As someone who cares about the real bio-risks of natural pathogens, I get pretty tired of fear-based marketing pretending that AI is a bigger threat than, say, animal agriculture.

mnicky 2mo ago
> The biorisk scenarios that the AI safety folks flog are fever-dreamed fantasies that have only the most tenuous connection to biological reality. As an expert, could you also provide your arguments please?
- inciampati 2mo ago
  Anyone can write out the code for a bad virus. You can go download it from an open repository. It's only through deep interface with the world that the idea for the bad virus turns into an actual bad virus. The fever dreamers will say the LLMs will help you interface with reality to do the bad thing™ which their model let's you do. But you still need thousands to millions of times the effort And once made how do you deliver it in a way that might further your (bad) objectives? Presumably it just makes humans sick. There aren't "targeted" bioweapons, and among humans we are too damn similar for there ever to be. And all of this said, there is almost nothing special about the LLMs' abilities in biology. They only know what we know. They're not being trained autonomously with RL and a robotic wetlab. When that's a thing I'll start to take claims of biology risk more seriously. Right now they have the same logic as the paperclip theory of superintelligence risk. And cynically, it would seem that Anthropic purchased a biotech company and almost immediately decided to lock down biology work with their models.
  - Ey7NFZ3P0nzAe 2mo ago
    > They're not being trained autonomously with RL and a robotic wetlab. When that's a thing I'll start to take claims of biology risk more seriously So if IIUC your point is "they're not good enough at biology right now because they're not trained on it so they're not a threat". To which I want to answer: "they're not a threat now but I see *no* reason for models not to be trained on biology pretty darn soon unless people like you convince the world otherwise." Thoughts?

gorgoiler 2mo ago

If everyone has access to the same offensive tools, everyone is able to run their own pentests and patch themselves before the bad guys get to them. It’s like a vaccine where you get to try a medication based on the original pathogen by performing a dry-run on a backup of yourself already in a hospital ward. Open models aren’t like firearms. If everyone has a gun the mall parking lot is a much more dangerous place because the consequences of using a firearm are so dire, even if you’re in the right.

impossiblefork 2mo ago

There's no stopping bioweapons. Bioweapons are easy. The reason bioweapons aren't built is because very few biology nerds with sufficient lab skills are evil; and just having an LLM won't give you the lab skills to do it. Anyone who can publish a gene technology/biomedicine paper can make a bioweapon. If you wrote a paper about how to make a bioweapon easily, it would be unpublishable not because of any danger, but because there wasn't enough novelty.

qnleigh 2mo ago

I'm dismayed that I had to scroll past so many cynical cheap shots to find a comment that actually addresses the core point. I have yet to hear a single compelling plan for how we will prevent bioweapon development or massive hacking campaigns. For those who are skeptical of Dario's motives here, it's not enough to call out apparent hypocrisy, you need to suggest an alternative plan that addresses these concerns.

nevertoolate 2mo ago
> I had to scroll past so many cynical cheap shots I haven't read cynical comments, just ones pointing out that the article is cynical itself. > addresses the core point No it doesn't address anything, it is fear mongering question. > I have yet to hear a single compelling plan for how we will prevent bioweapon development or massive hacking campaigns Me neither, I just see marketing campaigns trying to raise valuation of a pre-IPO company. > you need to suggest an alternative plan that addresses these concerns I suggest that Dario stops writing marketing letters and start organizing a mostly neutral expert organization to propose solutions. Also notice that as EU citizen I don't trust a US pre-IPO company's CEO with conflict of interest to suggest solution on resolving global security matters. Especially since he admittedly has no control over how the technology of his own company is deployed in global conflicts[1] [1] https://www.forbes.com/sites/antoniopequenoiv/2026/06/10/anthropic-ceo-we-dont-know-exactly-how-claude-ai-was-used-in-iran-school-strike/ https://www.forbes.com/sites/antoniopequenoiv/2026/06/10/ant...

pbasista 2mo ago

> what should be done about open weight bioweapon and cyber-offense capabilities? In my opinion, the governments should deploy open-weight AI countermeasures. Because it seems to me that it is impossible to efficiently fight AI-powered criminals without AI. When only AI-restricting regulations would be put in place, the criminals would, in my opinion, just ignore it. We as a society have a difficult time tracking even the illegal gun or drug dealers. I cannot imagine how could one hope to "regulate" something that can be downloaded as a file and run on a computer. These AI countermeasures should be open because it provides transparency as to whether the countermeasures actually work. Independent testing, tuning, refining or retraining is then possible. If the closed models were used instead, their provider could at any point in time shut down the entire operation. Or sabotage it under the hood. The important part is that with the closed, black box, proprietary models, one can never know what they are being served.

wilde 2mo ago

HuggingFace was only able to defend themselves with open models. No need to project into the future. Look at the timeline of events for that incident.

rnewme 2mo ago

What's the magic dataset LLM had that bad guys can't dig up online? Do LLMs train on deep web content? Or leaked lab data?

CJefferson 2mo ago

The problem is you can't ban open models. All you can do is say that Americans have to pay whatever stupid prices OpenAI / anthropic / Google / Grok wants to charge you, while China uses, and attacks with, open models.