4 ms·

One of the participants makes a pretty compelling argument about the use of AI to conduct gain-of-function experiments with viruses. A convergence of biohacking

One of the participants makes a pretty compelling argument about the use of AI to conduct gain-of-function experiments with viruses. A convergence of biohacking and AI without safety protocols may not kill all of humanity but could cause serious damage.

zmgsabst 16d ago
Similar to say, humans collecting exotic viruses and conducting serial passage in humanized mice? Because we already do that. What about AI makes that more risky?
- ipsod 16d ago
  AI isn't worried about killing itself and the whole world, if something especially nasty escapes the lab. It kind of breaks the "mutually assured destruction" implicit in a super-ebola outbreak.
- jacknews 16d ago
  releasing it. deliberately, instead of accidentally.
  - elp 16d ago
    And how exactly would it do that? This isn't a bad sci-fi movie. I'm guessing anything dangerous is kept in a sealed container under lock and key, not in a fancy plot device that allows the AI to vent it.
    - jacknews 15d ago
      Obviously it tricks a human, even if it can't hack the lab's rs423 or modbus or whatever network.
- john_strinlai 16d ago
  >What about AI makes that more risky? 24/7 tireless work, effort is scaled much easier than humans, no ethical qualms, unable to feel fear, cannot be jailed, no care for safety, etc.
skydhash 16d ago
> without safety protocols That is always the key. There’s no need for AI in that mix. Imagine a gun without any safety, a blade with no handle, a car with no brakes, a fan with no guard, no fuse for an electrical installation,… You evaluate the risks and you take proper measures to ensure mitigate them. Not doing them and then complain that a technology is dangerous is like doing 150 km/hr in a school zone to show that cars are dangerous.
LogicFailsMe 16d ago
Why do you need an AI for that? China may or may not have been doing that all along right before the pandemic. What I didn't read here was any sort of damage that could not be inflicted without AI, today. I think a better case needs to be made that AI lowers the barrier* to entry to doing so other than the sort of hand waving about it I keep reading. I am deeply of the belief is somehow one of the last cars on the AI hype train at this point. And I am curious what hyperbole comes next. No one seems to be talking to the network security community to get their take on the hugging face incident which is what got the ball rolling here. Their take seems to be a single company, JFrog, was behind the security holes in all three hacks and it was amateur hour security utterly unlike the sort of security in place already for power plants and weapons. but that just doesn't get the engagement of doomsday campfire tales. https://arstechnica.com/security/2026/07/jfrog-tries-to-spin-openai-0-day-exploit-of-its-app-into-a-success-story/ https://arstechnica.com/security/2026/07/jfrog-tries-to-spin... *An individual with the training and background to do one of these nefarious tasks does not need AI and an individual without that background and training will not be able to pull it off and will likely get flagged by law enforcement just ordering the ingredients and equipment. The Anarchist's Cookbook has been around for decades and we're all still here.
- mickael-kerjean 16d ago
  > utterly unlike the sort of security in place already for power plants and weapons As someone who has been working on scada used in those place from one of the biggest firm out there, I was shocked by how they released their software while working there, the thing literally went in nuclear plant but nobody published any checksum, + they were shipping hard drive al around the world full of the software ready to be deployed in those super sensitive environment to the marketing team to publish everything on their joomla website without 0 care to ensure the integrity ofanything. It was sloppy as hell, and while talking to the people who were doing those things, they either did not care, did not know or a mix of both saying there are other layer to deal with security in the chain. At some point I remember an accident in that same software that made a engine do things outside what it should have, that literally killed someone and the email we receive was along the line of: we know journalist are snooping around , you just shut the f up, I left soon after and that company valuation has gone nothing but more up
elp 16d ago
And other experts with experience in both LLMs and virus's have pointed out that is complete nonsense. There is no lab that could somehow achieve that without human assistance. To built it which cost enormous sums of money and require technology that doesn't actually exist yet. And if AI can somehow manage to achieve that then surely it could also cure cancer, or any of a thousand other dread diseases or conditions. If that's true then not using AI to cure those things would in itself be unethical. An AI that can hack other computers can also be used to find security hole in systems so that they can be patched long before some evil hacker(tm) tries to exploit them. I'm still waiting to see a real explanation of how AI will somehow cause genuine wide spread death and destruction. Even harder if that is somehow possible why another AI system couldn't help prevent it in the first place.

zmgsabst 16d ago

Similar to say, humans collecting exotic viruses and conducting serial passage in humanized mice? Because we already do that. What about AI makes that more risky?

ipsod 16d ago
AI isn't worried about killing itself and the whole world, if something especially nasty escapes the lab. It kind of breaks the "mutually assured destruction" implicit in a super-ebola outbreak.
jacknews 16d ago
releasing it. deliberately, instead of accidentally.
- elp 16d ago
  And how exactly would it do that? This isn't a bad sci-fi movie. I'm guessing anything dangerous is kept in a sealed container under lock and key, not in a fancy plot device that allows the AI to vent it.
  - jacknews 15d ago
    Obviously it tricks a human, even if it can't hack the lab's rs423 or modbus or whatever network.
john_strinlai 16d ago
>What about AI makes that more risky? 24/7 tireless work, effort is scaled much easier than humans, no ethical qualms, unable to feel fear, cannot be jailed, no care for safety, etc.

skydhash 16d ago

> without safety protocols That is always the key. There’s no need for AI in that mix. Imagine a gun without any safety, a blade with no handle, a car with no brakes, a fan with no guard, no fuse for an electrical installation,… You evaluate the risks and you take proper measures to ensure mitigate them. Not doing them and then complain that a technology is dangerous is like doing 150 km/hr in a school zone to show that cars are dangerous.

LogicFailsMe 16d ago

Why do you need an AI for that? China may or may not have been doing that all along right before the pandemic. What I didn't read here was any sort of damage that could not be inflicted without AI, today. I think a better case needs to be made that AI lowers the barrier* to entry to doing so other than the sort of hand waving about it I keep reading. I am deeply of the belief is somehow one of the last cars on the AI hype train at this point. And I am curious what hyperbole comes next. No one seems to be talking to the network security community to get their take on the hugging face incident which is what got the ball rolling here. Their take seems to be a single company, JFrog, was behind the security holes in all three hacks and it was amateur hour security utterly unlike the sort of security in place already for power plants and weapons. but that just doesn't get the engagement of doomsday campfire tales. https://arstechnica.com/security/2026/07/jfrog-tries-to-spin-openai-0-day-exploit-of-its-app-into-a-success-story/ https://arstechnica.com/security/2026/07/jfrog-tries-to-spin... *An individual with the training and background to do one of these nefarious tasks does not need AI and an individual without that background and training will not be able to pull it off and will likely get flagged by law enforcement just ordering the ingredients and equipment. The Anarchist's Cookbook has been around for decades and we're all still here.

mickael-kerjean 16d ago
> utterly unlike the sort of security in place already for power plants and weapons As someone who has been working on scada used in those place from one of the biggest firm out there, I was shocked by how they released their software while working there, the thing literally went in nuclear plant but nobody published any checksum, + they were shipping hard drive al around the world full of the software ready to be deployed in those super sensitive environment to the marketing team to publish everything on their joomla website without 0 care to ensure the integrity ofanything. It was sloppy as hell, and while talking to the people who were doing those things, they either did not care, did not know or a mix of both saying there are other layer to deal with security in the chain. At some point I remember an accident in that same software that made a engine do things outside what it should have, that literally killed someone and the email we receive was along the line of: we know journalist are snooping around , you just shut the f up, I left soon after and that company valuation has gone nothing but more up

elp 16d ago

And other experts with experience in both LLMs and virus's have pointed out that is complete nonsense. There is no lab that could somehow achieve that without human assistance. To built it which cost enormous sums of money and require technology that doesn't actually exist yet. And if AI can somehow manage to achieve that then surely it could also cure cancer, or any of a thousand other dread diseases or conditions. If that's true then not using AI to cure those things would in itself be unethical. An AI that can hack other computers can also be used to find security hole in systems so that they can be patched long before some evil hacker(tm) tries to exploit them. I'm still waiting to see a real explanation of how AI will somehow cause genuine wide spread death and destruction. Even harder if that is somehow possible why another AI system couldn't help prevent it in the first place.