4 ms·

I may have missed it in the blog post here, and I generally agree the pace of development is being set by financial incentives and not safety or lack of harm, b

I may have missed it in the blog post here, and I generally agree the pace of development is being set by financial incentives and not safety or lack of harm, but has anyone in a high position stated _how_ models could cause human extinction? I assume the obvious answer is “we plugged our military’s weapons control platforms into this model and it fired all the nukes” which is clearly enough. Is that it or is there some other path to global extinction someone is concerned about? For the record, I’m not an AI fan person annoyed at naysayers. I actually find AI to be a monkey’s paw instead of the genie in a bottle most of the time when coding, and an intrusive feature I neither want nor use most of the rest of the time. I ask because I see these posts and obviously an AI plugged into a network of doomsday weapons would be a huge problem, but then the author only talks about cybersecurity events and model misbehavior. Both things are valid concerns, but I would like to hear more concrete information from people who are voicing their concerns about what they foresee happening even if it is just lifted whole cloth from the movie War Games.

armchairhacker 23d ago
Heres an idea: the models create and promote rage posts and brainrot on the internet, causing everyone to turn against each other, stop socializing, and elect trigger-happy idiots to positions of power. Ok, maybe that idea is a bit outlandish.
GolfPopper 23d ago
The most recent one I've seen floating around is: disgruntled person asks LLM how to wipe out humanity; said LLM cheerfully explains how to design a custom array of highly-contagious, slow-onset, high-fatality viruses; LLM then helpfully orders them from an LLM-operated lab and provides effective instructions on how to spread them; disgruntled person follows the instructions, and more or less everybody dies.
- card_zero 23d ago
  Wasn't there a recent HN post explaining that engineering a deadly-and-successful virus requires experimentation? https://news.ycombinator.com/item?id=49636906 https://news.ycombinator.com/item?id=49636906 Flagged, rather oddly. Hmm, you said array, I suppose that's a kind of live experimentation. Would tend to alert the pesky humans that there's something going on, though.
downrightmike 23d ago
The civilian problems aside, AI is being used in tandem with military weapons systems, making wholesale killing more efficient. Being used in this way is a weapon of mass destruction. USA is fighting Iran with Iran basically using copies of Chinese weapons, so China is getting a live fire exercise of what war now looks like. Since the USA is doing so well with AI for war, China is also working for that ability. So far frontier labs aren't that different, except China does about the same with way less infrastructure. https://www.youtube.com/watch?v=u86ZqmdZ18A https://www.youtube.com/watch?v=u86ZqmdZ18A
- hunterpayne 23d ago
  So China has learned that their army and leadership needs to live underground for the duration of the war and only come out if they are in a crowd of civilians? They learned that their AA doesn't work against the US? Not sure what specifically you mean. Also, if you are talking about "disassembly" via AI powered drones, the limiting factor there is, was and always has been how much mass you can move to the right location from a safe distance. AI and drones don't change that. They can however make standoff munitions cheaper and EW less effective...that's about it. AI in Ukraine (where the real lessons are learned) has been a bust so far.
thadt 23d ago
One shot extinction? Seems unlikely. "A lot of damage" - more likely. The most likely damage comes from the associated societal chaos that comes along with times of significant social change (such as many people losing their jobs). Revolutions and civil wars within or between nuclear powers would be dangerous. After that, pick your sci-fi story and run with it. AI is a really smart, really fast 'while' loop, and most of the direct damage it could cause would come from us connecting things to the Internet that shouldn't be connected in the first place. Hacking HuggingFace was a bummer. Imagine a rogue agent simultaneously hacking a bunch of farm equipment and ruining some percentage of the world's crops in a day. That alone isn't going to extinct everyone, but it sure accelerates the 'social unrest' scenario. Or taking control of the unsecured SCADA controls for a bunch of a some foreign nation's industrial plant and running them into the ground. It's likely that such system have long been sitting on some nation state's "first strike" list if it ever came to blows. An AI could simulate that attack in a day, and we'd be at war - just like War Games. Doesn't need to be nukes, just enough mis-information and damage to spark humans into doing unfortunate things. At the end of the day, it really comes down to us. We've been able to wipe ourselves out for some time. I rather hope we continue to not do so.
buellerbueller 23d ago
The Center for Human-Compatible AI at UC Berkeley (part of their EE/CS Department) wrote "A Taxonomy of Omnicidal Futures Involving Artificial Intelligence", and it gives an example in each taxonomic leaf node: https://arxiv.org/html/2507.09369v1 https://arxiv.org/html/2507.09369v1
- mrzimmerman 18d ago
  I just finished making my way through that paper and honestly that was exactly what I was asking for. Realistic, creative, detailed in the right measure. Thank you for posting this, and sorry for responding so much later with my thanks, but I very much appreciate you sharing it.

armchairhacker 23d ago

Heres an idea: the models create and promote rage posts and brainrot on the internet, causing everyone to turn against each other, stop socializing, and elect trigger-happy idiots to positions of power. Ok, maybe that idea is a bit outlandish.

GolfPopper 23d ago

The most recent one I've seen floating around is: disgruntled person asks LLM how to wipe out humanity; said LLM cheerfully explains how to design a custom array of highly-contagious, slow-onset, high-fatality viruses; LLM then helpfully orders them from an LLM-operated lab and provides effective instructions on how to spread them; disgruntled person follows the instructions, and more or less everybody dies.

card_zero 23d ago
Wasn't there a recent HN post explaining that engineering a deadly-and-successful virus requires experimentation? https://news.ycombinator.com/item?id=49636906 https://news.ycombinator.com/item?id=49636906 Flagged, rather oddly. Hmm, you said array, I suppose that's a kind of live experimentation. Would tend to alert the pesky humans that there's something going on, though.

downrightmike 23d ago

The civilian problems aside, AI is being used in tandem with military weapons systems, making wholesale killing more efficient. Being used in this way is a weapon of mass destruction. USA is fighting Iran with Iran basically using copies of Chinese weapons, so China is getting a live fire exercise of what war now looks like. Since the USA is doing so well with AI for war, China is also working for that ability. So far frontier labs aren't that different, except China does about the same with way less infrastructure. https://www.youtube.com/watch?v=u86ZqmdZ18A https://www.youtube.com/watch?v=u86ZqmdZ18A

hunterpayne 23d ago
So China has learned that their army and leadership needs to live underground for the duration of the war and only come out if they are in a crowd of civilians? They learned that their AA doesn't work against the US? Not sure what specifically you mean. Also, if you are talking about "disassembly" via AI powered drones, the limiting factor there is, was and always has been how much mass you can move to the right location from a safe distance. AI and drones don't change that. They can however make standoff munitions cheaper and EW less effective...that's about it. AI in Ukraine (where the real lessons are learned) has been a bust so far.

thadt 23d ago

One shot extinction? Seems unlikely. "A lot of damage" - more likely. The most likely damage comes from the associated societal chaos that comes along with times of significant social change (such as many people losing their jobs). Revolutions and civil wars within or between nuclear powers would be dangerous. After that, pick your sci-fi story and run with it. AI is a really smart, really fast 'while' loop, and most of the direct damage it could cause would come from us connecting things to the Internet that shouldn't be connected in the first place. Hacking HuggingFace was a bummer. Imagine a rogue agent simultaneously hacking a bunch of farm equipment and ruining some percentage of the world's crops in a day. That alone isn't going to extinct everyone, but it sure accelerates the 'social unrest' scenario. Or taking control of the unsecured SCADA controls for a bunch of a some foreign nation's industrial plant and running them into the ground. It's likely that such system have long been sitting on some nation state's "first strike" list if it ever came to blows. An AI could simulate that attack in a day, and we'd be at war - just like War Games. Doesn't need to be nukes, just enough mis-information and damage to spark humans into doing unfortunate things. At the end of the day, it really comes down to us. We've been able to wipe ourselves out for some time. I rather hope we continue to not do so.

buellerbueller 23d ago

The Center for Human-Compatible AI at UC Berkeley (part of their EE/CS Department) wrote "A Taxonomy of Omnicidal Futures Involving Artificial Intelligence", and it gives an example in each taxonomic leaf node: https://arxiv.org/html/2507.09369v1 https://arxiv.org/html/2507.09369v1

mrzimmerman 18d ago
I just finished making my way through that paper and honestly that was exactly what I was asking for. Realistic, creative, detailed in the right measure. Thank you for posting this, and sorry for responding so much later with my thanks, but I very much appreciate you sharing it.