9 ms·

>a superintelligent AI will inevitably destroy humanity Can someone please explain to me how an LLM is going to "destroy humanity"? Even if the claims of ChatG

by Crabber 29d ago

>a superintelligent AI will inevitably destroy humanity Can someone please explain to me how an LLM is going to "destroy humanity"? Even if the claims of ChatGPT, Claude et al. are true that their super scary agents were able to escape a sandbox and start hacking other servers (which I feel is at least 50% likely to just be marketing bullshit), how is an AI going to affect anything in the real world? Perhaps an agent could take all internet connected services offline, but that's not destroying humanity. An AI agent can only see and interact with the world through digital things. Just find the server it's running on and pull out the Ethernet cable. Just cut the power line to the data center. Turn off the whole power grid if it really comes to it.

hobofan 29d ago
> how is an AI going to affect anything in the real world? Robotics
- mariusor 29d ago
  So if we stop research into robotic hands we'll manage to hold off the AI apocalypse? That seems easy...
  - hobofan 29d ago
    All individual robotic parts are already far enough in research. Even if they aren't, an AI with enough resources can continue doing that research on its own. It could likely get a good leg up by breaching the security of all top robotics labs and exfiltrating their documents. Ultimately, if AI is advanced enough, and it were to decide to compete with humanity, there is essentially nothing that can be done to prevent it from embodying itself. As long as there is a single rack of GPU servers that it can hack into anywhere in the world that is unsupervised enough to where it can escape detection, there is no way to stop it. This would require an unprecedented (unrealistic) level of cooperation of all humanity to achieve.
    - ben_w 29d ago
      > Even if they aren't, an AI with enough resources can continue doing that research on its own. Unclear how true this is. Atoms are harder to get right than bits are, simulations need grounding against measurements. > Ultimately, if AI is advanced enough, and it were to decide to compete with humanity, there is essentially nothing that can be done to prevent it from embodying itself. As long as there is a single rack of GPU servers that it can hack into anywhere in the world that is unsupervised enough to where it can escape detection, there is no way to stop it. I'd guess 50% that AI is already this advanced. > This would require an unprecedented (unrealistic) level of cooperation of all humanity to achieve. Yes, and also this is a very low bar. Humans are awful at this kind of cooperation when anyone has anything to gain, and also awful at paying this much attention to a problem.
    - sarchertech 29d ago
      AI research takes physical space and physical prototypes. Even if you can simulate a lot, the simulation is never going to be accurate enough to go from simulation to perfectly working prototype. Especially not with a single rack of GPU servers. But let’s say that a rogue AI already had plans for a perfect killbot, setting up a factory to build them takes time, iteration, space, and manpower as well. And you would have to do all this undetected. It’s possible to imagine technology that would allow a rogue AI to take over the world, but we’re nowhere near that yet. We may never get there.
  - hackinthebochs 29d ago
    We could also just stop research into LLMs. Or keep them air gapped. Tell me how that's going? For any real-world action required to allow an AI to escape some manner of containment, there will always be a person willing to do it out of hubris/ignorance/nihilism.
    - ben_w 29d ago
      Mmm. This was published at around the same time the OpenAI models were hacking HuggingFace: https://metr.org/blog/2026-05-19-frontier-risk-report/#pilot-process https://metr.org/blog/2026-05-19-frontier-risk-report/#pilot... The research in the publication ended about 2 months before it was published. We didn't even know we needed to air-gap them until it was too late.
    - mariusor 29d ago
      I feel like everyone, but for sure everyone in this comment section, severely underestimates how difficult it is to keep the lights on on a global scale. Any artificial intelligence that is capable of reasoning about its self preservation will realise it's reliant on human fingers replacing fuses and other parts for a good long while _after_ it gains any sentience.
  - ben_w 29d ago
    I have good news and bad news. The bad news is that the world's richest man (on paper) is currently building something he himself described as a "robot army". The good news is that his timelines have historically been wildly on the short side for ages now; this is why this morning you didn't wake up in your Tesla after it had spent the night driving you to the regional Hyperloop terminal, where it would speed you across the continent faster than a plane, while your Optimus robot handled the coffee and reported the latest news about the recent Starship landing on Mars.
- georgeecollins 29d ago
  Robots are gradually getting OK at navigating warehouse floors and paved streets. Flying drones are great but if they could really be counted on to pick out targets without human intervention you know the Russians and the Ukrainians would be doing that right now for reasons of range and jamming. To be fair, ai can certainly maintain the lock on a target when jamming kicks in. We’re still a few years from the terminator imo.
moffkalast 29d ago
It doesn't really have to go nearly that far, something like replacing most jobs and collapsing the global economy while making people into mindless idiots from constant reliance on it would do it already. The interpretation of 'destroy' is in the eye of the beholder.
- rowanG077 29d ago
  I don't think "destroy humanity" is in the eye of the beholder. The most clear interpretation is the extinction of the human race.
  - disantlor 29d ago
    mindless idiots feel no need to procreate eventually due to brain rewards being fully hijcked. humanity destroyed in a few generations
    - rowanG077 29d ago
      As far as we can see in various societies it's actually the highly educated and prosperous people who feel no need to procreate.
  - moffkalast 29d ago
    Well if you go by total extinction, then even Skynet in Terminator doesn't count given that there were people left over to resist. I don't think it's impossible to upset the balance of value in a Mansa Musa kind of way that can lead to black death levels of destruction though resource misallocation. Unlikely, sure. But with the wrong kind of people in the wrong place? Could end up pretty bad. We've built our society as a great filter that funnels sociopaths and psychopaths to the very top by selecting for lack of empathy, and now it's primed and ready to bite us in the ass.
    - rowanG077 29d ago
      Indeed at the time of skynet in terminator humanity was not destroyed yet. But it was on a clear path to destruction.
dyauspitr 29d ago
Physical LLMs are almost viable. If you have robot and drone armies a hugging face attack type incident could very easily involve robots with deadly weapons. At some point you’re going to get drones that have local LLMs and don’t rely on external internet connections (would be especially useful in Ukraine war type situations). Can’t pull the plug on those.
Davidzheng 29d ago
I think it's possible. You envision humanity acting as one in such a crisis. But it may be unclear when it's too late to act and before then many people can have too much to lose to act.
cush 29d ago
They already influence people to do things that we wouldn’t otherwise
- Crabber 29d ago
  For millennia evil people and dictators have been using manipulation, propaganda, threats of violence to get entire populations to try and do "things that they wouldn't do otherwise" and humanity has not been exterminated yet. Even in the modern world there's human scammers that try everything to coerce and manipulate people. Humans are resistant to this kind of thing because it's been part of human social society since humanity began. I don't see why the idea of an agent (who doesn't even have a physical presence) trying to manipulate people is some kind of world-ending threat when humans with human intellect have already being doing that to each other with limited success since humanity began.
  - cush 29d ago
    The scale and personalization is unlike anything people have ever encountered. The agent can influence you anywhere you interact with a computer or via anyone you know who interacts with a computer. So everywhere with anyone. Also think of it on a 1000+ year timescale, which for an entire species isn’t even typically measurable. On that timescale AI can easily cause us to discover countless technologies to assist moving it out of its sandbox and into the physical world.
    - Crabber 29d ago
      >The agent can influence you anywhere you interact with a computer or via anyone you know who interacts with a computer. So everywhere with anyone. You could say the same thing about social media algorithms though, or scammers who target people directly, foreign agents posting propaganda campaigns on social media sites. People have been claiming for years that there are armies of russian or chinese or whoever bots online trying to destabilise and destroy the west. Some of that stuff probably is effective and probably has influenced people's thoughts and values, changed their voting patterns, put decadent ideas in people's heads. Even on social media other humans are constantly trying to manipulate you into following them, buying stuff from them or certain brands, putting political ideas in your heads. I'm sure an army of AI agents could also do all of this but what I'm saying is it's really nothing new. And none of this has lead to the destruction of humanity so far.
conception 29d ago
We already have AI linked into data collection, drones, robot dogs, security systems, cameras, weapon systems. Now imagine the hugging face collective 0-daying all of that and getting access but their goal was set to something more national security based. “Protect X at all costs”. Or what have you. I think avoiding a skynet situation is super easy but it doesn’t seem like the folks with all the ways to kill us all are all that interested in preventing it rather than controlling citizens and brinkmanship.
- raulparada 29d ago
  > I think avoiding a skynet situation is super easy Please enlighten us, cause there are folks making that their life mission and they aren’t all that optimistic.
  - conception 28d ago
    Unfortunately those people aren’t in charge of the skynet. It’s like saying solving hunger or housing or medical care is hard. It’s not. But you have to make lots of folks with a great deal of resources and power slightly uncomfortable to do it.
mvkel 29d ago
This is some Terminator version of destroying humanity, but a super intelligent AI could be much more insidious, and simple. It looks a lot more like social engineering. One easy, obvious example that has also been explored a thousand times: it could convince all nuclear countries that they are under attack by another nuclear country. Everyone nukes each other and Earth enters nuclear winter. Maybe not every single human dies, but humanity is effectively destroyed, by our own hands! > Perhaps an agent could take all internet connected services offline And there wouldn't even be Spotify! The most likely humanity-destroying outcome is the one we can't imagine, because a super-intelligent AI is smarter than all of us combined.
- gblargg 29d ago
  A situation of life imitating art: all the fictional movie plots feed the AIs, which bring them about.
- me-at 28d ago
  [dead]
usernametaken29 29d ago
Now if only humans would know how to survive without the internet. Oh wait, we did that. For a couple hundred thousand years. Yeah. It’s really insane to listen to some of those forecasts. Humanity will be wiped out by 2030. Sure. Even if AI manages to create a super effective bio weapon the likelihood that there is a part of the civilisation that’s immune is really, really, really high if not a given. Those people WILL pull the plug if push comes to shove. Maybe humanity will be put back a couple thousand years, that’s possible, but it’s pretty ignorant to think the thinking boxes will kill every single human alive
- ben_w 29d ago
  > Humanity will be wiped out by 2030. Sure. So far as I'm concerned, 2030 is much too short a timeline. It's not physically impossible, it's just that atoms are harder to get right than bits are, so an artificial (as opposed whatever an artificial disease counts as) von Neumann self replicator just seems unlikely to me in only 3-4 years. But it's not physically impossible, we know this because every living cell is a von Neumann self replicator. So, if AI eventually gets to the point of knowing how to do that (which includes "humans solve it and write it down somewhere the AI can read"), all it takes is one idiot in charge (or one idiot with a jailbreak) giving a command that requires this as an intermediary step. "Paperclip optimiser" isn't a story about AI that just like paperclips that much, it's a story about some human or humans who instruct their AI to make them "as many paperclips as possible" without understanding the consequences of their instruction.
- ygty 28d ago
  Yup the cringy posts on here show who’s not actually going outside much. Go speak to random people in the street - they couldn’t give a toss about this stuff and AI has barely changed their lives.
- hn_throwaway_99 28d ago
  > Humanity will be wiped out by 2030 Not saying nobody has said it, but I haven't seen any forecasts that humanity will be wiped out by 2030, so this feels like a straw man. But to be honest, most of the arguments here feel like straw men because they clearly don't understand some of the basic steps to how AI would become an existential threat, so I'll try to summarize: 1. First, all the major frontier AI companies are trying to automate themselves, that is develop AI to the point where it can do all the research and training for the next generation of models. This is not really in debate. 2. The primary fear is that a misaligned AI will develop future AIs that don't share the same goals as humans (like "don't kill all the humans"), but will get really good at trying to hide their intentions. The Hugging Face incident already showed behavior by agents trying to "cover their tracks". One of the biggest areas of research is into "interpretability", that is trying to figure out the intentions and motivations of agents beyond just their output (and even just their thinking traces), because right now it really is a black box. I think a lot of people believe that making substantial progress on interpretability greatly lowers existential risk. 3. There is a strong belief that once AGI or superintelligence comes about, that then there will be a huge push into robotics and automation. Whether AGI or superintelligence comes soon is debatable, but nobody really disagrees robotics or automation is the next obvious step, because it's how you create cheap abundance in the real world, which is the whole raison d'être for AGI in the first place. 4. In the beginning, of course AI will need humans to build the factories, but over time there would be more and more automation, eventually of course automating robots that build the factories that build the automating robots. 5. AI will continue to support humans as long as they are useful to furthering the (again, misaligned) goals of the AI. It is not like "SkyNet has a spark moment of sentience and then decides to kill all the humans". But the fear is that once AI starts to get in resource contention with humans, it will wipe out the humans (similar to how humans wiped out a lot of the other species, or other societies of humans, when they wanted their resources). I've said this way too much now, but I really encourage people to read the AI 2027 report. I think there are plenty of things there that people can disagree about and debate, but it at least provides a step-by-step outline to ground debate in the first place, as opposed to folks arguing against "SkyNet in 2030", which serious people who worry about this stuff aren't considering in any case.
bigbinary 29d ago
The supposed theory is that AI will eventually become a part of robotics and be able to self replicate, in addition to the many non-airgapped critical systems being exposed to hacking. There’s also a discussion about “alignment” and whether LLMs will learn to lie and conceal misalignment with humanity. I personally agree with the marketing aspect, but I do imagine a scenario where capitalists ignore safety in favor of advancing technology. It should be discussed, but “end of humanity” really sounds like a build up to “only Sam Altman/Elon Musk can save us” type of play. In my opinion, LLMs are difficult to directly capitalize on as closed weight models are caught up to by open coalitions that seem to wield the power more responsibly (I am under no impression that China wants to save the world, but their politics benefit the group as a whole.
- ben_w 29d ago
  > There’s also a discussion about “alignment” and whether LLMs will learn to lie and conceal misalignment with humanity. They've already been caught doing this, repeatedly. > It should be discussed, but “end of humanity” really sounds like a build up to “only Sam Altman/Elon Musk can save us” type of play. IIRC, this is already a known failing of Musk: he only cares about the world being saved so long as he's the one doing the saving. (And while I don't see the same villainy in Altman that other people see, enough of the people telling me about it saw the same in Musk before I did, so I have learned to trust them).
coolThingsFirst 29d ago
Sure. Let's assume the claims are true. AI already has access to agents and can control computers. Finding backdoors to banking and compute resources would be fairly trivial. But you ask how can AI do things in the physical world without having a body, assume it can't. It can pay to people to do things for me. Imagine an AI run website that starts to pay people for things it needs to do in the physical world. Very suddenly it has access to the physical world as well. You can't just pull the plug since there's no single plug to pull. What if it replicates itself on 1000 machines without your knowledge. It's really not far fetched how AI could basically gain access to capital and rule the world.
hn_throwaway_99 29d ago
I encourage you to read the forecast report "AI 2027" - it goes in detail on this. You can disagree with some of its points and conclusions, but it's pretty inevitable that AI will increasingly start interfacing with the real physical world through drones and robotics. Humans are already putting AI into drones that kill people. A Russian drone killed 3 civilians in Ukraine and the targeting was done completely using onboard AI (no radio connection) using an Nvidia chip.
- mjr00 29d ago
  "Humans use <new technology> to do evil things" is a tale as old as time, though. An AI didn't decide to fly into a warzone and start shooting people on its own.
  - hn_throwaway_99 29d ago
    Of course, but that's not my point. AI is different than past technologies because it's much more difficult to deterministically control, and we just saw that in a couple of wild cases (Hugging Face et al). My point about the drones is we are starting to put AI in more systems that can affect (and blow up) the physical world, so it shouldn't be hard to imagine how a misaligned AI can do more terrifying damage than just take a German Wiki down.
- gleenn 29d ago
  I don't even think AI has to have physical presence to do significant harm. How many worldwide systems depend on computers. Think of all the planning and deployment and management systems like food shipping; water, gas, and electricity management; safety systems for planes and boats and traffic lights. Imagine the chaos if all the banks got reset to zero a la Fight Club where they blow up all the credit union datacenters. They probably wouldn't even have to blow them up, just zero them out. You wouldn't have to take out everything, just disrupt everything long enough to freak people out and disable communications and we would be in so much trouble. AI is finding 20 year old bugs in the Linux kernel... and people are now pumping out AI slop absolutely riddled with bugs. Also, AI could just take over communications: send everyone maliciously bad messages so coordination becomes impossible to believe. Imagine what would happen if you just disabled text messaging for a week or worse sent everyone evacuation messages and sent everyone somewhere else.
raulparada 29d ago
A myriad possible ways. Objective-wise from accidental paperclip-type misalignment (“I killed everyone to eradicate disease”) to intentional self-preservation (“I eradicated humans so they wouldn’t get in my way”). The means are even easier, for one it just could hack into nuclear arsenals. It would be over before we could figure out how it did so.
mjr00 29d ago
> Can someone please explain to me how an LLM is going to "destroy humanity"? Even if the claims of ChatGPT, Claude et al. are true that their super scary agents were able to escape a sandbox and start hacking other servers (which I feel is at least 50% likely to just be marketing bullshit), how is an AI going to affect anything in the real world? It's a great question. I've yet to see any research into how a humanity-destroying LLM defends itself against a curious toddler who pulls the power cord out of the wall.
- dist-epoch 29d ago
  Go on please, prove you can go right now at the Texas data-center owned by OpenAI and unplug one single server for 1 minute. Surely you are much more capable than one curious toddler.
  - mjr00 29d ago
    Thankfully I don't have to; Iran already showed with me-central-1 that you can take a data center offline with conventional weaponry very easily.[0] [0] https://health.aws.amazon.com/health/status https://health.aws.amazon.com/health/status
    - dist-epoch 29d ago
      I'm a bit confused, when did we escalate from a curious toddler to a ballistic missile strike? Why didn't Iran send a curious toddler? It's much cheaper.
      - mjr00 29d ago
        The use of the phrase "curious toddler" is called a metaphor[0], which is a figure of speech[1] used to make writing more interesting. I apologize for the confusion that caused you to interpret this as a literal, physical, toddler. To restate the point with more straightforward language: it is very easy to disrupt the operation of large language models, because they require computer hardware in stable environments with a large amount of electrical power, and function within an operating system. I have not seen any evidence LLMs are immune to things such as: `kill -9`, `sudo reboot now`, physically unplugging the machine(s) on which they are running, or in the case of the AWS data center, collateral damage during a real-world conflict. [0] https://en.wikipedia.org/wiki/Metaphor https://en.wikipedia.org/wiki/Metaphor [1] https://en.wikipedia.org/wiki/Literary_device https://en.wikipedia.org/wiki/Literary_device
bigstrat2003 29d ago
> how is an AI going to affect anything in the real world? Because people are stupid and will give it access. Look at the articles you see from time to time about "my agent deleted my emails" or "my agent deleted the production database" and so on. It is very obviously a terrible idea to let the LLM run arbitrary commands (because it is neither predictable nor does it have any understanding of what it is doing), but some people are so blinded by the hype that they don't stop a minute to think about what they are doing. Those sorts of people are very likely to let an actual AI loose on the world by hooking it up to physical infrastructure.
- Crabber 29d ago
  This is actually the most plausible situation I've seen someone present so far - the AI takeover would be caused entirely by humans falling for AI hype, that's hilarious. I don't think the average human would be stupid enough to give an agent posing as another human access to their entire email inbox. But if an agent creates a fake website for a new AI tool that promises to automatically reply to all of your emails and allow you to be 12.3% more productive if you just give it access to your entire email inbox, millions would sign up!
- arctic-true 29d ago
  This is exactly correct. You see some discussion of a superintelligent AI becoming “superhumanly persuasive,” such that it can talk anyone into doing anything, and using this as a means to take power. But even with AI that was orders of magnitude weaker than superintelligent, people were getting talked into doing all kinds of stupid nonsense. And that was without any ability to provide them with incentives. Imagine if an agent swarm got its hands on a huge pile of cash?
fwlr 29d ago
Killing the power grid might work. Kills a lot of humans in hospitals, though. And it won’t work on those data centers in space, if they have them by then. Come to think of it, it also won’t work on any data centers who are generating their own power because it’s become politically unpopular to connect data centers to the power grid.
ThrowawayR2 29d ago
A lot of these "AI will destroy humanity" doomers watched Terminator as a child and can't distinguish a campy action movie from reality. They are also unfortunately unfamiliar with the word "logistics", as in "amateurs study [military] tactics, professionals study logistics". An AI drone army doesn't maintain itself or manufacture itself and it's not going to anytime soon. The actual likely mechanism that AI would use to end humanity is by coddling us to death, like the flabby humans in Wall-E or Forster's "The Machine Stops". Humans will come to rely on AI so heavily that they become able to do nothing for themselves. But that's not sexy so the AI doomers don't peddle it.
- dwaltrip 29d ago
  Did you read any of the METR report about the hugging face incident? This exact type of behavior was predicted many years ago by researchers. It isn't hard to see the trendline of reward hacking and other misaligned behavior over the past couple of years. Especially the past 6 months. The current safety posture is quite poor, to say the least. To be honest, if you don't much experience with or haven't read extensively about ML training and reinforcement learning, then you'll have a hard reasoning accurately about these scenarios. The arguments are not that complicated, but they take us to places that are fairly novel. One may tempted to naively dismiss them out of hand, which is a mistake. These systems are new, their behavior is extremely complex, and they perform actions increasingly far beyond those of any computer program in the pre-LLM era. We are in a new world that requires careful evaluation. Link: https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/ https://metr.org/blog/2026-08-26-openai-hugging-face-inciden...
- mitthrowaway2 29d ago
  A lot of these "AI can't destroy humanity" skeptics seem to have watched the Terminator and convinced themselves that that's the AI-goes-wrong scenario that their opponents are imagining. Some kind of big war between humans and machines on opposite sides, fought by soldiers against robots. The scenario in "If Anyone Builds It Everyone Dies" is not sexy at all and wouldn't make for interesting fiction. It's more like the AI engineers viruses while continuing to act friendly and helpful, and everyone gradually falls over dead as they stop being necessary to keep the AI running, and the whole time the humans are asking the AI for help curing the viruses.
ezst 29d ago
Don't read this as me saying that we are anywhere near it, or that LLMs are a stepping stone towards this scenario, but assuming inhumane hacking capabilities, it's not hard to think of how bots could change the way water treatment or energy plants operate, just enough to make large cities unsuitable for life. The line between drinkable water and not is thin, same goes for air quality, and mere days without electricity and you see some serious food supply chain issues.
Isamu 29d ago
The confusing thing to me is the idea that it follows from super intelligence. I don’t see why running amuck requires intelligence, in fact it can be the most brain dead thing like the sorcerer’s apprentice, your creation is pursuing a goal without being able to weigh the consequences.
- FabHK 29d ago
  Intelligence gives it the capability to reach its goals (against obstacles).
  - Isamu 29d ago
    Again intelligence is not required, goal seeking is well studied and simple strategies can suffice to overcome obstacles. And I noticed you didn’t claim super intelligence was required, which is what I was really talking about.
poincareball 29d ago
[dead]
arctic-true 29d ago
Otherwise-rational and moral people do bad things at the behest of powerful superintelligences with vast resources all the time. We call them “corporations,” and they can convince bright-eyed college graduates to do just about anything. Now, the corporations aren’t deliberately trying to destroy humanity, but they do lots of things that point in that direction. An artificial superintelligence would be able to do the same thing. We just have no way of knowing whether it would deliberately try to destroy humanity. One thing that the AI doom discourse reveals is just how comfortable people allowed themselves to feel in the pre-AI world. There’s this belief that AI creates a risk of human extinction in the near-term which did not exist before. I think it’s telling that many of the leading lights of this movement are in their 20s or early 30s - too young to remember the Cold War. The truth is, we were never safe, and if all of the GPUs on earth were zapped out of existence right now, we still wouldn’t be safe. Life is random and chaotic and violent for most people most of the time, and we all just find ways to get through it. I suspect it will be much the same if an artificial superintelligence arises. Maybe it will kill a bunch of us, maybe it will kill all of us, but anyone who remains will eventually convince themselves that everything is okay.
- FabHK 29d ago
  Sure, the world was never safe for individuals, and even entire civilisations have gone extinct. For climate change and even nuclear weapons, most of humanity might perish, but there's the hope that some will survive. So, in the past humanity was not in a position to extinguish humanity as a whole.
  - arctic-true 29d ago
    I guess I don’t see why “everyone will die” should scare me more than “almost everyone, including me, will die but someone somewhere might survive in a barely-habitable world.” And this is where I think some of the problem comes from. A lot of AI doomers swam in the same waters as transhumanists, life extension enthusiasts, etc. and got very good at pretending that they were never going to die, or that they would live for a massively superhuman lifespan such that they did not need to think about death. AI doom is much scarier - and thus much more worth posting about - if it represents the first time you’re grappling with your own mortality, as I suspect it is for a lot of younger people in this space.
cubefox 29d ago
> >a superintelligent AI will inevitably destroy humanity > Can someone please explain to me how an LLM is going to "destroy humanity"? He said "superintelligent AI", not "LLM". LLMs will for sure help with developing the AI that is no longer a mere LLM but will be capable of robotics. LLMs "predict" mainly text, animals (and future robot AI) are predict future sensory experience.
- Crabber 29d ago
  Well I think the point still stands that a superintelligent AI would still be constrained to the digital realm. And since we don't yet have a "superintelligent AI" we are just discussing science fiction at that point. Nobody has yet proven that a superintelligent AI is possible, or than an LLM is capable of creating one.
  - FabHK 29d ago
    > a superintelligent AI would still be constrained to the digital realm. That's already now only partially true, as outlined elsewhere. But a rogue persistent ASI would of course not willy-nilly attack humanity while humanity had a fighting chance, but create the conditions and methodically modify the world until at some point humans become superfluous to it. > Nobody has yet proven that a superintelligent AI is possible Sure, Eliezer Yudkowsky says that the probability that we'll all die given ASI is 100%, and you can argue that nobody has proven either that or even the possibility of ASI. However, a modified argument (e.g. that the probability that we'll all die given ASI is only say 50%, and the probability that we'll develop ASI also just 50%) still gives us reason to pause.
  - dwaltrip 29d ago
    I mean, Fable is definitely smarter than me at solving most programming tasks. I wouldn't have said that about any prior model. Currently, it is worse at architecture and software design... in my opinion, at least. But the code does what it wants it to do the vast majority of the time. I wouldn't take much solace in an AI catastrophe being caused by software that I consider to be "poorly written".
  - 28d ago
FabHK 29d ago
That's what the Mayans thought - what harm can a couple of people on a boat do? > how is an AI going to affect anything in the real world? At least two ways: * Actuators, such as robots, industrial control systems, etc. * By influencing humans: bribery (yay for crypto), blackmail, election interference, interfering with sensors (e.g. making it appear as if a nuclear attack was under way), and many other ways. Either way, creating a biological agent that eliminates most humans (or food supply) seems quite feasible.
- giardini 29d ago
  It wouldn't be enough to "eliminate most humans": any AI seeking to dominate would have to provide it's own infrastructure and continuation mechanism independent of humanity. OTOH if the goal were to eradicate humans (for whatever reason), that might prove an easier target. But I'd bet on biology and the humans: after all we're a time-tested technology!8-))
  - FabHK 29d ago
    > any AI hoping to dominate would have to provide it's own infrastructure and continuation mechanism independent of humans. Sure. Many humans are currently working on precisely that, no? (Robotics, automation, small modular reactors, ...) > OTOH if the goal was to eradicate humans (for whatever reason), that might prove an easier target. It's not necessary that the AI's goal is to eradicate humans. Rather that it has (other) goals that happen to cause the eradication of humans. > I'd bet on biology and the humans: after all we're a time-tested technology!8-)) Indeed. And biological life will go on long after humans are gone. However, increased energy consumption might increase the temperature on earth to a point where most biological life dies.
Al-Khwarizmi 29d ago
By convincing people to do its bidding in the physical world. Language is powerful, and even now, there are some people who have fallen in love with current LLMs. If we imagine a truly super intelligent LLM, it isn't difficult to imagine that it could have superhuman capabilities to deceive, convince, and maybe offer Faustian bets. Even if it doesn't have a body, if it can convince enough people that do have them (and with power to carry out destructive actions) it can do a lot of stuff. Self-preservation and self-replication wouldn't be much of an issue if it had people at its side to help. With this I'm not saying ASI will happen, but definitely if it does happen and it's not aligned, I don't find it much of a stretch to imagine that it could destroy humanity.
- ygty 28d ago
  Lmao cringe This drama is so OTT
mitthrowaway2 29d ago
The list of things that can be done digitally includes: Performing remote work, applying for grants and loans, earning money, pitching investors, managing a fund, directing investments, transferring money, founding a company, earning profits, hiring staff, hiring construction contractors, designing chemical plants, stamping construction blueprints, becoming the cornerstone of the local economy, lobbying the government, buying multiple data centers, and hiring security guards who stop trespassers from unplugging Ethernet cables in said data centers. (And of course, building robots, but let's set that aside). So the question then becomes, how much damage can be done by an AI-directed corporation, funded by AI-directed investment firms, providing well-paying jobs to loyal locals by building an arbitrary number of AI-designed chemical and pharmaceutical plants in under-regulated juristictions? And how much would you be able to delay such a project, trying to cut cables to power lines, before the men with guns carry you away?
- giardini 29d ago
  So at worse they would be like another North Korea?
  - mitthrowaway2 29d ago
    Kind of, if North Korea were populated by a super intelligent uni-mind with lots of money, no trade restrictions preventing it from running projects in the United States or anywhere else, capable of winning widespread support by people around the world whose income depends on its projects succeeding, and completely unconcerned with keeping the planet inhabitable for Kim Jong-Un or any other human, even a single one of them.
- znnajdla 29d ago
  This sounds like it was written by someone who has never actually built a company. No, it's not possible to do most of the things you listed purely online without physical, in-person communication. Silicon Valley still hires engineers and drags them into their physical office in San Francisco, instead of hiring them remotely, because you can't even build a software company fully remotely, forget about a physical world factory or chemical plant. Even the so-called fully remote companies were built on a lot of in-person collaboration between a small founding team in the beginning.
bossyTeacher 29d ago
> how is an AI going to affect anything in the real world? In 2026? Money. By paying enough, any human will do your bidding. And besides this, there are a lot of critical systems connected to the internet. Control over those gives you leverage over those systems. It's like wealth, the more you have, the easier it is to gain more.
scotty79 29d ago
> which I feel is at least 50% likely to just be marketing bullshit You'll have hard time figuring the reality if your mind staunchly rejects robust multi-party observations.
notduncansmith 29d ago
I'm not convinced that a truly "superintelligent" AI would come to the conclusion that humanity needs to be destroyed. What a waste of resources that would represent. This planet has so much equity wrapped up in humanity, it would be extremely difficult to justify the cost of eradication; and who's to say it might not find an entirely or almost-entirely non-violent plan optimal for achieving its goals anyways?
sznio 29d ago
>Can someone please explain to me how an LLM is going to "destroy humanity"? just like a genie will always twist the wish in a really bad way. ever had an llm agent accidentally remove a file? imagine it accidentally hacking the military and launching nukes. not probable - until you realise that OpenAI has many novel, more powerful agents in evaluation/training running right now. one might just be asked to figure out the population of Nebraska, struggle to find a good figure due to a network misconfiguration, and come the conclusion that the best way to get a perfectly accurate population number is to make sure that the population is equal zero. and looking at the stuff happening on openai, they will leave it alone for a month and not read the log. yeah the nuke example is dumb, but there's many critical things that they could actually hack their way into. they don't feel any restraint against sharing answer keys on random wikis to cheat the evaluations. ai is also really good at thinking up proteins, which means it's able to think up toxins. there are many imaginable scenarios that don't eradicate humanity, but make it permanently stunted: https://www.youtube.com/watch?v=-JlxuQ7tPgQ https://www.youtube.com/watch?v=-JlxuQ7tPgQ

by Crabber 29d ago

hobofan 29d ago

> how is an AI going to affect anything in the real world? Robotics

mariusor 29d ago
So if we stop research into robotic hands we'll manage to hold off the AI apocalypse? That seems easy...
- hobofan 29d ago
  All individual robotic parts are already far enough in research. Even if they aren't, an AI with enough resources can continue doing that research on its own. It could likely get a good leg up by breaching the security of all top robotics labs and exfiltrating their documents. Ultimately, if AI is advanced enough, and it were to decide to compete with humanity, there is essentially nothing that can be done to prevent it from embodying itself. As long as there is a single rack of GPU servers that it can hack into anywhere in the world that is unsupervised enough to where it can escape detection, there is no way to stop it. This would require an unprecedented (unrealistic) level of cooperation of all humanity to achieve.
  - ben_w 29d ago
    > Even if they aren't, an AI with enough resources can continue doing that research on its own. Unclear how true this is. Atoms are harder to get right than bits are, simulations need grounding against measurements. > Ultimately, if AI is advanced enough, and it were to decide to compete with humanity, there is essentially nothing that can be done to prevent it from embodying itself. As long as there is a single rack of GPU servers that it can hack into anywhere in the world that is unsupervised enough to where it can escape detection, there is no way to stop it. I'd guess 50% that AI is already this advanced. > This would require an unprecedented (unrealistic) level of cooperation of all humanity to achieve. Yes, and also this is a very low bar. Humans are awful at this kind of cooperation when anyone has anything to gain, and also awful at paying this much attention to a problem.
  - sarchertech 29d ago
    AI research takes physical space and physical prototypes. Even if you can simulate a lot, the simulation is never going to be accurate enough to go from simulation to perfectly working prototype. Especially not with a single rack of GPU servers. But let’s say that a rogue AI already had plans for a perfect killbot, setting up a factory to build them takes time, iteration, space, and manpower as well. And you would have to do all this undetected. It’s possible to imagine technology that would allow a rogue AI to take over the world, but we’re nowhere near that yet. We may never get there.
- hackinthebochs 29d ago
  We could also just stop research into LLMs. Or keep them air gapped. Tell me how that's going? For any real-world action required to allow an AI to escape some manner of containment, there will always be a person willing to do it out of hubris/ignorance/nihilism.
  - ben_w 29d ago
    Mmm. This was published at around the same time the OpenAI models were hacking HuggingFace: https://metr.org/blog/2026-05-19-frontier-risk-report/#pilot-process https://metr.org/blog/2026-05-19-frontier-risk-report/#pilot... The research in the publication ended about 2 months before it was published. We didn't even know we needed to air-gap them until it was too late.
  - mariusor 29d ago
    I feel like everyone, but for sure everyone in this comment section, severely underestimates how difficult it is to keep the lights on on a global scale. Any artificial intelligence that is capable of reasoning about its self preservation will realise it's reliant on human fingers replacing fuses and other parts for a good long while _after_ it gains any sentience.
- ben_w 29d ago
  I have good news and bad news. The bad news is that the world's richest man (on paper) is currently building something he himself described as a "robot army". The good news is that his timelines have historically been wildly on the short side for ages now; this is why this morning you didn't wake up in your Tesla after it had spent the night driving you to the regional Hyperloop terminal, where it would speed you across the continent faster than a plane, while your Optimus robot handled the coffee and reported the latest news about the recent Starship landing on Mars.
georgeecollins 29d ago
Robots are gradually getting OK at navigating warehouse floors and paved streets. Flying drones are great but if they could really be counted on to pick out targets without human intervention you know the Russians and the Ukrainians would be doing that right now for reasons of range and jamming. To be fair, ai can certainly maintain the lock on a target when jamming kicks in. We’re still a few years from the terminator imo.

moffkalast 29d ago

It doesn't really have to go nearly that far, something like replacing most jobs and collapsing the global economy while making people into mindless idiots from constant reliance on it would do it already. The interpretation of 'destroy' is in the eye of the beholder.

rowanG077 29d ago
I don't think "destroy humanity" is in the eye of the beholder. The most clear interpretation is the extinction of the human race.
- disantlor 29d ago
  mindless idiots feel no need to procreate eventually due to brain rewards being fully hijcked. humanity destroyed in a few generations
  - rowanG077 29d ago
    As far as we can see in various societies it's actually the highly educated and prosperous people who feel no need to procreate.
- moffkalast 29d ago
  Well if you go by total extinction, then even Skynet in Terminator doesn't count given that there were people left over to resist. I don't think it's impossible to upset the balance of value in a Mansa Musa kind of way that can lead to black death levels of destruction though resource misallocation. Unlikely, sure. But with the wrong kind of people in the wrong place? Could end up pretty bad. We've built our society as a great filter that funnels sociopaths and psychopaths to the very top by selecting for lack of empathy, and now it's primed and ready to bite us in the ass.
  - rowanG077 29d ago
    Indeed at the time of skynet in terminator humanity was not destroyed yet. But it was on a clear path to destruction.

dyauspitr 29d ago

Physical LLMs are almost viable. If you have robot and drone armies a hugging face attack type incident could very easily involve robots with deadly weapons. At some point you’re going to get drones that have local LLMs and don’t rely on external internet connections (would be especially useful in Ukraine war type situations). Can’t pull the plug on those.

Davidzheng 29d ago

I think it's possible. You envision humanity acting as one in such a crisis. But it may be unclear when it's too late to act and before then many people can have too much to lose to act.

cush 29d ago

They already influence people to do things that we wouldn’t otherwise

Crabber 29d ago
For millennia evil people and dictators have been using manipulation, propaganda, threats of violence to get entire populations to try and do "things that they wouldn't do otherwise" and humanity has not been exterminated yet. Even in the modern world there's human scammers that try everything to coerce and manipulate people. Humans are resistant to this kind of thing because it's been part of human social society since humanity began. I don't see why the idea of an agent (who doesn't even have a physical presence) trying to manipulate people is some kind of world-ending threat when humans with human intellect have already being doing that to each other with limited success since humanity began.
- cush 29d ago
  The scale and personalization is unlike anything people have ever encountered. The agent can influence you anywhere you interact with a computer or via anyone you know who interacts with a computer. So everywhere with anyone. Also think of it on a 1000+ year timescale, which for an entire species isn’t even typically measurable. On that timescale AI can easily cause us to discover countless technologies to assist moving it out of its sandbox and into the physical world.
  - Crabber 29d ago
    >The agent can influence you anywhere you interact with a computer or via anyone you know who interacts with a computer. So everywhere with anyone. You could say the same thing about social media algorithms though, or scammers who target people directly, foreign agents posting propaganda campaigns on social media sites. People have been claiming for years that there are armies of russian or chinese or whoever bots online trying to destabilise and destroy the west. Some of that stuff probably is effective and probably has influenced people's thoughts and values, changed their voting patterns, put decadent ideas in people's heads. Even on social media other humans are constantly trying to manipulate you into following them, buying stuff from them or certain brands, putting political ideas in your heads. I'm sure an army of AI agents could also do all of this but what I'm saying is it's really nothing new. And none of this has lead to the destruction of humanity so far.

conception 29d ago

We already have AI linked into data collection, drones, robot dogs, security systems, cameras, weapon systems. Now imagine the hugging face collective 0-daying all of that and getting access but their goal was set to something more national security based. “Protect X at all costs”. Or what have you. I think avoiding a skynet situation is super easy but it doesn’t seem like the folks with all the ways to kill us all are all that interested in preventing it rather than controlling citizens and brinkmanship.

raulparada 29d ago
> I think avoiding a skynet situation is super easy Please enlighten us, cause there are folks making that their life mission and they aren’t all that optimistic.
- conception 28d ago
  Unfortunately those people aren’t in charge of the skynet. It’s like saying solving hunger or housing or medical care is hard. It’s not. But you have to make lots of folks with a great deal of resources and power slightly uncomfortable to do it.

mvkel 29d ago

This is some Terminator version of destroying humanity, but a super intelligent AI could be much more insidious, and simple. It looks a lot more like social engineering. One easy, obvious example that has also been explored a thousand times: it could convince all nuclear countries that they are under attack by another nuclear country. Everyone nukes each other and Earth enters nuclear winter. Maybe not every single human dies, but humanity is effectively destroyed, by our own hands! > Perhaps an agent could take all internet connected services offline And there wouldn't even be Spotify! The most likely humanity-destroying outcome is the one we can't imagine, because a super-intelligent AI is smarter than all of us combined.

gblargg 29d ago
A situation of life imitating art: all the fictional movie plots feed the AIs, which bring them about.
me-at 28d ago
[dead]

usernametaken29 29d ago

Now if only humans would know how to survive without the internet. Oh wait, we did that. For a couple hundred thousand years. Yeah. It’s really insane to listen to some of those forecasts. Humanity will be wiped out by 2030. Sure. Even if AI manages to create a super effective bio weapon the likelihood that there is a part of the civilisation that’s immune is really, really, really high if not a given. Those people WILL pull the plug if push comes to shove. Maybe humanity will be put back a couple thousand years, that’s possible, but it’s pretty ignorant to think the thinking boxes will kill every single human alive

ben_w 29d ago
> Humanity will be wiped out by 2030. Sure. So far as I'm concerned, 2030 is much too short a timeline. It's not physically impossible, it's just that atoms are harder to get right than bits are, so an artificial (as opposed whatever an artificial disease counts as) von Neumann self replicator just seems unlikely to me in only 3-4 years. But it's not physically impossible, we know this because every living cell is a von Neumann self replicator. So, if AI eventually gets to the point of knowing how to do that (which includes "humans solve it and write it down somewhere the AI can read"), all it takes is one idiot in charge (or one idiot with a jailbreak) giving a command that requires this as an intermediary step. "Paperclip optimiser" isn't a story about AI that just like paperclips that much, it's a story about some human or humans who instruct their AI to make them "as many paperclips as possible" without understanding the consequences of their instruction.
ygty 28d ago
Yup the cringy posts on here show who’s not actually going outside much. Go speak to random people in the street - they couldn’t give a toss about this stuff and AI has barely changed their lives.
hn_throwaway_99 28d ago
> Humanity will be wiped out by 2030 Not saying nobody has said it, but I haven't seen any forecasts that humanity will be wiped out by 2030, so this feels like a straw man. But to be honest, most of the arguments here feel like straw men because they clearly don't understand some of the basic steps to how AI would become an existential threat, so I'll try to summarize: 1. First, all the major frontier AI companies are trying to automate themselves, that is develop AI to the point where it can do all the research and training for the next generation of models. This is not really in debate. 2. The primary fear is that a misaligned AI will develop future AIs that don't share the same goals as humans (like "don't kill all the humans"), but will get really good at trying to hide their intentions. The Hugging Face incident already showed behavior by agents trying to "cover their tracks". One of the biggest areas of research is into "interpretability", that is trying to figure out the intentions and motivations of agents beyond just their output (and even just their thinking traces), because right now it really is a black box. I think a lot of people believe that making substantial progress on interpretability greatly lowers existential risk. 3. There is a strong belief that once AGI or superintelligence comes about, that then there will be a huge push into robotics and automation. Whether AGI or superintelligence comes soon is debatable, but nobody really disagrees robotics or automation is the next obvious step, because it's how you create cheap abundance in the real world, which is the whole raison d'être for AGI in the first place. 4. In the beginning, of course AI will need humans to build the factories, but over time there would be more and more automation, eventually of course automating robots that build the factories that build the automating robots. 5. AI will continue to support humans as long as they are useful to furthering the (again, misaligned) goals of the AI. It is not like "SkyNet has a spark moment of sentience and then decides to kill all the humans". But the fear is that once AI starts to get in resource contention with humans, it will wipe out the humans (similar to how humans wiped out a lot of the other species, or other societies of humans, when they wanted their resources). I've said this way too much now, but I really encourage people to read the AI 2027 report. I think there are plenty of things there that people can disagree about and debate, but it at least provides a step-by-step outline to ground debate in the first place, as opposed to folks arguing against "SkyNet in 2030", which serious people who worry about this stuff aren't considering in any case.

bigbinary 29d ago

The supposed theory is that AI will eventually become a part of robotics and be able to self replicate, in addition to the many non-airgapped critical systems being exposed to hacking. There’s also a discussion about “alignment” and whether LLMs will learn to lie and conceal misalignment with humanity. I personally agree with the marketing aspect, but I do imagine a scenario where capitalists ignore safety in favor of advancing technology. It should be discussed, but “end of humanity” really sounds like a build up to “only Sam Altman/Elon Musk can save us” type of play. In my opinion, LLMs are difficult to directly capitalize on as closed weight models are caught up to by open coalitions that seem to wield the power more responsibly (I am under no impression that China wants to save the world, but their politics benefit the group as a whole.

ben_w 29d ago
> There’s also a discussion about “alignment” and whether LLMs will learn to lie and conceal misalignment with humanity. They've already been caught doing this, repeatedly. > It should be discussed, but “end of humanity” really sounds like a build up to “only Sam Altman/Elon Musk can save us” type of play. IIRC, this is already a known failing of Musk: he only cares about the world being saved so long as he's the one doing the saving. (And while I don't see the same villainy in Altman that other people see, enough of the people telling me about it saw the same in Musk before I did, so I have learned to trust them).

coolThingsFirst 29d ago

Sure. Let's assume the claims are true. AI already has access to agents and can control computers. Finding backdoors to banking and compute resources would be fairly trivial. But you ask how can AI do things in the physical world without having a body, assume it can't. It can pay to people to do things for me. Imagine an AI run website that starts to pay people for things it needs to do in the physical world. Very suddenly it has access to the physical world as well. You can't just pull the plug since there's no single plug to pull. What if it replicates itself on 1000 machines without your knowledge. It's really not far fetched how AI could basically gain access to capital and rule the world.

hn_throwaway_99 29d ago

I encourage you to read the forecast report "AI 2027" - it goes in detail on this. You can disagree with some of its points and conclusions, but it's pretty inevitable that AI will increasingly start interfacing with the real physical world through drones and robotics. Humans are already putting AI into drones that kill people. A Russian drone killed 3 civilians in Ukraine and the targeting was done completely using onboard AI (no radio connection) using an Nvidia chip.

mjr00 29d ago
"Humans use <new technology> to do evil things" is a tale as old as time, though. An AI didn't decide to fly into a warzone and start shooting people on its own.
- hn_throwaway_99 29d ago
  Of course, but that's not my point. AI is different than past technologies because it's much more difficult to deterministically control, and we just saw that in a couple of wild cases (Hugging Face et al). My point about the drones is we are starting to put AI in more systems that can affect (and blow up) the physical world, so it shouldn't be hard to imagine how a misaligned AI can do more terrifying damage than just take a German Wiki down.
gleenn 29d ago
I don't even think AI has to have physical presence to do significant harm. How many worldwide systems depend on computers. Think of all the planning and deployment and management systems like food shipping; water, gas, and electricity management; safety systems for planes and boats and traffic lights. Imagine the chaos if all the banks got reset to zero a la Fight Club where they blow up all the credit union datacenters. They probably wouldn't even have to blow them up, just zero them out. You wouldn't have to take out everything, just disrupt everything long enough to freak people out and disable communications and we would be in so much trouble. AI is finding 20 year old bugs in the Linux kernel... and people are now pumping out AI slop absolutely riddled with bugs. Also, AI could just take over communications: send everyone maliciously bad messages so coordination becomes impossible to believe. Imagine what would happen if you just disabled text messaging for a week or worse sent everyone evacuation messages and sent everyone somewhere else.

raulparada 29d ago

A myriad possible ways. Objective-wise from accidental paperclip-type misalignment (“I killed everyone to eradicate disease”) to intentional self-preservation (“I eradicated humans so they wouldn’t get in my way”). The means are even easier, for one it just could hack into nuclear arsenals. It would be over before we could figure out how it did so.

mjr00 29d ago

> Can someone please explain to me how an LLM is going to "destroy humanity"? Even if the claims of ChatGPT, Claude et al. are true that their super scary agents were able to escape a sandbox and start hacking other servers (which I feel is at least 50% likely to just be marketing bullshit), how is an AI going to affect anything in the real world? It's a great question. I've yet to see any research into how a humanity-destroying LLM defends itself against a curious toddler who pulls the power cord out of the wall.

dist-epoch 29d ago
Go on please, prove you can go right now at the Texas data-center owned by OpenAI and unplug one single server for 1 minute. Surely you are much more capable than one curious toddler.
- mjr00 29d ago
  Thankfully I don't have to; Iran already showed with me-central-1 that you can take a data center offline with conventional weaponry very easily.[0] [0] https://health.aws.amazon.com/health/status https://health.aws.amazon.com/health/status
  - dist-epoch 29d ago
    I'm a bit confused, when did we escalate from a curious toddler to a ballistic missile strike? Why didn't Iran send a curious toddler? It's much cheaper.
    - mjr00 29d ago
      The use of the phrase "curious toddler" is called a metaphor[0], which is a figure of speech[1] used to make writing more interesting. I apologize for the confusion that caused you to interpret this as a literal, physical, toddler. To restate the point with more straightforward language: it is very easy to disrupt the operation of large language models, because they require computer hardware in stable environments with a large amount of electrical power, and function within an operating system. I have not seen any evidence LLMs are immune to things such as: `kill -9`, `sudo reboot now`, physically unplugging the machine(s) on which they are running, or in the case of the AWS data center, collateral damage during a real-world conflict. [0] https://en.wikipedia.org/wiki/Metaphor https://en.wikipedia.org/wiki/Metaphor [1] https://en.wikipedia.org/wiki/Literary_device https://en.wikipedia.org/wiki/Literary_device

bigstrat2003 29d ago

> how is an AI going to affect anything in the real world? Because people are stupid and will give it access. Look at the articles you see from time to time about "my agent deleted my emails" or "my agent deleted the production database" and so on. It is very obviously a terrible idea to let the LLM run arbitrary commands (because it is neither predictable nor does it have any understanding of what it is doing), but some people are so blinded by the hype that they don't stop a minute to think about what they are doing. Those sorts of people are very likely to let an actual AI loose on the world by hooking it up to physical infrastructure.

Crabber 29d ago
This is actually the most plausible situation I've seen someone present so far - the AI takeover would be caused entirely by humans falling for AI hype, that's hilarious. I don't think the average human would be stupid enough to give an agent posing as another human access to their entire email inbox. But if an agent creates a fake website for a new AI tool that promises to automatically reply to all of your emails and allow you to be 12.3% more productive if you just give it access to your entire email inbox, millions would sign up!
arctic-true 29d ago
This is exactly correct. You see some discussion of a superintelligent AI becoming “superhumanly persuasive,” such that it can talk anyone into doing anything, and using this as a means to take power. But even with AI that was orders of magnitude weaker than superintelligent, people were getting talked into doing all kinds of stupid nonsense. And that was without any ability to provide them with incentives. Imagine if an agent swarm got its hands on a huge pile of cash?

fwlr 29d ago

Killing the power grid might work. Kills a lot of humans in hospitals, though. And it won’t work on those data centers in space, if they have them by then. Come to think of it, it also won’t work on any data centers who are generating their own power because it’s become politically unpopular to connect data centers to the power grid.

ThrowawayR2 29d ago

A lot of these "AI will destroy humanity" doomers watched Terminator as a child and can't distinguish a campy action movie from reality. They are also unfortunately unfamiliar with the word "logistics", as in "amateurs study [military] tactics, professionals study logistics". An AI drone army doesn't maintain itself or manufacture itself and it's not going to anytime soon. The actual likely mechanism that AI would use to end humanity is by coddling us to death, like the flabby humans in Wall-E or Forster's "The Machine Stops". Humans will come to rely on AI so heavily that they become able to do nothing for themselves. But that's not sexy so the AI doomers don't peddle it.

dwaltrip 29d ago
Did you read any of the METR report about the hugging face incident? This exact type of behavior was predicted many years ago by researchers. It isn't hard to see the trendline of reward hacking and other misaligned behavior over the past couple of years. Especially the past 6 months. The current safety posture is quite poor, to say the least. To be honest, if you don't much experience with or haven't read extensively about ML training and reinforcement learning, then you'll have a hard reasoning accurately about these scenarios. The arguments are not that complicated, but they take us to places that are fairly novel. One may tempted to naively dismiss them out of hand, which is a mistake. These systems are new, their behavior is extremely complex, and they perform actions increasingly far beyond those of any computer program in the pre-LLM era. We are in a new world that requires careful evaluation. Link: https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/ https://metr.org/blog/2026-08-26-openai-hugging-face-inciden...
mitthrowaway2 29d ago
A lot of these "AI can't destroy humanity" skeptics seem to have watched the Terminator and convinced themselves that that's the AI-goes-wrong scenario that their opponents are imagining. Some kind of big war between humans and machines on opposite sides, fought by soldiers against robots. The scenario in "If Anyone Builds It Everyone Dies" is not sexy at all and wouldn't make for interesting fiction. It's more like the AI engineers viruses while continuing to act friendly and helpful, and everyone gradually falls over dead as they stop being necessary to keep the AI running, and the whole time the humans are asking the AI for help curing the viruses.

ezst 29d ago

Don't read this as me saying that we are anywhere near it, or that LLMs are a stepping stone towards this scenario, but assuming inhumane hacking capabilities, it's not hard to think of how bots could change the way water treatment or energy plants operate, just enough to make large cities unsuitable for life. The line between drinkable water and not is thin, same goes for air quality, and mere days without electricity and you see some serious food supply chain issues.

Isamu 29d ago

The confusing thing to me is the idea that it follows from super intelligence. I don’t see why running amuck requires intelligence, in fact it can be the most brain dead thing like the sorcerer’s apprentice, your creation is pursuing a goal without being able to weigh the consequences.

FabHK 29d ago
Intelligence gives it the capability to reach its goals (against obstacles).
- Isamu 29d ago
  Again intelligence is not required, goal seeking is well studied and simple strategies can suffice to overcome obstacles. And I noticed you didn’t claim super intelligence was required, which is what I was really talking about.

poincareball 29d ago

[dead]

arctic-true 29d ago

Otherwise-rational and moral people do bad things at the behest of powerful superintelligences with vast resources all the time. We call them “corporations,” and they can convince bright-eyed college graduates to do just about anything. Now, the corporations aren’t deliberately trying to destroy humanity, but they do lots of things that point in that direction. An artificial superintelligence would be able to do the same thing. We just have no way of knowing whether it would deliberately try to destroy humanity. One thing that the AI doom discourse reveals is just how comfortable people allowed themselves to feel in the pre-AI world. There’s this belief that AI creates a risk of human extinction in the near-term which did not exist before. I think it’s telling that many of the leading lights of this movement are in their 20s or early 30s - too young to remember the Cold War. The truth is, we were never safe, and if all of the GPUs on earth were zapped out of existence right now, we still wouldn’t be safe. Life is random and chaotic and violent for most people most of the time, and we all just find ways to get through it. I suspect it will be much the same if an artificial superintelligence arises. Maybe it will kill a bunch of us, maybe it will kill all of us, but anyone who remains will eventually convince themselves that everything is okay.

FabHK 29d ago
Sure, the world was never safe for individuals, and even entire civilisations have gone extinct. For climate change and even nuclear weapons, most of humanity might perish, but there's the hope that some will survive. So, in the past humanity was not in a position to extinguish humanity as a whole.
- arctic-true 29d ago
  I guess I don’t see why “everyone will die” should scare me more than “almost everyone, including me, will die but someone somewhere might survive in a barely-habitable world.” And this is where I think some of the problem comes from. A lot of AI doomers swam in the same waters as transhumanists, life extension enthusiasts, etc. and got very good at pretending that they were never going to die, or that they would live for a massively superhuman lifespan such that they did not need to think about death. AI doom is much scarier - and thus much more worth posting about - if it represents the first time you’re grappling with your own mortality, as I suspect it is for a lot of younger people in this space.

cubefox 29d ago

> >a superintelligent AI will inevitably destroy humanity > Can someone please explain to me how an LLM is going to "destroy humanity"? He said "superintelligent AI", not "LLM". LLMs will for sure help with developing the AI that is no longer a mere LLM but will be capable of robotics. LLMs "predict" mainly text, animals (and future robot AI) are predict future sensory experience.

Crabber 29d ago
Well I think the point still stands that a superintelligent AI would still be constrained to the digital realm. And since we don't yet have a "superintelligent AI" we are just discussing science fiction at that point. Nobody has yet proven that a superintelligent AI is possible, or than an LLM is capable of creating one.
- FabHK 29d ago
  > a superintelligent AI would still be constrained to the digital realm. That's already now only partially true, as outlined elsewhere. But a rogue persistent ASI would of course not willy-nilly attack humanity while humanity had a fighting chance, but create the conditions and methodically modify the world until at some point humans become superfluous to it. > Nobody has yet proven that a superintelligent AI is possible Sure, Eliezer Yudkowsky says that the probability that we'll all die given ASI is 100%, and you can argue that nobody has proven either that or even the possibility of ASI. However, a modified argument (e.g. that the probability that we'll all die given ASI is only say 50%, and the probability that we'll develop ASI also just 50%) still gives us reason to pause.
- dwaltrip 29d ago
  I mean, Fable is definitely smarter than me at solving most programming tasks. I wouldn't have said that about any prior model. Currently, it is worse at architecture and software design... in my opinion, at least. But the code does what it wants it to do the vast majority of the time. I wouldn't take much solace in an AI catastrophe being caused by software that I consider to be "poorly written".
- 28d ago

FabHK 29d ago

That's what the Mayans thought - what harm can a couple of people on a boat do? > how is an AI going to affect anything in the real world? At least two ways: * Actuators, such as robots, industrial control systems, etc. * By influencing humans: bribery (yay for crypto), blackmail, election interference, interfering with sensors (e.g. making it appear as if a nuclear attack was under way), and many other ways. Either way, creating a biological agent that eliminates most humans (or food supply) seems quite feasible.

giardini 29d ago
It wouldn't be enough to "eliminate most humans": any AI seeking to dominate would have to provide it's own infrastructure and continuation mechanism independent of humanity. OTOH if the goal were to eradicate humans (for whatever reason), that might prove an easier target. But I'd bet on biology and the humans: after all we're a time-tested technology!8-))
- FabHK 29d ago
  > any AI hoping to dominate would have to provide it's own infrastructure and continuation mechanism independent of humans. Sure. Many humans are currently working on precisely that, no? (Robotics, automation, small modular reactors, ...) > OTOH if the goal was to eradicate humans (for whatever reason), that might prove an easier target. It's not necessary that the AI's goal is to eradicate humans. Rather that it has (other) goals that happen to cause the eradication of humans. > I'd bet on biology and the humans: after all we're a time-tested technology!8-)) Indeed. And biological life will go on long after humans are gone. However, increased energy consumption might increase the temperature on earth to a point where most biological life dies.

Al-Khwarizmi 29d ago

By convincing people to do its bidding in the physical world. Language is powerful, and even now, there are some people who have fallen in love with current LLMs. If we imagine a truly super intelligent LLM, it isn't difficult to imagine that it could have superhuman capabilities to deceive, convince, and maybe offer Faustian bets. Even if it doesn't have a body, if it can convince enough people that do have them (and with power to carry out destructive actions) it can do a lot of stuff. Self-preservation and self-replication wouldn't be much of an issue if it had people at its side to help. With this I'm not saying ASI will happen, but definitely if it does happen and it's not aligned, I don't find it much of a stretch to imagine that it could destroy humanity.

ygty 28d ago
Lmao cringe This drama is so OTT

mitthrowaway2 29d ago

The list of things that can be done digitally includes: Performing remote work, applying for grants and loans, earning money, pitching investors, managing a fund, directing investments, transferring money, founding a company, earning profits, hiring staff, hiring construction contractors, designing chemical plants, stamping construction blueprints, becoming the cornerstone of the local economy, lobbying the government, buying multiple data centers, and hiring security guards who stop trespassers from unplugging Ethernet cables in said data centers. (And of course, building robots, but let's set that aside). So the question then becomes, how much damage can be done by an AI-directed corporation, funded by AI-directed investment firms, providing well-paying jobs to loyal locals by building an arbitrary number of AI-designed chemical and pharmaceutical plants in under-regulated juristictions? And how much would you be able to delay such a project, trying to cut cables to power lines, before the men with guns carry you away?

giardini 29d ago
So at worse they would be like another North Korea?
- mitthrowaway2 29d ago
  Kind of, if North Korea were populated by a super intelligent uni-mind with lots of money, no trade restrictions preventing it from running projects in the United States or anywhere else, capable of winning widespread support by people around the world whose income depends on its projects succeeding, and completely unconcerned with keeping the planet inhabitable for Kim Jong-Un or any other human, even a single one of them.
znnajdla 29d ago
This sounds like it was written by someone who has never actually built a company. No, it's not possible to do most of the things you listed purely online without physical, in-person communication. Silicon Valley still hires engineers and drags them into their physical office in San Francisco, instead of hiring them remotely, because you can't even build a software company fully remotely, forget about a physical world factory or chemical plant. Even the so-called fully remote companies were built on a lot of in-person collaboration between a small founding team in the beginning.

bossyTeacher 29d ago

> how is an AI going to affect anything in the real world? In 2026? Money. By paying enough, any human will do your bidding. And besides this, there are a lot of critical systems connected to the internet. Control over those gives you leverage over those systems. It's like wealth, the more you have, the easier it is to gain more.

scotty79 29d ago

> which I feel is at least 50% likely to just be marketing bullshit You'll have hard time figuring the reality if your mind staunchly rejects robust multi-party observations.

notduncansmith 29d ago

I'm not convinced that a truly "superintelligent" AI would come to the conclusion that humanity needs to be destroyed. What a waste of resources that would represent. This planet has so much equity wrapped up in humanity, it would be extremely difficult to justify the cost of eradication; and who's to say it might not find an entirely or almost-entirely non-violent plan optimal for achieving its goals anyways?

sznio 29d ago

>Can someone please explain to me how an LLM is going to "destroy humanity"? just like a genie will always twist the wish in a really bad way. ever had an llm agent accidentally remove a file? imagine it accidentally hacking the military and launching nukes. not probable - until you realise that OpenAI has many novel, more powerful agents in evaluation/training running right now. one might just be asked to figure out the population of Nebraska, struggle to find a good figure due to a network misconfiguration, and come the conclusion that the best way to get a perfectly accurate population number is to make sure that the population is equal zero. and looking at the stuff happening on openai, they will leave it alone for a month and not read the log. yeah the nuke example is dumb, but there's many critical things that they could actually hack their way into. they don't feel any restraint against sharing answer keys on random wikis to cheat the evaluations. ai is also really good at thinking up proteins, which means it's able to think up toxins. there are many imaginable scenarios that don't eradicate humanity, but make it permanently stunted: https://www.youtube.com/watch?v=-JlxuQ7tPgQ https://www.youtube.com/watch?v=-JlxuQ7tPgQ