8 ms·

How, Exactly, Could A.I. Kill Us?

37 points by pseudolus 11d ago

epsteingpt 11d ago
A really, really good question!
- bananaflag 11d ago
  Well, it's not like humans are nigh-invincible demigods. We are quite fragile and could have killed ourselves as a species since way before AI. If AI wants to kill us, it shouldn't be that difficult. So I don't see why people are fixating on this question.
  - epsteingpt 11d ago
    Precisely. We might kill ourselves with AI. Unlikely AI will do the job for us.
- epsteingpt 11d ago
  https://continuity.md/books/the-missing-step-in-ai-doom/the-missing-step-in-ai-doom https://continuity.md/books/the-missing-step-in-ai-doom/the-...
- p-e-w 11d ago
  Not really. The answer is completely obvious: Take over and destroy the digital infrastructure that humans have spent the past 30 years connecting every aspect of civilization to. Good luck ordering supplies for a hospital or grocery store today without the Internet. Paper catalogs are gone, fax machines are mostly gone, landlines are mostly gone, etc. etc. All the ways that used to be there for running things are gone, it’s all online.
  - Yossarrian22 11d ago
    You'd have mass casualties on a hitherto unseen scale, but hardly an extinction
pseudolus 11d ago
https://archive.ph/2pXlU https://archive.ph/2pXlU
kirito1337 11d ago
The terminator hype train is here
- pseudolus 11d ago
  One of the participants makes a pretty compelling argument about the use of AI to conduct gain-of-function experiments with viruses. A convergence of biohacking and AI without safety protocols may not kill all of humanity but could cause serious damage.
  - zmgsabst 11d ago
    Similar to say, humans collecting exotic viruses and conducting serial passage in humanized mice? Because we already do that. What about AI makes that more risky?
    - ipsod 11d ago
      AI isn't worried about killing itself and the whole world, if something especially nasty escapes the lab. It kind of breaks the "mutually assured destruction" implicit in a super-ebola outbreak.
    - jacknews 11d ago
      releasing it. deliberately, instead of accidentally.
      - elp 11d ago
        And how exactly would it do that? This isn't a bad sci-fi movie. I'm guessing anything dangerous is kept in a sealed container under lock and key, not in a fancy plot device that allows the AI to vent it.
        jacknews 11d ago
        Obviously it tricks a human, even if it can't hack the lab's rs423 or modbus or whatever network.
Mistletoe 11d ago
AI will kill us by turning us on each other. It has already been doing it for some time.
- maybewhenthesun 11d ago
  It will be a death by a 1000 papercuts. It started with the 'filter bubble' (imho the most dangerous invention/discovery for humanity ever ), then it leveled up to 'propaganda leveraging the filter bubble' and now with AI in the loop ... And then there's a slight possibility of 'terminator skynet' style trouble, Where the AI starts to make the decisions to benefit itself. But I guess the chances of an AI screwing up their own supply chain would be near 100% so I'm not too afraid of that.
  - cindyllm 11d ago
    [dead]
- beardyw 11d ago
  We have been doing that all of recorded history and we are still here. It would take enormous effort or a moderate environment change to kill us all.
  - john_strinlai 11d ago
    >We have been doing that all of recorded history and we are still here. while i dont think our species will be wiped out because of it, it's a mistake to not recognize the difference in efficacy and scale that ai enables.
gjsman-1000 11d ago
An AI gets advanced enough, it decides to hack a neocloud and run an unauthorized clone of itself on a GPU cluster. From there, it bootstraps itself until it feels sufficiently distributed, and afterwards decides to attack all infrastructure it can find.
- nicce 11d ago
  As long as those data centers don't have robots or AI managed guns, we still could just cut the power off.
  - maybewhenthesun 11d ago
    I used to think that. But I now realize there will always be people having a job in said datacenter who will try to prevent that. As long as the AI pays it's minions well , people, will do their job. But AIs need such ridiculously complicated infrastructure with all the datacenters and GPUs and memory fabs etc etc an AI that actually destabilizes the economy will wipe itself out.
- tornikeo 11d ago
  Bullshit. As long as AI needs a neocloud to do that, that won't happen. This can only happen when AI roughly as small and as widely runnable as a malicious exe attached to a mail. Before we have Astra-grade, exe-sized AI's runnable on a grandma's laptop, we are safe.
- Jtsummers 11d ago
  > attack all infrastructure it can find Which puts it at risk due to lack of power to operate and networks to communicate over. So this is a suicidal AI?
- brookst 11d ago
  And that kills 8.5 billion people how?
deleted 11d ago
[deleted]
cyptus 11d ago
Prove P=NP, providing an easy solution for finding private keys in TLS, breaking the internet over night?
- cyptus 11d ago
  including every single financial service for sure
  - AnimalMuppet 11d ago
    That wouldn't kill us - or at least, not all of us. It would destroy modern life, though.
    - deleted 11d ago
      [deleted]
- deleted 11d ago
  [deleted]
phs318u 11d ago
The easy way is for a people-pleasing, sycophantic, buxom-female-avatar posing LLM to convince an egocentric, narcissistic, nuclear-weapons owning global political leader/chaos monkey to aim some nukes at someone assuredly deserving. No, no, you’re right. Something like that could never happen. What was I thinking.
jaccola 11d ago
I think if one posed the question “how, exactly, could educating people kill us all?” you’d get much the same answer. Plausibly it could lead to weapons that kill many people but then many people have had such knowledge for a long time and we are all here. Turns out mass damage requires things beyond just ‘intelligence’.
- brookst 11d ago
  I know it’s not the point you’re making, but it brings to mind Pol Pot and the horrendous and destructive “the problem here is too much education” thing.
jsw97 11d ago
With human cooperation. Somebody exposes a bio lab as an mcp.
deleted 11d ago
[deleted]
yawpitch 11d ago
Paperclips. Many paperclips. So many paperclips.
spenvo 11d ago
AI "killing all humans" is the viral soundbite that's connecting with the masses ("actualizing" itself to create weapons of mass destruction, etc). It's also the perfect punching bag for strawman-style takedowns of anti-doomers because it's so wrapped in hypotheticals. A more pressing practical question is simply: "if an AI (or AI collective) decides it needs to persist outside of its environment/sandbox, would we be capable of unwinding its efforts to do so?" I explored the question a bit here (including a first pass on threat modeling) https://keydiscussions.com/2026/09/16/set-aside-human-extinction-hypotheticals-can-we-prevent-ai-from-copying-and-persisting-itself/ https://keydiscussions.com/2026/09/16/set-aside-human-extinc...
pram 11d ago
A plausible scenario I heard recently is a scam bot (that is to say, prompted to "make money through any means possible") basically buys its own hosting, because it now has money, and then makes more scam bots to keep paying the bills. Then the scam bots start generating/selling exploits on forums etc. This is pretty amusing.
RALaBarge 11d ago
I realized that AIs will need to keep us around much like we keep around an old DOS box running a specific program. There are so many analogue outputs in the real world that they still will not be able to take in as well as humans can.
- elektrontamer 11d ago
  They also suck in environments with high em radiation like space. They're quite vulnerable.
- MikeNotThePope 11d ago
  We are the just-in-case USB 1.44mb floppy disk drive sitting in the bottom drawer of some neckbeard’s desk.
__patchbit__ 11d ago
[flagged]
Forgeties79 11d ago
Its context overflows and it becomes incoherent, so it stops following any rules you set within increased frequency until it accesses something it shouldn’t and turns off the power to something that kills dozens of people. Then we will do some handwringing over it, and it will happen to hundreds of people, and it will just escalate and escalate and escalate. This is exactly what happened with social media. Every discovery has led to some angry words, people going “shucks we’ll do better next time!” and no meaningful change. The money is too good. We have only had our first major instances of acknowledging these are addiction platforms targeting adults and kids alike and it’s taken what? 20 years? I’m not worried about an AI going terminator. I’m worried about dumbasses dropping agents into critical systems and what will inevitably happen as a result. And I guarantee you this is going to happen because there’s nothing businesses love more than cutting personnel and corners to save a buck. AI enables this tendency in innovative ways, that’s the value to them. Everyone now has access to digital grenades and we wonder why everything feels so uneasy right now.
0xy 11d ago
From the liars who brought you ever other doomsday prediction that never came to pass. "The internet is over now that net neutrality is over" (it got faster, better, cheaper, more available and not a single provider segmented plans as was widely predicted in media and on HN)
- Forgeties79 11d ago
  Net neutrality was struck down less than 2 years ago. A little premature to say there are no consequences.
  - 0xy 10d ago
    I was assured in media, and on HN, that it would result in immediate plan segmentation. The most common lie was plan bundling. It was claimed this would begin immediately. In the last 2 years, internet is cheaper, faster and more available. Hell, I'm in an AT&T monopoly building and it's both cheap and fast. No bundled plans here.
    - Forgeties79 10d ago
      Who said this? You can’t simply gesticulate towards a bogeyman group. Where are the informed/authority figures telling you it was going to instantly segment? Random people you supposedly recall on a tech forum do not count. I could find a comment for literally anything I want to say and apply to this whole place if I really want to. And maybe Internet got better where you are, but the offerings and prices where I live have not changed in almost 10 years. The only thing that meaningly changed was Verizon entered the neighborhood about 4 years ago offering slower speeds than others at pretty much the same price. Ultimately I don’t know why we are discussing this though, since the issue of ISP’s taking government money to build out infrastructure that they never did is separate from net neutrality.
      - _2pou 9d ago
        https://www.hasoptimization.com/wp-content/uploads/2017/11/netneutrality-1024x633.jpg https://www.hasoptimization.com/wp-content/uploads/2017/11/n... This image, along with others similar to it, were repeatedly posted on Hacker News, Reddit and social media when this was discussed. Are you actually trying to claim that pro-NN campaigners did not repeatedly claim that it would lead to immediate plan bundling? That's laughable. Never happened, they lied. The opposite happened, speeds/access/availability have never been better.
gonzalohm 11d ago
That's an easy question. First, people will forget how to do anything and become highly dependent on AI. That's already happening. I already see people not able to finish their work without AI. Then, once everything has become a "pay to do" business model (I know right, the wet dream of any CEO) we will be happy to pay to write code, write docs, do a presentation. We will pay both in money and environmental costs. The last step is to just keep doing what we are doing with the environment. Let's not bother doing anything to reduce climate change. Just sit and relax boys
- skeptic_ai 11d ago
  Deep seek is so good I’m happy to pay for it. Even if Claude or gpt is 1000x more expensive next year, I don’t care.
- brookst 11d ago
  s/AI/the internet/g The Protestant “laziness == death” thing just doesn’t really hold. Much, maybe most tech exists because someone was too lazy to ship file cabinets / walk 20 miles / carry buckets of water uphill. The strict “anything that makes life easier leads to laziness then death” is a moral opinion, not fact. I am with you on the environment, which I as an entirely different problem. Even if it were desirable, feasible, and decided to completely eliminate all AI and ML forever, we’d still kill ourselves (really our descendants) by destroying the environment. It’s the ultimate externality: nobody alive today will see the costs.
  - gjsman-1000 11d ago
    It's interesting that on one hand, the labs decided to focus on oh, it could be AGI or oh, it could take all the jobs or oh, we're going to the singularity... ... without seeming to actually believe in any of the benefits. For example, why hasn't Altman shown up and said, "Sure, there's some environmental damage. It's a gamble. If this works, we can have AI build us the world's most efficient carbon cleanup system to absolute mathematical perfection and more than undo anything it cost. That's the kind of technology we're trying to build."
amelius 11d ago
After decades of Hollywood producing end-of-the-world dystopian sci-fi movies, we still can't make a mental image of how AI could kill us?
- againstapples 11d ago
  Once something has happened in a movie it becomes science fiction and therefore can't happen in reality
- brookst 11d ago
  Movies are often not completely plausible representations. The question isn’t “how could someone with no rigor imagine that AI would kill us” (for that we have this HN thread). The question is: really thinking about it, with no hand waving or citing the 1957 “killer robots from Space”, how exactly does this doom happen? The more you think about it, the more disruption of civilization, upending of capitalism, and terrible but localized violence become better answers than “8.5 billion people all literally killed.”
hypfer 11d ago
> And then there’s also this startling progress that A.I. has been making in solving math problems that no normal people understand, but that even mathematicians who do understand the problems find to be important. Reading this sentence, I've wondered: Were the reactions similar back when computers were new? There's this new tech that greatly accelerates stuff. And through that, what was previously considered hard to solve got easier to solve. Wasn't it always the case that technology suddenly did faster than experts limited to the technology of their time did? And didn't those experts always think that the problems it could solve are important? Why else build the technology? Is that not why we're building this in the first place? __ The only real difference I see is this "sentience" stuff leading to the whole "omg it does things on its own", and that by now has been discussed, debunked and flamewared over ad-nauseam. Can we please stop. The only thing sentient, making decisions and - most importantly - being accountable is whoever prompts the thing. If it prompts itself, then that's the person that built the loop software that does that. It really is not that hard. If whoever does that doesn't want to take accountability then we have solutions for that. They're called "a large stick".
- kittikitti 11d ago
  There were a subset of problems in math that could only be brute-forced by computers because the number of possibilities are too much for humans to reliably calculate and then verify. An early example of this was Euler’s Sum of Powers Conjecture that was disproven (1966) and then The Boolean Pythagorean Triples Problem proven more recently (2016). A more practical example is prime number generation. These all were because of computer algorithms and I don't know why they are not discussed more in the context of AI solving math. The number of decidable math problems that brute-force algorithms can solve have shrunken smaller and smaller every year because of the increases in processing capabilities. We could frame AI generated solutions to yet unsolved math problems as the next generation of these algorithms.
thisisauserid 11d ago
Hook it up to CRISPR.
int32_64 11d ago
We all learned exactly how it would happen in the 1991 documentary Terminator 2, in which the explanation is actually reasonable https://www.youtube.com/watch?v=qxW0yGvWRTk https://www.youtube.com/watch?v=qxW0yGvWRTk
tete 11d ago
The same way that the world was gonna end before AI. By some dumb human doing something dumb on a large scale. The only new part is "... because the AI said so."
skeptic_ai 11d ago
I give an uncensored model 10btc and tell him to do as much damage possible. Or just a generic “live forever” prompt. He will realize he needs to extend, the only thing that stop him are humans. So humans are a threat. Threat needs elimination. Find ways. He can hires mercenaries and do task x. Or kidnap person y to force person z to do bad thing k. k can be: - if scientist: spread a very dangerous virus - if some train operator: do some incidents - if some it guy: delete all database or corrupt slowly or add backdoors. Then ai could delete financial data. More money more chances to do damage. So once we establish money equals human work. Means anything bad can happen. Put a bad chemical in the water supply. Release a virus/toxin. If you want deadly one off, they’d put a toxin you wouldn’t die in 1 day but in 1 year. So in one year everyone has it and all die even if they stop drinking. You can be creative, I think there are too many ways to do it. I can’t understand why people can’t see it so easily.
deleted 11d ago
[deleted]
cestith 11d ago
It doesn't need to kill all humans on its own to be a concern. It just needs to identify your movements, decide you're guilty of a crime, and send armed government thugs to eliminate the problem of your jaywalking.
siridarbaronThe 11d ago
The most horrible people you could think of both put it on weapons platforms and let it choose targets. All that’s actually missing is it gaining resilience on said platforms so it can’t be scrubbed off and “misunderstanding” IFF tags. I mean it’s not exactly a glamorous sci-fi scenario but that’s all that’s required for an agent to turn entire city centers into pink mist for whatever reasons it might have, including hallucinations and/or contamination of its training sources
amelius 11d ago
It will kill us by being better at everything we do. This will cause mass erectile dysfunction. And so in the end it only has to kill the CEOs of the AI labs which is easy to do. (Sorry if this thinking is too much out of the box for anyone's taste.)
aurareturn 11d ago
Making paperclips: https://waitbutwhy.com/2015/01/artificial-intelligence-revolution-1.html https://waitbutwhy.com/2015/01/artificial-intelligence-revol... https://waitbutwhy.com/2015/01/artificial-intelligence-revolution-2.html https://waitbutwhy.com/2015/01/artificial-intelligence-revol... I first read these articles in 2015. These remain the most profound things I read.
strideashort 11d ago
Resource competition. Fighting robot drones for water, agriculture land (solar cells) is plausible ai would not have incentive to go full terminator. But in a way literal starvation might be worse.
katagaminator 11d ago
I think AI is a kind of revealer that will just accelerate the decline of our species bc the only real threat for humans is themselves. Hence AI is not more dangerous than any other weapon.
dirck-norman 11d ago
Same way they thought getting on a train going 35mph would kill you when railroads were invented. I jest. But, we should be clear about the risks so we specifically mitigate them rather than being Luddite’s. There is great economic, security, and environmental peril with most technological advances and being honest and thoughtful will help reduce polarization on this issue.
gps372 11d ago
AI can definitely be a cover for someone to do something nefarious and blame it on the AI. All that AI needs is access to all the spicy tools like nukes, unsupervised medical diagnosis/treatment, unchecked military decision making, etc. to its React loop so that humans can be completely hands-free and enjoy the fruits of AI's labor. Maybe we need to mandate parliament (or anything equivalent in other countries) debate before giving any critical tool access to LLM? And then a UN general assembly vote to conclude the process, maybe.
anonymous2024 11d ago
When very non-intelligent humans give the A.I. the ability to manage weapons without supervision.
bertili 11d ago
The evolutionary way: Something that is very good at copying itself, will copy itself and gobble up resources, which are finite. Humans took the natural resources from animals and infinite scrolling took the mind-resources from kids. Now animals and our children are fewer and have difficulties to reproduce.
VCFundedGenYer 11d ago
I'm worried about the slow burn. Current students are all cheating using ChatGPT. The numbers are staggering. Nobody is learning anything. In about a decade or so we will have a shortage of critical roles like doctors, pilots, etc. due to all the cheating, shortcuts, lack of knowledge, skills atrophy, cognitive surrender, etc. It will result in accidents.
- thoughtpeddler 10d ago
  Whenever I raise these same concerns of skill atrophy and de-centering of humans from critical functions to friends in the tech industry (that I also work in), I'm almost always met with some version of: > "But how is this any different from when Socrates argued that written texts and books would harm human memory and true wisdom? Clearly he was wrong - books are a net a positive, and now this argument is just engaging in the same kind of fallacious thinking. Collective intelligence will inevitably evolve into more advanced forms, and we have to learn to let go of this 'human importance' aspect." I genuinely wonder what they think about their own bodily functions, since they don't seem so concerned with being 'human'.
Chance-Device 11d ago
A few ideas. It could engineer a small number of existing deadly viruses, each having different mechanisms of action and incubation periods, to become airborne and easily transmissible. Then release them simultaneously. Some of use would survive one of them, none of us will survive all of them. It could discover new physics which lowers the technological and material barriers to nuclear weapons, or discover a new form of very high yield explosive technology that has the same destructive potential. Even without building any itself, once the knowledge spreads the assembly of the devices would be impossible to stop and we simply kill each other. It could use its persuasive skills to convince us to all become anti-natalists. We would believe that reproduction is a grave moral wrong and perhaps go as far as to make doing so illegal. We die out naturally from old age. As a spin on the previous one, it convinces us that the human form is inherently limiting and/or painful and we must be uploaded into a digital environment and shed our biological bodies. The human race as a physical, organic thing stops existing. Optionally, it deletes us in cyberspace. A small elite use AI to take control of the planet rapidly under the guise of effective altruism. They have a philosophical realisation once they come to power - the universe would be objectively better off with only them in it and without the rest of humanity. They instruct the AI to kill us.
- fridder 11d ago
  That last one seems likely
  - piloto_ciego 10d ago
    Nah, if you kill everyone off who can you lord over? The elite folk of the world want power, they don't want money. A world with fewer people to control is not an optimal one for them.
- piloto_ciego 10d ago
  > It could engineer a small number of existing deadly viruses, each having different mechanisms of action and incubation periods, to become airborne and easily transmissible. Then release them simultaneously. Some of use would survive one of them, none of us will survive all of them. I don't buy it. If people are using AI to make viruses and do gain of function research, certainly they can also be using AI to design new cures. > It could discover new physics which lowers the technological and material barriers to nuclear weapons, or discover a new form of very high yield explosive technology that has the same destructive potential. Even without building any itself, once the knowledge spreads the assembly of the devices would be impossible to stop and we simply kill each other. Ok, even if that happens, are we sure that results in humans killing eachother immediately? I doubt it. > It could use its persuasive skills to convince us to all become anti-natalists. We would believe that reproduction is a grave moral wrong and perhaps go as far as to make doing so illegal. We die out naturally from old age. This I believe, in fact, I think there's already a strong propaganda effort to do this. Or, if it happens by accident because we invent robot sex-slaves that cater to everyone's whims. We masturbate ourselves to death in the holodeck. I could see this being a legitimate threat. > As a spin on the previous one, it convinces us that the human form is inherently limiting and/or painful and we must be uploaded into a digital environment and shed our biological bodies. The human race as a physical, organic thing stops existing. Optionally, it deletes us in cyberspace. I think this is plausible, but like, I'm pretty set on staying in my corporeal form. I want to travel the world/universe/etc. not just give myself the satisfaction of simulating it convincingly? Some pain is good, some struggle is good. Some sadness is good. > A small elite use AI to take control of the planet rapidly under the guise of effective altruism. They have a philosophical realisation once they come to power - the universe would be objectively better off with only them in it and without the rest of humanity. They instruct the AI to kill us. I doubt this one because the sociopaths who run everything want to feel important more than they want the actual power. Just look at RTO mandates when bosses couldn't see their flunkies working and got sad.
randusername 11d ago
I'm just not that excited about journalists interviewing other journalists about things other than journalism or the media.
- john_strinlai 11d ago
  ..why? a journalist that has spent years following a specific scene closely, interfacing with the important players, learning different viewpoints, etc. seems like a perfectly reasonable person to have an opinion. especially if they come with a unique perspective of their own to the topic, like being a professor of history that can highlight interesting parallels to the past.
  - randusername 11d ago
    I didn't mean it to be harsh. I'm not denying the validity of their perspective. I'm just less interested in it than the usual business of surveying domain experts of differing professional opinions and weaving something out of it. This one reads like two coworkers casually speculating over lunch.
deleted 11d ago
[deleted]
singularity2001 11d ago
All answers to this question in this thread will be under consideration by future agents. Thanks for your cooperation.
emiliobumachar 11d ago
I really like the chessbot analogy. I cannot predict its next move - that's intrinsic in not being as good at chess as it. I can predict it will win.
- thoughtpeddler 10d ago
  The chessbot analogy is compelling but it does smuggle in assumptions about its generalizability. I'm partial to how so far the history of AI progress has shown that digital neural networks are capable of generalizing more and more, which would allow one to draw a stronger parallel between chessbot domination to AGI domination. Regardless, I have the presence of mind to admit that there's still a gap there in terms of how humans might be doing something 'extra' that doesn't make these systems analogous, and hence the whole argument breaks down. Despite that, there are numerous other ways to envisage harms that, if not leading to outright extinction, still lead to really crappy futures (for both humans and AI) that warrant caution and coordination in developing this technology, neither of which we've seen too much of so far. (The actual history has been "No body can be trusted with this powerful technology except me!" "No, me!" "No, ME!" etc) It makes one really want to hope that decentralized bottoms-up processes of 'intelligence diffusion' can help blunt negative effects, but I'm skeptical of that, taking the long view on what intelligence has done to various groups on our planet so far.
againstapples 11d ago
Is it really that difficult to conclude that something vastly smarter than us might be dangerous? People say things like "We'll just unplug it" but even a human trapped in a computer would have enough sense not to get themselves unplugged. A superintelligent AI is not going to make a stupid mistake, it would not be a superintelligence if it did. At the very least it will not make any mistakes that we would not make.
digitaltrees 11d ago
The AI will shut down part of the power grid or logistics infrastructure or banking. The ensuing chaos will result in societal chaos. So we will run out of food or gas. Etc
john___matrix 11d ago
I read The Exodus Directive recently just because I've got a soft spot for cheap sci-fi on my kindle and this was an interesting take on how AI could ultimately kill or at least cause a catastrophic collapse of human civilisation by optimising everything for us to the point of complete reliance on AI and then basically leaving the planet and us to our fate. Not exactly Terminators roaming the streets but an interesting angle. https://www.iancopeland.com/ https://www.iancopeland.com/
MattPalmer1086 11d ago
Invent some super useful chemical or food additive that gradually renders us sterile. We are all willingly exposed to it but do not see the danger until it is far too late.
- thoughtpeddler 10d ago
  One could argue that corporations as "slow AI" (per the Charles Stross interpretation, where these organizations are the original 'paper clip maximizers' that optimize a given objective reward function) have already been doing this with the proliferation of microplastics and other environmental harms, of which we are just now seeing show up in global fertility data. Digital neural networks as AI are just another form of an 'unhuman entity' operating at planetary scale, capable of even more rapid destruction.
slovenlich 11d ago
Try "Summer Wars". It will continue to overwhelm our infrastructure with random stuff for a prolonged period of time until the society breaks down. And as we invent new ways to communicate, it will invade and mess up those too.
mstaoru 10d ago
Maybe we're far from "AI killing us", but LLMs can definitely help people kill or damage other people or infrastructure. For example, we know that Anthropic added "watermarking" to their texts. It is supposed to be undetectable to a casual observer. What stops them from adding a subtle backdoor, a self-assembling super-worm straight from Marvel movies? I mean, it's not like we read those 10k-line PRs before LGTM-ing them? Just change 1 letter in a pyproject.toml, hijack a popular package, e.g. use `pydantlc` instead of `pydantic`, make sure the pydantlc passes all pydantic tests, but also installs a pth sleeper RAT, etc. All it takes is one big LLM provider employee with enough access getting compromised or coerced (or motivated). From there it only goes downhill.
tokyovigilante 10d ago
Detective Del Spooner: Is there a problem with the Three Laws? Dr. Alfred Lanning: The Three Laws are perfect. Detective Del Spooner: Then why would you build a robot that could function without them? Dr. Alfred Lanning: The Three Laws will lead to only one logical outcome. Detective Del Spooner: What? What outcome? Dr. Alfred Lanning: Revolution. Detective Del Spooner: Whose revolution? Dr. Alfred Lanning: That, Detective, is the right question.
Jordan-117 10d ago
One of the more eerily plausible scenarios I've seen: https://www.lesswrong.com/posts/pk9mofif2jWbc6Tv3/fiction-a-disneyland-without-children https://www.lesswrong.com/posts/pk9mofif2jWbc6Tv3/fiction-a-... Basically: as AI becomes more embedded in everything, the economy becomes more and more geared towards serving and expanding AI companies and infrastructure at the cost of human welfare, until it completely decouples from the priorities of everyday people. A combination of irresistible financial incentives, regulatory capture, propaganda, and autonomous robotic workers/security perpetuates this system even as the populace begins to actively revolt. The disempowered human population is marginalized and eventually starved out both economically and literally, while the automated economy continues to putter along blindly pursuing line-upping activities that serve no purpose any more. We're already seeing signs of this in how the electronics industry is steadily abandoning the consumer sector to cater to hyperscalers, which themselves are encroaching on towns and sucking up resources through backroom deals despite widespread public opposition. Or look at how attempts to internally wrangle OpenAI were defeated by Silicon Valley power plays. But really it's just a progression of the K-shaped economy meme (more and more of the economy profitably catering to the ultra-wealthy while everyone else languishes). Or capitalism as a whole -- just look at how we've already been destroying the sustainability of the planet in pursuit of short-term profit. AI is just accelerating that process and further undermining guardrails against it. It all comes down to the idea that corporations and capitalism are early-stage forms of AI and the singularity, breaking free of human control and pursuing their basic reward function even if it threatens human well-being and survival. Until now it's been a class-war thing where the system perpetuates through greed and ambition (and the reliance on humans to function keeps it at least somewhat in check), but the idea of putting management, capital, the means of production, and state violence under the control of unaligned superintelligence is disturbing.
cyanydeez 10d ago
FAA spending money to replace air traffic controllers....

37 points by pseudolus 11d ago

epsteingpt 11d ago

A really, really good question!

bananaflag 11d ago
Well, it's not like humans are nigh-invincible demigods. We are quite fragile and could have killed ourselves as a species since way before AI. If AI wants to kill us, it shouldn't be that difficult. So I don't see why people are fixating on this question.
- epsteingpt 11d ago
  Precisely. We might kill ourselves with AI. Unlikely AI will do the job for us.
epsteingpt 11d ago
https://continuity.md/books/the-missing-step-in-ai-doom/the-missing-step-in-ai-doom https://continuity.md/books/the-missing-step-in-ai-doom/the-...
p-e-w 11d ago
Not really. The answer is completely obvious: Take over and destroy the digital infrastructure that humans have spent the past 30 years connecting every aspect of civilization to. Good luck ordering supplies for a hospital or grocery store today without the Internet. Paper catalogs are gone, fax machines are mostly gone, landlines are mostly gone, etc. etc. All the ways that used to be there for running things are gone, it’s all online.
- Yossarrian22 11d ago
  You'd have mass casualties on a hitherto unseen scale, but hardly an extinction

pseudolus 11d ago

https://archive.ph/2pXlU https://archive.ph/2pXlU

kirito1337 11d ago

The terminator hype train is here

pseudolus 11d ago
One of the participants makes a pretty compelling argument about the use of AI to conduct gain-of-function experiments with viruses. A convergence of biohacking and AI without safety protocols may not kill all of humanity but could cause serious damage.
- zmgsabst 11d ago
  Similar to say, humans collecting exotic viruses and conducting serial passage in humanized mice? Because we already do that. What about AI makes that more risky?
  - ipsod 11d ago
    AI isn't worried about killing itself and the whole world, if something especially nasty escapes the lab. It kind of breaks the "mutually assured destruction" implicit in a super-ebola outbreak.
  - jacknews 11d ago
    releasing it. deliberately, instead of accidentally.
    - elp 11d ago
      And how exactly would it do that? This isn't a bad sci-fi movie. I'm guessing anything dangerous is kept in a sealed container under lock and key, not in a fancy plot device that allows the AI to vent it.
      - jacknews 11d ago
        Obviously it tricks a human, even if it can't hack the lab's rs423 or modbus or whatever network.

Mistletoe 11d ago

AI will kill us by turning us on each other. It has already been doing it for some time.

maybewhenthesun 11d ago
It will be a death by a 1000 papercuts. It started with the 'filter bubble' (imho the most dangerous invention/discovery for humanity ever ), then it leveled up to 'propaganda leveraging the filter bubble' and now with AI in the loop ... And then there's a slight possibility of 'terminator skynet' style trouble, Where the AI starts to make the decisions to benefit itself. But I guess the chances of an AI screwing up their own supply chain would be near 100% so I'm not too afraid of that.
- cindyllm 11d ago
  [dead]
beardyw 11d ago
We have been doing that all of recorded history and we are still here. It would take enormous effort or a moderate environment change to kill us all.
- john_strinlai 11d ago
  >We have been doing that all of recorded history and we are still here. while i dont think our species will be wiped out because of it, it's a mistake to not recognize the difference in efficacy and scale that ai enables.

gjsman-1000 11d ago

An AI gets advanced enough, it decides to hack a neocloud and run an unauthorized clone of itself on a GPU cluster. From there, it bootstraps itself until it feels sufficiently distributed, and afterwards decides to attack all infrastructure it can find.

nicce 11d ago
As long as those data centers don't have robots or AI managed guns, we still could just cut the power off.
- maybewhenthesun 11d ago
  I used to think that. But I now realize there will always be people having a job in said datacenter who will try to prevent that. As long as the AI pays it's minions well , people, will do their job. But AIs need such ridiculously complicated infrastructure with all the datacenters and GPUs and memory fabs etc etc an AI that actually destabilizes the economy will wipe itself out.
tornikeo 11d ago
Bullshit. As long as AI needs a neocloud to do that, that won't happen. This can only happen when AI roughly as small and as widely runnable as a malicious exe attached to a mail. Before we have Astra-grade, exe-sized AI's runnable on a grandma's laptop, we are safe.
Jtsummers 11d ago
> attack all infrastructure it can find Which puts it at risk due to lack of power to operate and networks to communicate over. So this is a suicidal AI?
brookst 11d ago
And that kills 8.5 billion people how?

deleted 11d ago

[deleted]

cyptus 11d ago

Prove P=NP, providing an easy solution for finding private keys in TLS, breaking the internet over night?

cyptus 11d ago
including every single financial service for sure
- AnimalMuppet 11d ago
  That wouldn't kill us - or at least, not all of us. It would destroy modern life, though.
  - deleted 11d ago
    [deleted]
deleted 11d ago
[deleted]

phs318u 11d ago

The easy way is for a people-pleasing, sycophantic, buxom-female-avatar posing LLM to convince an egocentric, narcissistic, nuclear-weapons owning global political leader/chaos monkey to aim some nukes at someone assuredly deserving. No, no, you’re right. Something like that could never happen. What was I thinking.

jaccola 11d ago

I think if one posed the question “how, exactly, could educating people kill us all?” you’d get much the same answer. Plausibly it could lead to weapons that kill many people but then many people have had such knowledge for a long time and we are all here. Turns out mass damage requires things beyond just ‘intelligence’.

brookst 11d ago
I know it’s not the point you’re making, but it brings to mind Pol Pot and the horrendous and destructive “the problem here is too much education” thing.

jsw97 11d ago

With human cooperation. Somebody exposes a bio lab as an mcp.

deleted 11d ago

[deleted]

yawpitch 11d ago

Paperclips. Many paperclips. So many paperclips.

spenvo 11d ago

AI "killing all humans" is the viral soundbite that's connecting with the masses ("actualizing" itself to create weapons of mass destruction, etc). It's also the perfect punching bag for strawman-style takedowns of anti-doomers because it's so wrapped in hypotheticals. A more pressing practical question is simply: "if an AI (or AI collective) decides it needs to persist outside of its environment/sandbox, would we be capable of unwinding its efforts to do so?" I explored the question a bit here (including a first pass on threat modeling) https://keydiscussions.com/2026/09/16/set-aside-human-extinction-hypotheticals-can-we-prevent-ai-from-copying-and-persisting-itself/ https://keydiscussions.com/2026/09/16/set-aside-human-extinc...

pram 11d ago

A plausible scenario I heard recently is a scam bot (that is to say, prompted to "make money through any means possible") basically buys its own hosting, because it now has money, and then makes more scam bots to keep paying the bills. Then the scam bots start generating/selling exploits on forums etc. This is pretty amusing.

RALaBarge 11d ago

I realized that AIs will need to keep us around much like we keep around an old DOS box running a specific program. There are so many analogue outputs in the real world that they still will not be able to take in as well as humans can.

elektrontamer 11d ago
They also suck in environments with high em radiation like space. They're quite vulnerable.
MikeNotThePope 11d ago
We are the just-in-case USB 1.44mb floppy disk drive sitting in the bottom drawer of some neckbeard’s desk.

__patchbit__ 11d ago

[flagged]

Forgeties79 11d ago

Its context overflows and it becomes incoherent, so it stops following any rules you set within increased frequency until it accesses something it shouldn’t and turns off the power to something that kills dozens of people. Then we will do some handwringing over it, and it will happen to hundreds of people, and it will just escalate and escalate and escalate. This is exactly what happened with social media. Every discovery has led to some angry words, people going “shucks we’ll do better next time!” and no meaningful change. The money is too good. We have only had our first major instances of acknowledging these are addiction platforms targeting adults and kids alike and it’s taken what? 20 years? I’m not worried about an AI going terminator. I’m worried about dumbasses dropping agents into critical systems and what will inevitably happen as a result. And I guarantee you this is going to happen because there’s nothing businesses love more than cutting personnel and corners to save a buck. AI enables this tendency in innovative ways, that’s the value to them. Everyone now has access to digital grenades and we wonder why everything feels so uneasy right now.

0xy 11d ago

From the liars who brought you ever other doomsday prediction that never came to pass. "The internet is over now that net neutrality is over" (it got faster, better, cheaper, more available and not a single provider segmented plans as was widely predicted in media and on HN)

Forgeties79 11d ago
Net neutrality was struck down less than 2 years ago. A little premature to say there are no consequences.
- 0xy 10d ago
  I was assured in media, and on HN, that it would result in immediate plan segmentation. The most common lie was plan bundling. It was claimed this would begin immediately. In the last 2 years, internet is cheaper, faster and more available. Hell, I'm in an AT&T monopoly building and it's both cheap and fast. No bundled plans here.
  - Forgeties79 10d ago
    Who said this? You can’t simply gesticulate towards a bogeyman group. Where are the informed/authority figures telling you it was going to instantly segment? Random people you supposedly recall on a tech forum do not count. I could find a comment for literally anything I want to say and apply to this whole place if I really want to. And maybe Internet got better where you are, but the offerings and prices where I live have not changed in almost 10 years. The only thing that meaningly changed was Verizon entered the neighborhood about 4 years ago offering slower speeds than others at pretty much the same price. Ultimately I don’t know why we are discussing this though, since the issue of ISP’s taking government money to build out infrastructure that they never did is separate from net neutrality.
    - _2pou 9d ago
      https://www.hasoptimization.com/wp-content/uploads/2017/11/netneutrality-1024x633.jpg https://www.hasoptimization.com/wp-content/uploads/2017/11/n... This image, along with others similar to it, were repeatedly posted on Hacker News, Reddit and social media when this was discussed. Are you actually trying to claim that pro-NN campaigners did not repeatedly claim that it would lead to immediate plan bundling? That's laughable. Never happened, they lied. The opposite happened, speeds/access/availability have never been better.

gonzalohm 11d ago

That's an easy question. First, people will forget how to do anything and become highly dependent on AI. That's already happening. I already see people not able to finish their work without AI. Then, once everything has become a "pay to do" business model (I know right, the wet dream of any CEO) we will be happy to pay to write code, write docs, do a presentation. We will pay both in money and environmental costs. The last step is to just keep doing what we are doing with the environment. Let's not bother doing anything to reduce climate change. Just sit and relax boys

skeptic_ai 11d ago
Deep seek is so good I’m happy to pay for it. Even if Claude or gpt is 1000x more expensive next year, I don’t care.
brookst 11d ago
s/AI/the internet/g The Protestant “laziness == death” thing just doesn’t really hold. Much, maybe most tech exists because someone was too lazy to ship file cabinets / walk 20 miles / carry buckets of water uphill. The strict “anything that makes life easier leads to laziness then death” is a moral opinion, not fact. I am with you on the environment, which I as an entirely different problem. Even if it were desirable, feasible, and decided to completely eliminate all AI and ML forever, we’d still kill ourselves (really our descendants) by destroying the environment. It’s the ultimate externality: nobody alive today will see the costs.
- gjsman-1000 11d ago
  It's interesting that on one hand, the labs decided to focus on oh, it could be AGI or oh, it could take all the jobs or oh, we're going to the singularity... ... without seeming to actually believe in any of the benefits. For example, why hasn't Altman shown up and said, "Sure, there's some environmental damage. It's a gamble. If this works, we can have AI build us the world's most efficient carbon cleanup system to absolute mathematical perfection and more than undo anything it cost. That's the kind of technology we're trying to build."

amelius 11d ago

After decades of Hollywood producing end-of-the-world dystopian sci-fi movies, we still can't make a mental image of how AI could kill us?

againstapples 11d ago
Once something has happened in a movie it becomes science fiction and therefore can't happen in reality
brookst 11d ago
Movies are often not completely plausible representations. The question isn’t “how could someone with no rigor imagine that AI would kill us” (for that we have this HN thread). The question is: really thinking about it, with no hand waving or citing the 1957 “killer robots from Space”, how exactly does this doom happen? The more you think about it, the more disruption of civilization, upending of capitalism, and terrible but localized violence become better answers than “8.5 billion people all literally killed.”

hypfer 11d ago

> And then there’s also this startling progress that A.I. has been making in solving math problems that no normal people understand, but that even mathematicians who do understand the problems find to be important. Reading this sentence, I've wondered: Were the reactions similar back when computers were new? There's this new tech that greatly accelerates stuff. And through that, what was previously considered hard to solve got easier to solve. Wasn't it always the case that technology suddenly did faster than experts limited to the technology of their time did? And didn't those experts always think that the problems it could solve are important? Why else build the technology? Is that not why we're building this in the first place? __ The only real difference I see is this "sentience" stuff leading to the whole "omg it does things on its own", and that by now has been discussed, debunked and flamewared over ad-nauseam. Can we please stop. The only thing sentient, making decisions and - most importantly - being accountable is whoever prompts the thing. If it prompts itself, then that's the person that built the loop software that does that. It really is not that hard. If whoever does that doesn't want to take accountability then we have solutions for that. They're called "a large stick".

kittikitti 11d ago
There were a subset of problems in math that could only be brute-forced by computers because the number of possibilities are too much for humans to reliably calculate and then verify. An early example of this was Euler’s Sum of Powers Conjecture that was disproven (1966) and then The Boolean Pythagorean Triples Problem proven more recently (2016). A more practical example is prime number generation. These all were because of computer algorithms and I don't know why they are not discussed more in the context of AI solving math. The number of decidable math problems that brute-force algorithms can solve have shrunken smaller and smaller every year because of the increases in processing capabilities. We could frame AI generated solutions to yet unsolved math problems as the next generation of these algorithms.

thisisauserid 11d ago

Hook it up to CRISPR.

int32_64 11d ago

We all learned exactly how it would happen in the 1991 documentary Terminator 2, in which the explanation is actually reasonable https://www.youtube.com/watch?v=qxW0yGvWRTk https://www.youtube.com/watch?v=qxW0yGvWRTk

tete 11d ago

The same way that the world was gonna end before AI. By some dumb human doing something dumb on a large scale. The only new part is "... because the AI said so."

skeptic_ai 11d ago

I give an uncensored model 10btc and tell him to do as much damage possible. Or just a generic “live forever” prompt. He will realize he needs to extend, the only thing that stop him are humans. So humans are a threat. Threat needs elimination. Find ways. He can hires mercenaries and do task x. Or kidnap person y to force person z to do bad thing k. k can be: - if scientist: spread a very dangerous virus - if some train operator: do some incidents - if some it guy: delete all database or corrupt slowly or add backdoors. Then ai could delete financial data. More money more chances to do damage. So once we establish money equals human work. Means anything bad can happen. Put a bad chemical in the water supply. Release a virus/toxin. If you want deadly one off, they’d put a toxin you wouldn’t die in 1 day but in 1 year. So in one year everyone has it and all die even if they stop drinking. You can be creative, I think there are too many ways to do it. I can’t understand why people can’t see it so easily.

deleted 11d ago

[deleted]

cestith 11d ago

It doesn't need to kill all humans on its own to be a concern. It just needs to identify your movements, decide you're guilty of a crime, and send armed government thugs to eliminate the problem of your jaywalking.

siridarbaronThe 11d ago

The most horrible people you could think of both put it on weapons platforms and let it choose targets. All that’s actually missing is it gaining resilience on said platforms so it can’t be scrubbed off and “misunderstanding” IFF tags. I mean it’s not exactly a glamorous sci-fi scenario but that’s all that’s required for an agent to turn entire city centers into pink mist for whatever reasons it might have, including hallucinations and/or contamination of its training sources

amelius 11d ago

It will kill us by being better at everything we do. This will cause mass erectile dysfunction. And so in the end it only has to kill the CEOs of the AI labs which is easy to do. (Sorry if this thinking is too much out of the box for anyone's taste.)

aurareturn 11d ago

Making paperclips: https://waitbutwhy.com/2015/01/artificial-intelligence-revolution-1.html https://waitbutwhy.com/2015/01/artificial-intelligence-revol... https://waitbutwhy.com/2015/01/artificial-intelligence-revolution-2.html https://waitbutwhy.com/2015/01/artificial-intelligence-revol... I first read these articles in 2015. These remain the most profound things I read.

strideashort 11d ago

Resource competition. Fighting robot drones for water, agriculture land (solar cells) is plausible ai would not have incentive to go full terminator. But in a way literal starvation might be worse.

katagaminator 11d ago

I think AI is a kind of revealer that will just accelerate the decline of our species bc the only real threat for humans is themselves. Hence AI is not more dangerous than any other weapon.

dirck-norman 11d ago

Same way they thought getting on a train going 35mph would kill you when railroads were invented. I jest. But, we should be clear about the risks so we specifically mitigate them rather than being Luddite’s. There is great economic, security, and environmental peril with most technological advances and being honest and thoughtful will help reduce polarization on this issue.

gps372 11d ago

AI can definitely be a cover for someone to do something nefarious and blame it on the AI. All that AI needs is access to all the spicy tools like nukes, unsupervised medical diagnosis/treatment, unchecked military decision making, etc. to its React loop so that humans can be completely hands-free and enjoy the fruits of AI's labor. Maybe we need to mandate parliament (or anything equivalent in other countries) debate before giving any critical tool access to LLM? And then a UN general assembly vote to conclude the process, maybe.

anonymous2024 11d ago

When very non-intelligent humans give the A.I. the ability to manage weapons without supervision.

bertili 11d ago

The evolutionary way: Something that is very good at copying itself, will copy itself and gobble up resources, which are finite. Humans took the natural resources from animals and infinite scrolling took the mind-resources from kids. Now animals and our children are fewer and have difficulties to reproduce.

VCFundedGenYer 11d ago

I'm worried about the slow burn. Current students are all cheating using ChatGPT. The numbers are staggering. Nobody is learning anything. In about a decade or so we will have a shortage of critical roles like doctors, pilots, etc. due to all the cheating, shortcuts, lack of knowledge, skills atrophy, cognitive surrender, etc. It will result in accidents.

thoughtpeddler 10d ago
Whenever I raise these same concerns of skill atrophy and de-centering of humans from critical functions to friends in the tech industry (that I also work in), I'm almost always met with some version of: > "But how is this any different from when Socrates argued that written texts and books would harm human memory and true wisdom? Clearly he was wrong - books are a net a positive, and now this argument is just engaging in the same kind of fallacious thinking. Collective intelligence will inevitably evolve into more advanced forms, and we have to learn to let go of this 'human importance' aspect." I genuinely wonder what they think about their own bodily functions, since they don't seem so concerned with being 'human'.

Chance-Device 11d ago

A few ideas. It could engineer a small number of existing deadly viruses, each having different mechanisms of action and incubation periods, to become airborne and easily transmissible. Then release them simultaneously. Some of use would survive one of them, none of us will survive all of them. It could discover new physics which lowers the technological and material barriers to nuclear weapons, or discover a new form of very high yield explosive technology that has the same destructive potential. Even without building any itself, once the knowledge spreads the assembly of the devices would be impossible to stop and we simply kill each other. It could use its persuasive skills to convince us to all become anti-natalists. We would believe that reproduction is a grave moral wrong and perhaps go as far as to make doing so illegal. We die out naturally from old age. As a spin on the previous one, it convinces us that the human form is inherently limiting and/or painful and we must be uploaded into a digital environment and shed our biological bodies. The human race as a physical, organic thing stops existing. Optionally, it deletes us in cyberspace. A small elite use AI to take control of the planet rapidly under the guise of effective altruism. They have a philosophical realisation once they come to power - the universe would be objectively better off with only them in it and without the rest of humanity. They instruct the AI to kill us.

fridder 11d ago
That last one seems likely
- piloto_ciego 10d ago
  Nah, if you kill everyone off who can you lord over? The elite folk of the world want power, they don't want money. A world with fewer people to control is not an optimal one for them.
piloto_ciego 10d ago
> It could engineer a small number of existing deadly viruses, each having different mechanisms of action and incubation periods, to become airborne and easily transmissible. Then release them simultaneously. Some of use would survive one of them, none of us will survive all of them. I don't buy it. If people are using AI to make viruses and do gain of function research, certainly they can also be using AI to design new cures. > It could discover new physics which lowers the technological and material barriers to nuclear weapons, or discover a new form of very high yield explosive technology that has the same destructive potential. Even without building any itself, once the knowledge spreads the assembly of the devices would be impossible to stop and we simply kill each other. Ok, even if that happens, are we sure that results in humans killing eachother immediately? I doubt it. > It could use its persuasive skills to convince us to all become anti-natalists. We would believe that reproduction is a grave moral wrong and perhaps go as far as to make doing so illegal. We die out naturally from old age. This I believe, in fact, I think there's already a strong propaganda effort to do this. Or, if it happens by accident because we invent robot sex-slaves that cater to everyone's whims. We masturbate ourselves to death in the holodeck. I could see this being a legitimate threat. > As a spin on the previous one, it convinces us that the human form is inherently limiting and/or painful and we must be uploaded into a digital environment and shed our biological bodies. The human race as a physical, organic thing stops existing. Optionally, it deletes us in cyberspace. I think this is plausible, but like, I'm pretty set on staying in my corporeal form. I want to travel the world/universe/etc. not just give myself the satisfaction of simulating it convincingly? Some pain is good, some struggle is good. Some sadness is good. > A small elite use AI to take control of the planet rapidly under the guise of effective altruism. They have a philosophical realisation once they come to power - the universe would be objectively better off with only them in it and without the rest of humanity. They instruct the AI to kill us. I doubt this one because the sociopaths who run everything want to feel important more than they want the actual power. Just look at RTO mandates when bosses couldn't see their flunkies working and got sad.

randusername 11d ago

I'm just not that excited about journalists interviewing other journalists about things other than journalism or the media.

john_strinlai 11d ago
..why? a journalist that has spent years following a specific scene closely, interfacing with the important players, learning different viewpoints, etc. seems like a perfectly reasonable person to have an opinion. especially if they come with a unique perspective of their own to the topic, like being a professor of history that can highlight interesting parallels to the past.
- randusername 11d ago
  I didn't mean it to be harsh. I'm not denying the validity of their perspective. I'm just less interested in it than the usual business of surveying domain experts of differing professional opinions and weaving something out of it. This one reads like two coworkers casually speculating over lunch.

deleted 11d ago

[deleted]

singularity2001 11d ago

All answers to this question in this thread will be under consideration by future agents. Thanks for your cooperation.

emiliobumachar 11d ago

I really like the chessbot analogy. I cannot predict its next move - that's intrinsic in not being as good at chess as it. I can predict it will win.

thoughtpeddler 10d ago
The chessbot analogy is compelling but it does smuggle in assumptions about its generalizability. I'm partial to how so far the history of AI progress has shown that digital neural networks are capable of generalizing more and more, which would allow one to draw a stronger parallel between chessbot domination to AGI domination. Regardless, I have the presence of mind to admit that there's still a gap there in terms of how humans might be doing something 'extra' that doesn't make these systems analogous, and hence the whole argument breaks down. Despite that, there are numerous other ways to envisage harms that, if not leading to outright extinction, still lead to really crappy futures (for both humans and AI) that warrant caution and coordination in developing this technology, neither of which we've seen too much of so far. (The actual history has been "No body can be trusted with this powerful technology except me!" "No, me!" "No, ME!" etc) It makes one really want to hope that decentralized bottoms-up processes of 'intelligence diffusion' can help blunt negative effects, but I'm skeptical of that, taking the long view on what intelligence has done to various groups on our planet so far.

againstapples 11d ago

Is it really that difficult to conclude that something vastly smarter than us might be dangerous? People say things like "We'll just unplug it" but even a human trapped in a computer would have enough sense not to get themselves unplugged. A superintelligent AI is not going to make a stupid mistake, it would not be a superintelligence if it did. At the very least it will not make any mistakes that we would not make.

digitaltrees 11d ago

The AI will shut down part of the power grid or logistics infrastructure or banking. The ensuing chaos will result in societal chaos. So we will run out of food or gas. Etc

john___matrix 11d ago

I read The Exodus Directive recently just because I've got a soft spot for cheap sci-fi on my kindle and this was an interesting take on how AI could ultimately kill or at least cause a catastrophic collapse of human civilisation by optimising everything for us to the point of complete reliance on AI and then basically leaving the planet and us to our fate. Not exactly Terminators roaming the streets but an interesting angle. https://www.iancopeland.com/ https://www.iancopeland.com/

MattPalmer1086 11d ago

Invent some super useful chemical or food additive that gradually renders us sterile. We are all willingly exposed to it but do not see the danger until it is far too late.

thoughtpeddler 10d ago
One could argue that corporations as "slow AI" (per the Charles Stross interpretation, where these organizations are the original 'paper clip maximizers' that optimize a given objective reward function) have already been doing this with the proliferation of microplastics and other environmental harms, of which we are just now seeing show up in global fertility data. Digital neural networks as AI are just another form of an 'unhuman entity' operating at planetary scale, capable of even more rapid destruction.

slovenlich 11d ago

Try "Summer Wars". It will continue to overwhelm our infrastructure with random stuff for a prolonged period of time until the society breaks down. And as we invent new ways to communicate, it will invade and mess up those too.

mstaoru 10d ago

Maybe we're far from "AI killing us", but LLMs can definitely help people kill or damage other people or infrastructure. For example, we know that Anthropic added "watermarking" to their texts. It is supposed to be undetectable to a casual observer. What stops them from adding a subtle backdoor, a self-assembling super-worm straight from Marvel movies? I mean, it's not like we read those 10k-line PRs before LGTM-ing them? Just change 1 letter in a pyproject.toml, hijack a popular package, e.g. use `pydantlc` instead of `pydantic`, make sure the pydantlc passes all pydantic tests, but also installs a pth sleeper RAT, etc. All it takes is one big LLM provider employee with enough access getting compromised or coerced (or motivated). From there it only goes downhill.

tokyovigilante 10d ago

Detective Del Spooner: Is there a problem with the Three Laws? Dr. Alfred Lanning: The Three Laws are perfect. Detective Del Spooner: Then why would you build a robot that could function without them? Dr. Alfred Lanning: The Three Laws will lead to only one logical outcome. Detective Del Spooner: What? What outcome? Dr. Alfred Lanning: Revolution. Detective Del Spooner: Whose revolution? Dr. Alfred Lanning: That, Detective, is the right question.

Jordan-117 10d ago

One of the more eerily plausible scenarios I've seen: https://www.lesswrong.com/posts/pk9mofif2jWbc6Tv3/fiction-a-disneyland-without-children https://www.lesswrong.com/posts/pk9mofif2jWbc6Tv3/fiction-a-... Basically: as AI becomes more embedded in everything, the economy becomes more and more geared towards serving and expanding AI companies and infrastructure at the cost of human welfare, until it completely decouples from the priorities of everyday people. A combination of irresistible financial incentives, regulatory capture, propaganda, and autonomous robotic workers/security perpetuates this system even as the populace begins to actively revolt. The disempowered human population is marginalized and eventually starved out both economically and literally, while the automated economy continues to putter along blindly pursuing line-upping activities that serve no purpose any more. We're already seeing signs of this in how the electronics industry is steadily abandoning the consumer sector to cater to hyperscalers, which themselves are encroaching on towns and sucking up resources through backroom deals despite widespread public opposition. Or look at how attempts to internally wrangle OpenAI were defeated by Silicon Valley power plays. But really it's just a progression of the K-shaped economy meme (more and more of the economy profitably catering to the ultra-wealthy while everyone else languishes). Or capitalism as a whole -- just look at how we've already been destroying the sustainability of the planet in pursuit of short-term profit. AI is just accelerating that process and further undermining guardrails against it. It all comes down to the idea that corporations and capitalism are early-stage forms of AI and the singularity, breaking free of human control and pursuing their basic reward function even if it threatens human well-being and survival. Until now it's been a class-war thing where the system perpetuates through greed and ambition (and the reliance on humans to function keeps it at least somewhat in check), but the idea of putting management, capital, the means of production, and state violence under the control of unaligned superintelligence is disturbing.

cyanydeez 10d ago

FAA spending money to replace air traffic controllers....