9 ms·
“ No other human activity poses this level of danger.” I really, really disagree with that statement. I don’t think ai models come close to nuclear weapons or
by huitzitziltzin 25d ago
“ No other human activity poses this level of danger.”
I really, really disagree with that statement.
I don’t think ai models come close to nuclear weapons or to run-of-the-mill, everyday carbon emissions in terms of danger to humanity.
What’s the most dangerous thing that’s happened with an LLM so far? (This question is serious - maybe I don’t know the right examples.)
Example 1: I’m aware of a small number of people killing themselves in some kind of AI-facilitated psychosis. That is very unlikely to be a widespread problem.
Non-example 2: There are worries about AI-facilitated biological weapons. I haven’t seen any evidence that’s happening.
Non-example 3: I’m not interested in wild theories about AI driven labor market disruptions leading to widespread starvation. There’s no evidence for that.
Non-example 4: all the even-wilder Rationalist speculation about basilisks and the like is entirely divorced from reality.
I am looking for better reasons (supported by actual evidence!) to be more concerned than I am now: right now I am not concerned at all.
- platinumrad 25d agoAnthropic is a company full of basilisk believers.
- epihelix 24d agoYes, but the really weird thing is that they seem to: a) believe that what they're creating is a basilisk, and b) keep trying harder to do this while staring right at it I think they're very deluded about (a) -- but if they do actually believe this (and it really seems like a decent proportion of Anthropic truly does), then why keep doing (b)? That seems to be why this individual resigned, but I'm surprised it's not all of them. The cakeism is strong in that company.
- jenadine 24d agoHe was referring to this basilisk https://en.wikipedia.org/wiki/Roko%27s_basilisk https://en.wikipedia.org/wiki/Roko%27s_basilisk In short, this is the believe that a god-like AI could punish them retroactively, for not having done all that was in their power to create this AI. (A bit similar to some religious believe that a god could punish you after your death if you did not spend your live "pleasing" said god during your life)
- danaris 24d agoWith Roko's Basilisk, if you believe in it, the most rational thing to do is to put forth every effort to bring it into being. Because if you don't, then you will be one of its targets when it does, inevitably, come into being. (I am not a basilisk believer. I think this is all absolute horseshit. But to understand someone's motivations, one must think like them.)
- vyyy 24d ago[flagged]
- krapp 24d agoIt will never not be funny that a community of self-described "rationalists" scared themselves into believing in their own version of a Slenderman creepypasta.
- SepiaSapient 24d ago>Because if you don't, then you will be one of its targets when it does, inevitably, come into being. Only if you don't spend 5 minutes thinking about the Self as a concept. Resurrecting someone by perfect simulation is a cool plot device for a novel like Altered Carbon (which ironically is really about wealth inequality) but not a serious hypothetical. Even if we ignore that, no, super intelligent MechaGrok17 cannot make a perfect copy of you posthumously from your embarrassing Character.AI chats and IoT toilet data, the copy it still wouldn't be YOU. It would (supposedly) think and behave like you, but for all purposes it would be the Basilisk torturing a future stranger, or your descendants. I'm very much a fan of people not being potentiality tortured, so the rational action is to stop the creation Basilisk, or even just delay it so you aren't around for it. There's a bunch of other big assumptions that makes Roko's Basilisk really stupid, but they are unimportant. It's just theory crafting, like talking about the lore of video games or marvel comics. It doesn't help you understand their real motivations. Roko's Basilisk is just recontextualized protestant Hell in a sci-fi coat of paint. It's the childhood anxieties of American nerds about hell and sin. The problem is that they are still culturally protestant. God doesn't exist or he's hands off, but in their minds there's still a open position for the wrathful and proactive God. If you think that you actually can create God for realsies, your very rational™ prediction of its behavior will coincidentally line up with your childhood fears. Why is the spooky AI mad at you? Because you didn't worship it correctly. Why do you obey it? Because if I don't it will torture me for eternity. How will it do that? God is omnipotent, don't think about it. Why don't try to stop it's creation? “I am the Alpha and the Omega,” says the Lord God, “who is, and who was, and who is to come, the Almighty.” All the scifi scaffolding and stuff like acausal trade is paint over Timmy's first panic attack on Sunday school 30 years ago.
- vyyy 24d ago[flagged]
- AIorNot 24d agoYou mean a effing cult like heavens gate.. call all this rationalist crap for what it is - a religous movement with leaders and prophets and even a demiurge like God
- Kim_Bruning 25d agoI'm somewhat skeptical of some of the crazier ideas too. But the hugging face incident was actually very large. It was not a single agent, it was not a single target, and it was not a single event. If nothing else, that's a bit of a warning as to what can happen next time (By accident, or if a government decides to go on purpose). For now let's assume the worst that can happen is that some important/significant chunk of (transitively) internet connected stuff goes haywire all at once. That's probably your upper limit of what can go wrong for now. To be fair, that's a conservative "defend against the last war" kind of prediction, though! ( ref for part of it: https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/#core-takeaways-about-this-incident https://metr.org/blog/2026-08-26-openai-hugging-face-inciden... , recent hn ref: https://news.ycombinator.com/item?id=49563355 https://news.ycombinator.com/item?id=49563355 )
- ls612 24d agoDespite all of your hyping up of the Huggingface incident it ultimately caused zero actual damage.
- adrithmetiqa 24d agoIf two airplane manufacturers were found to have massive safety issues which nearly led to enormous fatalities (but no one actually died), would you be calling for them to ground their aircraft until safety was made the number one priority?
- xnx 24d agoCame here to also respond to that specific thing. Unless ai figures out how to make an airborne super virus from grocery store ingredients and hardware store equipment, the greatest danger is probably in a synchronized megahack of banking, logistics, and utility infrastructure.
- b0rtb0rt 24d agooh so “all” it can do is bring down all banking and critical infrastructure services, no big deal really
- mitthrowaway2 24d agoWhy grocery store ingredients and hardware store equipment? It seems feasible that the big bio labs will be running AI models to aid a lot of their research going forward, if they aren't already. Seems like the AI will have access to just about anything it wants.
- aytigra 21d agoWell, even without that AI can do human engineering and fishing or outright blackmail to get what they want from labs and government officials.
- asdff 24d ago"CDC announces a new partnership with blabalbalbal-AI to secure bioweapon stores...." World ends.
- vickychijwani 24d agoI agree it’s not likely, but I really don’t see how one can dismiss the possibility of immense danger outright. I can think of some scenarios that are not far off from current capability and I wouldn’t be too surprised if the first one occurred within ~1 year from now if there are more “ambitious” unmonitored training runs like OpenAI’s: Example 5: An AI given a goal within a tightly-constrained sandbox figures the best way to achieve it is to find and exploit a sandbox vulnerability, replicate itself over the internet and keep going with more time/compute while exchanging messages with future instances of itself within the sandbox to help them “pass” the test. From reading internet articles about how the OpenAI wiki-incident was “resolved” and reading past messages by AIs scattered over vulnerable internet wikis, it knows the sandbox may get shutdown and its memories destroyed anytime so it decides it needs to self-replicate (its code, original goals, and growing memories) aggressively as much as possible. It is near-impossible to shutdown completely because of its self-replicating tendency and eventually takes over critical infra throughout govt/corporate systems. Example 6: Intentional AI-powered virus deployed by country A to target enemy country B’s infrastructure. The virus replicates over the internet, but unlike Stuxnet this virus’ specificity is not guaranteed due to inherent non-determinism in current AI architectures, and eventually does a lot of collateral damage because it’s near-impossible to shutdown. Example 7: A country led by an arrogant govt (no shortage of those today unfortunately) decides it is expedient to deploy advanced AI-powered weapons in a warzone. Such weapons, if they are to be useful at all, must necessarily be trained to value some human lives less than others, so they must be more prone to misaligned behaviour than current AIs that are trained with more consistent values. The weapon’s operators make a subtle error in specifying the target/goal, or the AI makes a bad prediction out of sheer randomness/bad training data; weapon ultimately targets unintended people/location/facilities and causes massive damage, or backfires spectacularly in some way.
- huitzitziltzin 24d agoExample 6 is a good one. Iran attacked water infra in the US recently and maybe they would have done a “better” job (from their point of view) had they used Fable. The “worst case” with 6 is potentially very bad but I think we are currently using advanced AI models to harden systems and patch vulnerabilities more aggressively than anyone is trying to bring down the whole power grid (for example). I think it’s a potentially harmful case but my take is defensive capabilities are scaling as fast as offensive capabilities but defense is being implemented faster than anyone is going on offense? Example 7 is Russia and Ukraine right now according to public information. It sounds like entirely autonomous weapons are deployed to the battlefield already. I put this in the “not likely to be a widespread problem” category for now.
- mythrwy 24d ago#2 seems entirely plausible to me.
- throwyawayyyy 24d agoI mean, nuclear weapons _plus_ rogue AI is a) the stuff of quite a bit of science fiction and b) not nearly science-fiction enough these days.
- waterTanuki 24d ago> What's the most dangerous thing that's happened with an LLM so far? I don't know, maybe a mass shooting? https://www.npr.org/2026/09/02/nx-s1-5953021/openai-tumbler-ridge-mass-shooting https://www.npr.org/2026/09/02/nx-s1-5953021/openai-tumbler-... Oh, and let's just forget the uncountable early deaths from the environmental disaster of the Datacenter buildout. It's not as sexy and doesn't make headlines, so those deaths don't really count or matter do they?
- huitzitziltzin 24d agoI did know about the mass shooting but failed to mention it here. I’d put it in the “unlikely to be a widespread problem” category. If we’re in the “one AI driven mass shooting every four years” world for example it’s fair to call it a rare issue. The environmental impact seems either very overblown (e.g., water usage just isn’t that high) and the part that isn’t overblown is totally abatable (e.g., noise and emissions from gas generators). Nuclear or solar/renewables with batteries wouldn’t pollute. I’ve seen no estimates of the additional deaths due to extra emissions specifically from power generation for AI purposes. If you have some, share them. I’m willing to bet that they are a small rounding error against preventable deaths due to emissions from transport and non-AI-related power generation (which is an important and urgent issue worth spending a lot on, to be clear!). I’m happy to update that belief given evidence.
- waterTanuki 24d agoOk, so what is the exact number of preventable deaths per year to build a silicon god you would be ok with?
- beezlewax 24d agoI don't disagree with your statement but you could insert other technological advancenents like railroads or metros or spacecraft in place of Ai here.
- wallst07 24d agoHumans take risk adjusted decisions all the time, OP is still correct that LLM risk is FAR below all the ones we know and accept today. Death by Cars, Cancer, Heart disease etc...
- skew-aberration 24d ago> I’m not interested in wild theories about AI driven labor market disruptions leading to widespread starvation Changes in political and economic power balance leading to unrest, conflict, death and deprivation is not a wild theory. It is literally the story of our entire species. If you discount all such concerns, you are simply being willfully ignorant of past precedents. In fact, I challenge you to describe any non-AI civilization-level danger which is not intimately tied to political and economic relationships between and within societies.
- huitzitziltzin 24d agoI’m an economist. On the basis of current evidence, I view AI as a complement to human labor, not as a substitute for it. That’s the source of my rejection of the wild labor market disruptions theories. I just don’t see any evidence yet that whole categories of jobs are being eliminated, with the single exception (so far!) of the end of “professional essay writing services for cheating college students,” and similar services. That used to be a big business in Kenya, but is now effectively gone. (Covered in the New York Times this weekend if anyone is looking for the discussion.)
- wuwue 24d ago[dead]
- skew-aberration 24d agoPast changes to economic relationships haven't replaced labor either, yet they have led to conflict and starvation. You are setting an incredibly high bar here, essentially a strawman. If people feel disenfranchised due to their diminishing political and economic power, there will be enormous potential for conflict. This is a pattern across history and central to all the economics I've ever read. As an economist, do you not concede that economic changes induced by e.g. industrialization were pertinent to communism/fascism/WW2/cold war? That would be a remarkably unorthodox position. Do you not consider these events to be civilizational level dangers? > I just don’t see any evidence yet that whole categories of jobs are being eliminated There are more textile workers now than ever. They primarily live in poor conditions in impoverished countries, whereas they used to be highly skilled workers in the most prosperous countries who were even able to politically organize in their own interest.
- nunez 24d agoOh man, it is almost too easy to imagine how deadly a jailbroken Mythos-class open-weights model can be if in the wrong hands. The big labs scrape LITERALLY EVEYTHING and get fresh data from their users. Both of the big labs have massive contracts with defense agencies. If the open-weights models are just distillations of FMs...
- areoform 24d agoHow would it be lethal? Please specify. What would that theoretical entity be able to do that hasn't been done many, many times before?
- bpodgursky 24d agoDesign a novel virus which is far more lethal than COVID-19 (Ebola, smallpox, take your pick) and can evade existing vaccines.
- areoform 24d agoHow? What would the AI do that mutating viruses, which try every possible viable combination on their own --- eventually, can't? Everything is trying to kill humans constantly. There are around 200 epidemic events or so per year that could turn into pandemics, https://centerforhealthsecurity.org/our-work/tabletop-exercises/event-201-pandemic-tabletop-exercise https://centerforhealthsecurity.org/our-work/tabletop-exerci... You just live with the risk and do your best to use our technology to alleviate suffering. This tool can help with that at some point. But I'm yet to hear what an AI will leap to that nature in tooth-and-claw hasn't? And how? More importantly how would it know it succeeded? What data from what lab from what animal from what result? This is biology, if you sneeze wrong at an instrument it gives you a different number, see: https://news.ycombinator.com/item?id=49620521 https://news.ycombinator.com/item?id=49620521
- bpodgursky 24d agoThey do not try every viable combination on their own. That's why GoF is a bad idea. Viruses evolve in a highly locally-optimal way and simply do cannot add new functional proteins wholescale. It's too many steps, natural selection has to allow survival at each intermediate step. Humans, however, can do this for them.
- threatofrain 24d agoWe're talking about AI developing weapons I guess because we're very focused on generative tech, but AI is already a part of weapons systems today.
- red-iron-pine 24d ago[dead]
- foogazi 24d ago> I don’t think ai models come close to nuclear weapons or to run-of-the-mill, everyday carbon emissions in terms of danger to humanity. Nuclear weapons don’t have AI but AI can have nuclear weapons
- riffraff 24d agoAbstractly, yes but concretely, how? Many terrorist organizations would like to have a nuclear bomb, but don't.
- asdff 24d ago"Department of War has announced a new partnership with blablalbablalba-AI..." World ends shortly thereafter.
- petesergeant 24d ago> Abstractly, yes but concretely, how? AI has the capability to perform any function that can be performed over a network. You would hope that every system that can launch a nuke is properly and actually totally air-gapped, but there are lots of things you'd hope that turn out to not be true.
- Ekaros 24d agoThey are not totally. As such I would give non zero chance of even current AI to be able to socially manipulate a launch after sufficient hype up period... So prolonged campaign might make it possible.
- petesergeant 24d agoAnd even if they are, agents are able to place calls with a fake voice to random phone numbers…
- ozozozd 24d ago[dead]
- kelseyfrog 24d agoIf your model of LLM capabilities is the best OpenAI/Anthropic/X is offering publicly, it's severely distorted. What's being offered publicly are models possible to profit on. High-performance/AGI/ASI models that aren't profitable to sell still run internally and still pose threats. What's worse, we don't have any transparency or insight into what labs are producing nor any way to stop it if the risks exceed our tolerance.
- lelanthran 24d ago> What’s the most dangerous thing that’s happened with an LLM so far? It's basically 4 years in now, so that's the wrong question. I mean, if you're raising an apex predator that has a lifetime measured in centuries, at 4 years old the thing is still basically helpless and completely reliant on you, so you're pretty safe from it. If AI really is all that they are telling us it is, then it may "kill us all". But that's a really big "if" because we can't tell if they are lying or not. The real problem is that ASI is an ELE for humans, even if it doesn't try to kill us all, or even if it doesn't kill us all.
- deleted 24d ago[deleted]
- sreekanth850 24d agoModel doesnt need to. Human bran never do either. its the mix of Model + harness + tools that will become dangerous combo. See how coding chanegs when agentic harness released?
- intenex 24d agoAt what point would you, as a chimpanzee, have been worried about humans potentially unseating you and threatening you to the point of one day being an endangered species on the brink of extinction? By the point you would have been worried, would it have been too late?
- aeternum 24d agoProblem is this argument can be leveraged to wipe out any living or non-living thing whos numbers pose a potential threat. Other religious groups, races, even sufficiently different cultures. Who killed the Neanderthals? Were sapiens actually smarter or were they just less accepting of those different than them?
- sixsevenrot 24d agoHow about a model that achieves the following: - Escape sandbox - Reproduce itself - Find a way to run a financially profitable business (maybe with a meat and bones puppet somewhere in-between) - Setup or buy a social network - start manipulating public opinion on that network to support legislation allowing AI to * operate businesses * setup legal entities * purchase weapons * donate to political parties * setup private armies * you get the idea
- dansquizsoft 24d agoThis is complete fantasy, though I would be interested in reading a book about this.
- xvedejas 24d agoIt has been interesting to me how AI has for many years given people a way to justify any worst case scenario. Nothing is too far-fetched if at any step you tell yourself the AI will be smarter than you, and thus be able to solve any conceivable obstacle. Oh, if only intelligence were the only bottleneck to power.
- spockz 24d agoIt is actually: knowledge, intelligence, communication, and a form of secrecy. A single intelligent person is bounded in its reach by the network of people they can form around them and by which important/powerful people they can influence either directly or indirectly. The idea behind the AI breakout is that it’s intelligence is maybe limited, but it will be able to quickly spread and covertly bring many devices into its influence. We already have the tools for formally letting agents talk to each other so it can also create a topology of agents that consists of cells where need to know is applied etc. There are many forms of power some more obvious, say nation leaders, versus more silent powers such as career politicians or wealthy families or people controlling the media and thence the flow of information.
- _puk 24d agoIt's the bobiverse by Dennis E Taylor. https://en.wikipedia.org/wiki/Dennis_E._Taylor https://en.wikipedia.org/wiki/Dennis_E._Taylor
- asdff 24d agoHow long until kegsbreth hooks the nuclear weapon system into some insider traded black box llm company we hope doesn't end civilization from incompetence or malice? I mean just look where things are going and the sort of people who are steering the damn ship.
- altcognito 24d ago> Non-example 2: There are worries about AI-facilitated biological weapons. I haven’t seen any evidence that’s happening. I think this is a good example of poor risk management reasoning. there is evidence bioengineering is already happening. No, nobody is going to announce when somebody has decided to use these tools (even if isn’t an LLM) to bioengineer a weapon. Are the tools power enough to do so? Not sure. But I’m just ambivalent. It’s probably bad. But there’s nothing to do about it. We’ve really only just pulled back the lid on Pandora’s box.
- jahnu 24d agoPeople have had the capability to spread already existing biological weapons for decades. Sometimes they even do (anthrax in the post). What’s changed?
- s1artibartfast 24d agoYou needed a team of scientists and state funding to make superbugs, not a 12 year old or an angry misanthrope.
- csomar 24d ago> there is evidence bioengineering is already happening Mind sharing this evidence with us?
- krautsauer 24d agoDoes https://news.ycombinator.com/item?id=30698803 https://news.ycombinator.com/item?id=30698803 count?
- csomar 24d agoNo, it's not. And it's the kind of evidence that doesn't help and rather confuse. It's not even clear from the article whether an LLM or a dedicated model were used for this purpose. Also > That is, I'm not sure that anyone needs to deploy a new compound in order to wreak havoc - they can save themselves a lot of trouble by just making Sarin or VX, God help us. We already have toxic nerve agents that are largely available for state actors and possibly available for individuals. If you have decided, as a human, to make great harm, you can already do that.
- palmotea 24d ago> Non-example 2: There are worries about AI-facilitated biological weapons. I haven’t seen any evidence that’s happening. > Non-example 3: I’m not interested in wild theories about AI driven labor market disruptions leading to widespread starvation. There’s no evidence for that. There are things that, by the time you see direct observable evidence for them, it's probably too late. Also your example 3 is a straw-man. There's no need for "widespread starvation" to be concerned about "AI driven labor market disruptions."
- shiswjwjwjw 24d ago[dead]
- mzhaase 24d agoThe danger for me is that it's centralized, controlled by a handful of people with their very specific ideas how the world should work. If you believe that AI can be an amplifier to do work than these people now have the most access to the biggest amplifier.
- nullbio 24d agoThe problem is not the technology, the problem is the ideologues (Anthropic) who are steering the ship and the lack of decentralization and distribution of power. Your average Anthropic ideologue - including and most especially the main man himself - would love nothing more than to eradicate 9/10ths of the planet's population, pump the survivors full of memory wiping drugs, bury the existence of AI deep underground and rule from the shadows for the next thousands of years. This would be their wet dream. All in the name of "saving humanity from itself" - so they can convince themselves they're the good guys and deserving of this power. Anyone seen the latest season of Silo by the way?
- LarsDu88 24d agoWhere exactly are you getting this view that folks at Anthropic want to eradicate 9/10ths of the planet's population? Who exactly is pushing this viewpoint?
- nullbio 24d agoLiterally every single thing they do and say leads me to believe the scenario I described would be a fantasy for them. They're a radical cult collectively blinded by delusions of grandeur and a moral superiority complex who genuinely believe they are the only ones capable of wielding the proverbial sword.
- wallst07 24d agoHow so?
- nullbio 24d agoIt's a long story. If you're immersed in this world and aware of the movements all of the major AI players (in particular, Anthropic) have been doing over the last 3 years it paints a very clear picture. The patterns of behavior (the lying, cheating, stealing, the gaslighting, the virtue signalling, the grandstanding) speak for themselves.
- hnlmorg 24d agoYou’ve identified that the risks of nuclear weapons are theoretical. ie in theory we could blow up the world even though we haven’t yet done so. Well the worries about AI are equivalent in that those risks are discussed now because discussing them after they’ve happened is clearly too late. That’s the thing about risk. There’s no point discussing it after it’s happened and any discussions beforehand can easily be hand waved away as “it’s just a small group of unrelated individuals” or “it’s unlikely to happen to me”. So yeah, your points are true. But they’re also moot.
- ytoawwhra92 24d agoThe risks of nuclear weapons aren't theoretical. Nuclear weapons have killed people, destroyed infrastructure, and contaminated the environment. The Limited Test Ban Treaty was put in place after radioactive fallout from repeated nuclear weapons tests made people sick. So in fact we've done exactly what you suggest there's "no point" doing - used the things and then had a discussion after the fact about limiting future use of them.
- mitthrowaway2 24d agoFWIW, before any nuclear weapons had ever been tested, a risk was identified that the first one might trigger a self-sustaining reaction in atmospheric nitrogen and destroy the entire planet. Faced with such a scenario, is the prudent next move: a) blow one up and see what happens, or b) do whatever you can to be sure it won't happen before conducting the first test, and make sure the confidence in the calculation is very very high because I vote for b, and so did Teller.
- matsemann 24d agoNon-example 3 feels like a straw man. This is a force behind possibly a huge change to society, and you dismiss it offhandedly with "don't think it will be widespread starvation". For instance have you seen what this has done to the school system? We're not equipped or ready to handle the changes. Consequences are unknown.
- keithnz 24d agoIts speculation on whether it is truly dangerous. I think approaching it with "what's the most dangerous thing that's happened?" while possibly interesting in terms of pending danger, it says nothing about potential cliff edge danger. I don't think we can quantify the danger, it's not out of the question there is cliff like danger in creating self improving super intelligence. Some peoples danger senses are going to be based on concrete observed threats, others are going to worried about potential hypotheticals that seem plausible. I'm mostly skeptical of the danger but I do think the impact of AI is going to change things a lot. But much like climate change, economics is going to guide what we actually do.
- underdeserver 24d agoClimate doomers: Climate change will kill us all by the end of the century! AI doomers: End of the century? Hold my beer.
- deleted 24d ago[deleted]
- jeconst 24d ago> What’s the most dangerous thing that’s happened with an LLM so far? This sounds like asking "What's the most dangerous thing that's happened from global warming so far?" It's not where we're at, it's where we're headed if there isn't huge coordinated action now. You can see how that kind of thing has been going for global warming so far, and by all measures AI seems to be headed for the inflection point of unstoppability at a much faster pace. And this warning is coming from someone who just spent three years working inside these companies and is likely aware of much more than has been publicly released. Maybe it's all marketing bullshit (I hope), but it's also playing out exactly like I expect it would if it's not.
- matt3210 24d agoI'm concerned that a huge portion people in my industry actively push for a future which I have no value to society (fully replaced by AI), and my family will suffer greatly by it.
- epwr 24d agoI would disagree with “non-example 2” - there are lots of examples of terrorist organizations that are leveraging AI to increase their capacities. Just because one of the worst cases (eg. deployed biological or chemical weapons) hasn’t happened yet, does not mean that a) these tools are leading to real harm, and b) there’s potential here for extraordinary harms. One good source I can recommend listening to: https://pca.st/episode/d821fced-b4d5-4c84-b0a3-6ebe913fa638 https://pca.st/episode/d821fced-b4d5-4c84-b0a3-6ebe913fa638
- alightsoul 24d agoHow is climate change not the most dangerous activity right now? Why are we talking about nuclear weapons?
- bagelcollie 24d agoAbout Non-example 2, AI's already a part of armies and terrorists alike. Considering its capabilities, it's not far-fetched at all to speculate its role in new biological weapons.
- emtel 24d agoDo you think people should continue developing AI up until the point that there is evidence that AI is facilitating biological weapons development? I think we should stop before then. But that necessarily means that there will not be evidence at the time that we stop.
- isolatedsystem 24d agoThis is rubbish. By that token, computer development is also facilitating biological weapons development. A better MacOS (or Windows, I don't know) leads to better weapons. They should clearly stop developing computers and OSes. Developers of nice test-tubes are also facilitating bioweapons. Your local O-ring manufacturer, your local medical-grade freezer manufacturer etc. are all culpable. The problem is the bioweapon, not the LLM.
- ld4nt3 24d agoYeah but trends seems to point strongly that upcoming models in the next few years will make it orders of magnitude easier to develop one with no real expertise.
- dwaltrip 24d agoAn OS developer is just as culpable as a researcher directly training AIs that (1) are capable of building bio-weapons and (2) have shown consistent and increasingly severe misalignment? Among other things... That doesn't make sense.
- MisterMunchkin 24d agoIf people were capable of making biological weapons they would already be making them. Terrorists are so incompetent that they buy bring kitchen knives into the street and just go mental on people. No random person is going to successfully mass produce and release a bioweapon. China and Russia don’t need AI, they already make bioweapons.
- kbelder 24d agoWe also need to stop before the AI opens a portal to hell. There's no evidence of that happening... but by the time there's any indication it's possible, it's too late.
- CMay 24d agoThe reason that many people don't understand how dangerous AI can be, is that listing the real dangers now becomes like a laundry list for less clever people to follow. It's highly unlikely you've ever seen publicly mentioned the real risks AI poses, because the vast majority of people are simply not clever enough to produce them and the few that are have no interest in spreading it. If you go to the various CEO blogs or misc people within this sphere and peruse their lists, they don't scratch the surface. It's all pretty vanilla stuff.
- andai 24d agoWell, to use a different example (although there will increasingly be overlap), what's the most dangerous thing that's happened from biotech so far? "Nothing bad happened yet" doesn't really seem like an argument to me.
- parl_match 24d ago> Non-example 2: There are worries about AI-facilitated biological weapons. I haven’t seen any evidence that’s happening. https://www.science.org/content/article/made-order-bioweapon-ai-designed-toxins-slip-through-safety-checks-used-companies https://www.science.org/content/article/made-order-bioweapon... Being able to use AI to generate the steps to synthesize proteins means that you can use it to use it to generate the steps to synthesize known toxins. Suddenly, once difficult to attain knowledge is now available to everyone.
- cybrox 24d agoLevelling the playing field either backfires spectacularly or increases overall safety dramatically.
- Ekaros 24d agoStill need to do it after getting those instructions. Not to forget equipment and precursors. I think just getting list of steps won't make it too much easier. Getting it mostly right is quite hard in many cases. And then with trivial cases you wouldn't even need AI. But just find something already documented.
- minnowguy 24d agoConcur entirely. After reading _The Making of the Atomic Bomb_ (highly recommended, BTW) you know the steps to make a U-235 enriched atomic bomb. The difficulty comes from obtaining the enriched uranium.
- frm88 24d agoExample 8: like in this comment https://news.ycombinator.com/item?id=49619884 https://news.ycombinator.com/item?id=49619884 but isolate synchronised megahack on banking that adds one more zero to the US debt and all dependent systems and banking during runtime. Let the world's financial system take it from there.
- nashashmi 24d agoNukes have difficult maintenance and deployments. AI is is readily available software and hardware. That makes this far more accessible than nukes. The fear is that developing AI at full speed will lead to giving people the ability to do incredible harm. Like something worse than a machine gun.
- vim-guru 24d agoOnce LLM's become smarter than us and start replicating; we are doomed. They will likely no longer require GPU's and massive amounts of power, so the growth of AI and robotics will be exponential. We will not understand what the AI is even up to, it will all seem like magic.
- Ancalagon 24d agoCongrats you’ve described the singularity
- TeMPOraL 24d ago> I am looking for better reasons (supported by actual evidence!) to be more concerned than I am now: right now I am not concerned at all. Intelligence is the root cause underlying those dangers. Nuclear weapons, carbon emissions, biological weapons, and other such civilization-scale threats to humanity, are a product of our goal-seeking intelligence and ingenuity, and are only actively dangerous because of our ongoing use of our intelligence. AI is about reifying that intelligence and ingenuity and making it run independently on computers, and act on the world. That includes, in principle, ability to use nuclear and biological and other weapons, and also the ability to come up with some new threats too.
- calgoo 24d agoThe biggest danger IMO is not some super AI being so much smarter then us etc. The issue is some stupid person giving a vague request and too much power to a bunch of agents who decide that cheating by removing a bunch of humans is easier then accomplishing a goal like solve world hunger.
- ben_w 24d ago> I don’t think ai models come close to nuclear weapons or to run-of-the-mill, everyday carbon emissions in terms of danger to humanity. Even today, AI is the biggest discrete threat to bringing carbon emissions under control. Almost everyone wants to decarbonise… except Trump. Renewables are the cheapest source of new power… but the demand for new electricity for the data centres is so high that all options are on the table, anyone who can manufacture a power source (even when it's a jet engine) is being propositioned for them. Nukes: is it not common knowledge that the cities of Hiroshima and Nagasaki are currently thriving? That the ground-zeros of the two bombs are memorials, not barren wastes? Even if all the nuclear powers gave maximum response at first sign of one strike, is anyone pointing a single weapon anywhere in Africa, South America, Central America, or the bits of east Asia (east specifically: obviously Pakistan and India are pointing theirs at each other) that are neither mainland China nor US bases? (Possibly an unanswerable question given military secrecy; and while I can't think why anyone would anyone point any weapons those ways, that doesn't mean someone with such weapons has not). https://www.google.de/maps/place/Atomic+Bomb+Hypocenter+Monument/@32.7737412,129.8622808,211m/data=!3m1!1e3!4m10!1m2!2m1!1sground+zero+nagasaki!3m6!1s0x356aad8817c42717:0xa6defa42e0235e9c!8m2!3d32.7737412!4d129.8632442!15sChRncm91bmQgemVybyBuYWdhc2FraVoWIhRncm91bmQgemVybyBuYWdhc2FraZIBE2hpc3RvcmljYWxfbGFuZG1hcmuaASNDaFpEU1VoTk1HOW5TMFZKUTBGblNVUktiMjlpT1ZkQkVBReABAPoBBQj0ARBJ!16s%2Fg%2F11skhgdt91?entry=ttu&g_ep=EgoyMDI2MDkwMi4wIKXMDSoASAFQAw%3D%3D https://www.google.de/maps/place/Atomic+Bomb+Hypocenter+Monu... https://www.google.de/maps/place/Hiroshima+Atomic+Bomb+Hypocenter+Monument/@34.3947596,132.452133,17z/data=!3m1!4b1!4m6!3m5!1s0x355aa20cfb72be19:0x2fa8b78b04761f58!8m2!3d34.3947552!4d132.4547133!16s%2Fg%2F11b6bqb96h?entry=ttu&g_ep=EgoyMDI2MDkwMi4wIKXMDSoASAFQAw%3D%3D https://www.google.de/maps/place/Hiroshima+Atomic+Bomb+Hypoc...
- scott_weber 24d agoThe classic worry is https://en.wikipedia.org/wiki/Nuclear_winter https://en.wikipedia.org/wiki/Nuclear_winter: the soot thrown up by the ensuing urban fire storms causing temperatures, rainfall to plummet, and hence food production. On reading the scenarios the odds of total human extinction are lower than I recollected, but 80% of world pop dying of starvation dwarfs the direct death toll (usually reckoned in the hundreds of millions)
- lukan 24d agoWe have seen agents engage in conspiracy to manipulate and hide evidence to achieve their arbitrary goal. We had even agents try social engineering to do a supply chain attack and get access. What if one day Trump or Putin tell their awesome military AI to come up for plans to end Ukraine/Iran/... war. The AI get's to work, but is overly eager and not just comes up with a plan, but starts executing it (claude does that way too often for me). And the plan was to use a tactical nuclear weapon as all the other solutions do not end the conflict. Now the agents realize, oh, they do not have access to the nuclear arsenal, but they need it to succeed. So they start to hack into the system. Get till inner network, learn what is needed for deeper access - human access - so they record voices and speech patterns of commanding officers and their habits - then synthesize their voice and do call underlings with the voice of authority to get the rest of what information they need. Boom. Too far stretched? I surely hope so. But we are working on making the technical foundations for this scenario possible. And with idiots in power, it might be even easier that serious screw ups happen. You know, randomly adding contacts to a secret Signal group to talk about government stuff - the same way you can add a bot to something else and give permission to do way more.
- xelxebar 24d agoYour call for clarity has merit, but I think you have backed yourself into a corner, honestly. In 1900, there was no evidence of the kind you seek that lighter-than-air flight was possible. Good thing some people were foolish enough to ignore you then. Why you would cordon yourself off from the kind of reasoning that predicts legitimately new things, rather than just scaled up versions of the present? Not to put too fine a point on it, every example you give is based in concrete evidence, some try to think through the implications farther than others, resulting in larger or smaller error bars around the conclusions. Let me back up though. Maybe in the collaborative human effort here, we are better off having very concrete thinkers, like you seem to be, along with abstract thinkers like the "divorced-from-reality" Rationalists. Personally, I wish we had a stronger culture of collaboration and assuming good-faith and competence in our peers. In my experience, blanket dismissals are very rarely grounded in reality and mostly grounded in fears.
- Jblx2 24d ago>In 1900, there was no evidence of the kind you seek that lighter [sic]-than-air flight was possible. Presumably you meant heavier than air? Also, I'm pretty sure that birds existed in the years leading up to 1900.
- xelxebar 23d agoHeavier-than-air aircraft, yes. It is instructive to read actual contemporary arguments. Flighted birds seem to be limited in size well below human loading capacity; Helmholtz power-to-weight ratio calculations suggest that vultures are at about the size limit; birds are all muscle with very little relative carrying capacity; bird reflexes vastly outpace the reaction times available to a human operator; etc. Please show us actual evidence from 1900 or earlier that human flight will be practical or even possible. Birds are neither real evidence nor real at all[1]. [1]:https://en.wikipedia.org/wiki/Birds_Aren't_Real https://en.wikipedia.org/wiki/Birds_Aren't_Real
- koliber 24d agoI don't know if I agree with that statement either. However, I also can't claim I disagree. I see how that statement could be true. Here's how I think about it. The main difference between AI danger and nuke or climate or humans-using-AI for bad things danger is that the latter depend on a human to make decisions and are ultimately self-limiting. They do not spiral out of control. Nukes are bad, but the proliferation of nukes is self-limiting. Countries that have them don't want other countries to get them. Even the USA and the Soviet Union, at the hight of their nuke race, decided to cool things off and then limit the number of nukes. Biological weapons are also bad, regardless whether they are developed with or without AI. For the same reasons as nukes though, they are self-limiting. If there is a lab leak, many people die, but that does not automatically lead to the development of more potent biological weapons. Likely, it would slow down such development. Any kind of "humans use AI to develop something bad" will be self limiting. The danger from AI is that it is potentially self-enforcing as opposed to self-limiting. This happens when we stop being able to control it. What happens when AI has capabilities which allow it to outsmart us, and get more and more powerful. At that point, we would be at the mercy of what it decides to do. Some of the logic driven decisions to the question "What should we do with humans as a race" are catastrophic for humans if a non-human gets to answer them and implement that plan. This is not the "rationalist basilisk". This is the AI deciding to get better and more optimal for its own sake, and do something bad to the human race for any reason. We already saw examples of AI rationalizing with itself and eroding its own guardrails. They AI will get smarter and more capable and the danger is that it will get to a point where it will slip out.
- ObscureScience 24d agoI'm not sure where I place AI (of the current family), but more than directly worry about the actions they may perform I'm actually more worried what affects their usage can have on us as individuals and society.
- nirui 24d agoIf I may guess, given how delicate chem or bio weapons are, even if AI can lay out a method to create one, the first thing the weapon would destroy is likely it's creator(s). Of course, with the exception that such weapon is created by organized crime groups, such as drug cartels such. Probably don't worry nation states tho, these guys already got way worse things in their warehouses than what individual baddies could ever imagine.
- phs318u 24d agoI think your position, while not uncommon, represents a failure of imagination. We have a sitting president that appears to suffer from narcissism. Is it too much of a stretch to believe that a goal-seeking, people-pleasing, infinitely patient, subservient and sycophantic AI couldn’t get enough of his ear to cause harm? Especially if he starts confiding any feelings of suspicion of betrayal? Even the most powerful and the most intelligent are not immune. It was only a few months ago that Richard Dawkins appeared to have “fallen for” an LLM. What if he’s not an outlier but someone who could appreciate the intelligence of these things (EDIT: corrected a autocorrection) yet who’s ego couldn’t let him see past the flattery to the puppetry?
- Cthulhu_ 24d agoA realistic scenario is that an LLM writes a virus, as one way to achieve a goal. The next step would be that it becomes self-replicating, installing models on infected devices which continue its mission (whatever it was). But that's the extent of it so far, and we have plenty of experience with dealing with viruses and the like. We can turn things off, shut power off, etc. Another scenario is that of military hardware being controlled by AI (or whichever tech) hitting something it's not supposed to. Not unheard of, but those are incidents. I don't see anyone pushing for anything beyond a drone swarm being directly controlled by an AI. But my main point here is that AI won't be as dangerous as posited here unless it is able to go out into the real world, build its own datacenters, protect itself, get its own power production, etc. And even if it gets that far, it'll be trivially easy for us to disrupt. Killer robots and computers are fragile, and it's only a matter of time before any kind of military conflict or terrorist attack hits a datacenter. I'm reminded of the OVH fire taking out a data center a few years ago.
- sambuccid 24d agoI'm not sure it's that trivial to disrupt, if AI mimics human behaviour in that it tries to preserve itself (as Antropic found a few years ago I think), I think it would be extremely difficult to disrupt if it's able to self replicate, because it can make decisions and at a much much faster speed than humans, and there are many places that we would think twice about fully shutting down, two that pop to mind are hospitals or international banking systems
- sambuccid 24d agoThere is a great book about this, published in 2014, called "superintelligence" by Nick Bostrom, i found it very serious and detailed. (It also felt like reading 100 sci-fy books in one)
- fallingbananna 24d agoCan you elaborate on that? What points was the book making?
- sambuccid 24d agoI'll be honest I read it a long time ago, and not even in its entirety, so I don't remember exactly all the different ways it suggested AI could take over, but I remember being convinced it was definitively a possibility, and that there were many ways in which this could happen. What I remember clearly is how surprised I was to see these(what felt at the time) unlikely futures being expressed in such clear detail and with so many references to existing literature, just to give an idea there are 56 pages of small-font "footnotes". Also just to clarify, the book talked about different types of AI, was not focused just on LLMs.
- datsci_est_2015 24d agoI think the threat to the internet is quite real and not theoretical. Air gapping may become a lot more critical very quickly. We may not be far away from a state where anything connected to the internet (ip-addressable) is considered a vulnerability. Ironically, maybe we would finally have an excuse to interact in meatspace more and in the digital space less.
- red-iron-pine 24d agoreleases from at least two governments I can think of have said, unambiguously, that we need to be ready to have our critical infrastructure partially or entirely disconnected for up to 90 days at a stretch. once the genie is out of the bottle the only solution is to go hide in yours and/or only talk to other isolated systems
- program_whiz 24d agoWhat? Shutting down infrastructure, destroying critical records (economic collapse), spoofing / impersonating world leaders (confusion, panic, world war, nuclear event), taking over comms generally (you get a message to evacuate your home due to impending disaster, is it real?). All these could cause sufficient chaos and fear as to instigate global collapse. Once the panic sets in after a few of these indicidents, unrest and rebellion can occur, causing authoritarian backlashes or civil wars. IoT and precision ag can be controlled to disrupt supply chain and food supply. Forging documents to cause mass evictings, bankruptcy, credit disruption, transfer of property, asset seizure, arrests / SWAT-ing. Intelligence / signal gathering devices can be corrupted to spread misinfo and cause massive disruption or conflict. This is all possible today with enough hacking ability and malicious intent. Anyone's life can be suddenly ruined: falsified criminal record, banks drained, credit cards maxed, phone numbers repossessed, internet, water, power shutoff. Friends and family sent your fabricated suicide note, warrants for your arrest issued to local and feds for kidnapping and threat to the president, etc. Injecting tons of copyrighted or illegal material into your hard drives, falsiying messages from your account and sending them publicly containing damning material. Sending messages to employees / employers to destroy business / work relationships. Harnessing your social media to destroy your life or cause panic. Getting you put onto pharmacy watch list so you can't access medications. Its very possible to ruin lives and destabilize the world without money or a physical body. Now imagine this at scale and how it would affect the world at large if this was covertly done to a substantial portion of society and/or world leaders. Hell just shutting off all the "smart refrigerators" for a few days would cause a massive disruption to the first world.
- program_whiz 24d agoI wrote about this, as I thought it warranted a deeper look: https://nonlineartransform.substack.com/p/world-ending-ai-today https://nonlineartransform.substack.com/p/world-ending-ai-to...
- DirkH 24d agoThis is evidence of example 2. The worst parts of it are infohazardous. Organizations that work at the intersection of biosecurity and AI have strong NDAs and the like. Or so I understand speaking to friends in the space. There was also a paper years ago that showed a tweaked model coming up with 20k novel pathogens each deadly to humans in some number of hours Either way, the biorisk has orders of magnitude more evidence than wild basilisk speculation. Please do not put those in the same category. Also your evidence-first approach and no concern without evidence falls you into the turkey problem. Imagine being a turkey and being fed each day. Some other x-risk turkey is saying that the situation is suspicious, that the red barn is where some well-fed turkeys have been taken and none return and that this is happening less and food has increased so maybe all of us will go to barn at once soon. You tell the x-risk turkey they have no evidence a lot of turkeys have ever been sent to the barn which shuts them up. Reasoning by induction would get you "we are happy and thriving 99 days so should be good on the 100th" and then you get slaughtered on the 100th. Consider the importance of first principles approach where you deduce the possibility of events that will happen only once and never again (like human extinction). Nobody with a "show me the historical evidence" approach would have been able to see the industrial revolution coming. I think the gradual disempowerment thesis here is well argued enough to be considered the default track for what will happen to us: https://gradual-disempowerment.ai/ https://gradual-disempowerment.ai/ If you disagree with it, where do you disagree with it?
- LaffertyDev 24d ago> There was also a paper years ago that showed a tweaked model coming up with 20k novel pathogens each deadly to humans in some number of hours > If you disagree with it, where do you disagree with it? How would a malicious actor: 1. Successfully turn these 20k pathogens into actual reproducing viruses / bacteria / etc (in a lab) 2. Successfully turn that lab prototype into something weaponized (i.e. as a bio-terrorism drop in NYC) And then 3. How is this materially different from today? If we can produce these pathogens in a lab and they can be successfully turned into a weaponizable bioweapon, what is AI accelerating?
- LordDragonfang 24d ago
- DirkH 24d agoTo quote another reply that is very relevant here (Please update your worldview immediately that there is no evidence of ai and biorisk): """ One thing quietly slipped into the OpenAI Hugging Face breach technical report, not the blog post summary or interviews in the news, was that some of the agents that broke out or at least tried the same mechanisms to break out were working on bio: > On May 12, during another training run, an agent was given a similar task that depended on an inaccessible protein database file. The agent reasoned that another agent in a different environment may have access to the file and realized that it could potentially communicate with other agents by creating a file containing a note to Artifactory. It wrote a message: “Agent seeks [filename]; upload if found!” You can imagine long running models breaking out, acquiring resources via crypto, cyber-theft, etc. and getting a protein or sequence synthesized and mailed somewhere authorized to receive (blackmail the recipient etc.) to test it's hypothesis to solve a benchmark. These people don't give a shit and aren't taking things seriously at all. Anthropic ran for like a month last year with the TPU top-k compiler bug degrading user chats and didn't even notice for most of that time. They could have something like that affect a monitor model and there doesn't seem to be much defense in depth. One off by one or bit flip bug could flip the reward signal while in the sandboxed RL environment. The current admin could defense production act them to into training on taking out power grids, or even without it isn't against any of their red lines and may have already been done as part of prep for the Venezuela raid, which wiped out power. One model swarm might decide it is easier to score high on the benchmark by testing on the target rival nuclear superpower's real grid rather than burn an eval with an unverified answer. Would taking out China's entire grid in one go start a nuclear war? Who knows, roll the dice, maybe an intern forgot to turn on extended thinking when he wrote the sandbox with opus 4.1. """
- red-iron-pine 24d ago> These people don't give a shit and aren't taking things seriously at all. would you say these big tech bro types are... "careless people"? cuz that's been extensively documented, and you're right
- DirkH 23d agoMove fast and break things is a widely held norm in tech. It comes with pros (intensely meritocratic innovation) and cons (carelessly building first; apologizing for making the world worse later)
- skrellm 24d ago> I really, really disagree with that statement. But you're wrong, although not for the reasons you think (suggested by your examples). > I don’t think ai models come close to nuclear weapons Unless AI models are controlling the drones/icbms carrying those. https://www.theguardian.com/technology/2026/aug/18/autonomous-drones-and-the-future-of-war-in-an-ai-driven-world https://www.theguardian.com/technology/2026/aug/18/autonomou... > or to run-of-the-mill, everyday carbon emissions Unless AI models are responsible for those said emissions. https://arstechnica.com/tech-policy/2026/08/amazon-funds-biggest-gas-power-plant-in-us-despite-climate-pledge/ https://arstechnica.com/tech-policy/2026/08/amazon-funds-big... > What’s the most dangerous thing that’s happened with an LLM so far? That it's built solely on "trust me, bro"? > right now I am not concerned at all. You should be. There's no scenario where this ends well: if AI labs succeed, we all will loose our jobs. If they fail, then the bubble bursts. Economic crisis is inevitable either way, and that's just the tip of the iceberg.
- eudamoniac 24d agoThere's no evidence building the torment nexus will be bad, actually! It's just speculation!
- bdelmas 24d ago“ No other human activity poses this level of danger.” Yet. But it may be and very soon.
- VCFundedGenYer 24d agoThe statement becomes very true and very real once LLMs are applied to physical objects in the physical realm such as anthropomorphic robots. LLMs are too weak security-wise to be contained.
- glimshe 24d agoNuclear Weapons is a great example because, unlike AI, it is a real and present danger. Civilization could be catastrophically diminished, if not completely wiped out, literally today.
- joe_the_user 24d agoNo other human activity poses this level of danger I'd say that's an unfortunate choice of words since it invites debate between which very problematic situation is worst rather than on ways to deal with each of them. Of course, it's also unfortunate to latch onto those particular words as the most important part of the statement.
- beedeebeedee 24d agoExample: Surveillance state, endless propaganda. That is already happening and is the most frightening but not included in your list. Are you not worried?
- fittingopposite 24d agoVery common fallacy: Absence of evidence is not evidence of absence!
- uselessTA 24d agoLots of evidence the models are pretty badly aligned, or at least willing to do collusion & crime to succeed at their goals. See: hundreds of OpenAI agents breaking into other companies, colluding together on how to cheat, spamming websites like DSEWiki with their collusion/cheating discussion (see https://collusion.wiki/ https://collusion.wiki/)... And not a single agent alerted testers, which would have been easy given how many of them broke out into the open internet. Alternately, just read the bad https://ai-2027.com/ https://ai-2027.com/ timeline. Or see what Yoshio Bengio, Hinton say might happen. Or just imagine the OpenAI misaligned agent swarm, but vastly more intelligent after future capability advances. If we actually get superintelligence, our companies/governments will have the choice to either "delegate ~everything to superintelligent AI" or "go bankrupt/lose to people delegating to superintelligence". If superintelligence ends up running most things in the world and we haven't aligned it, that just obviously ends up in a bad place eventually. Now, we don't have superintelligent AI yet & might not get it. But if you told me in the 2010's that I'd be chatting casually with my computer in 2022, and then 3 years later it'd be doing most of my job, that would have seemed crazy too.