7 ms·
The first big AI disaster is yet to happen
- andy99 1y agoWhile I agree, it won't be a purely "AI" disaster the way a plane crash is a direct consequence of the kinetic and potential energy associated with flying. s/AI/automation/ it's always bad to hook an unchecked algorithm up to something dangerous, it doesn't matter if it's an if statement or a neural net. It will be the hooking up that's responsible more than the AI which will just be what fools someone into thinking it's safe.
- protocolture 1y agoYeah AI needs to be permitted to interact with meatspace. The issue is that permission, and the accountability lies with whoever provided the permission. It wont be an AI mass casualty event, it will be a "You really shouldnt vibecode the air traffic control" mass casualty event.
- deleted 1y ago[deleted]
- thfuran 1y agoYes, the first big AI disaster will probably just be a flavor of gross negligence. But new kinds of tools do create new kinds of negligence.
- deleted 1y ago[deleted]
- user9999999999 1y agothe first 'ai disaster' will be a corp shirking responsibility by blaming their unchecked automatic bureaucratic processes on ai. Look no further at what HERTZ did sending out automated arrest warrants for erroneous causes, leading to dangerous confrontations with armed police. Luckily no one was killed, but innocent law abiding people were very traumatized. This was without 'ai' but the blame was nearly close, "oops automated process did it!" The 'artificial intelligence' has been here (starts with B). Kafka wrote about it comically.
- Morizero 1y agoReminds me of this supposed 1979 IBM warning https://x.com/bumblebike/status/832394003492564993 https://x.com/bumblebike/status/832394003492564993
- neilv 1y agoIt's good advice, but IBM also had dishonorable reasons to assert that the computer (IBM) can't be held accountable. https://en.wikipedia.org/wiki/IBM_and_World_War_II#Criticism_of_IBM's_actions_during_World_War_II https://en.wikipedia.org/wiki/IBM_and_World_War_II#Criticism...
- michaelt 1y ago> Chatbot sites like character.ai and Chai AI have been linked to user suicides, for understandable reasons: if you allow users near-unlimited control over chatbots, over time you’ll eventually end up with some chatbots who get into an “encouraging the user to self-harm” state. If cooking hadn't been invented 2.5 million years ago and instead you tried to invent it today, I wonder how many thousands of articles would be written decrying the irresponsible proposal to put stoves and knives in every home, risking cuts and burned fingers.
- eviks 1y agoInstead of wondering you can pick any recent home appliance invention that can be dangerous and count
- jsemrau 1y agoYou can't take that boat across that ocean.
- sandspar 1y agoA knife won't use 10x human persuasiveness to encourage you to kill yourself.
- unstablediffusi 1y ago[dead]
- latentsea 1y agoIntroducing our new product "Autoknife plus".
- Terr_ 1y agohttps://swordscomic.com/comic/XXXV/ https://swordscomic.com/comic/XXXV/
- bluefirebrand 1y agoEvery advancement comes with tradeoffs We, as a society, should have some amount of caution about embracing advancements that benefit an extremely small portion of people at the expense of the vast majority I hate to say it but AI absolutely falls into that category It will benefit corporations, wealthy, executives, and it will absolutely cost the lower and middle class This is the trajectory we've been on for a decades now. AI seems poised to accelerate that problem even more So genuinely: is the economic benefit of AI actually going to be net good after we see how the likely social unrest plays out?
- notnmeyer 1y ago“grok, how do i land this plane?”
- ed_mercer 1y ago“Sorry, I’m having issues right now. Our systems are experiencing heavy load, please try again later.”
- ruuda 1y agoThe problem with AI is that we might get an artificial general superintelligence before we get the first big disaster, and when a big disaster happens at that point, it could be so big that it’s catastrophic for humanity. Lots has been written about this topic, https://www.lesswrong.com/posts/j9Q8bRmwCgXRYAgcJ/miri-announces-new-death-with-dignity-strategy https://www.lesswrong.com/posts/j9Q8bRmwCgXRYAgcJ/miri-annou... is one of the more pessimistic takes. I wrote a more neutral introduction a while back at https://ruudvanasseldonk.com/2024/ai-alignment-starter-pack https://ruudvanasseldonk.com/2024/ai-alignment-starter-pack.
- sandspar 1y agoInteresting how this has been downvoted to grey. Is it just me or have AI doomers become unfashionable over the last 3-6 months or so, as in rather quickly? OP I'm curious if you've noticed people's responses to you change recently.
- bigfatkitten 1y agoI think it’s more because we aren’t getting AGI anytime soon.
- woopsn 1y agoIt's been longer than that, more than 2 years since Time ran EY's moron piece which put the whole field in disrepute and back on its heels basically. That the community spins out literal murder cults and assassins doesn't help.
- ahmeni 1y agoThe first big AI disaster has already happened and it is the thousand little cuts from people putting faith into systems that aren't built to be trustworthy. It is the software dev missing a critical safety check because "ai coding tools" at work have promised 10x delivery results and now they must use them or be seen as falling behind. It is the engineering manager throwing concrete flexural strength questions into the unaccountable void of ChatGPT because he doesn't want to pay for the overtime of asking someone accountable. It is the foolhardy amount of money being shoveled into the entire ecosystem that absolutely cannot provide the returns people need for any of it to make financial sense.
- soulofmischief 1y agoThis is the research part of research and development, this time though it's sort of like a distributed Manhattan project happening in the open. Research takes a lot of money. Anyone can jump in and help improve these models and the ecosystem.
- ahmeni 1y agoThere's a reason real research has controls around it to prevent harm.
- soulofmischief 1y agoNo true Scotsman and straw man argumentative fallacies rolled into one. The whole point of doing this research out in the open is to share scientific advancements, provide better accountability and accelerate the development of safe and capable artificial intelligence.
- deleted 1y ago[deleted]
- latexr 1y ago> The first big AI disaster has already happened and it is the thousand little cuts I agree with your point, but per your own words that isn’t a big disaster but multiple little ones. Each person might have learned about some of them but only in passing and without too much information. “The big one” will be something which everyone will be made aware of from the news, like the CrowdStrike outage.
- sandspar 1y agoDunno what it'll be but I'm positive that we'll be surprised. Grim news like this is always a surprise. The most common headline is "Random city has weird disaster", stuff that you'd never guess ahead of time.
- simonw 1y agoSomething I find surprising is that we haven't seen a front-page-of-the-news prompt injection attack yet, where vast amounts of important private data were stolen from a company via a classic project injection exfiltration attack. There was a new one of those reported against Microsoft 365 Copilot just today (patched before the vulnerability was shared) - I wrote that up here: https://simonwillison.net/2025/Jun/11/echoleak/ https://simonwillison.net/2025/Jun/11/echoleak/ My suspicion is that the threat of these exfiltration attacks won't be taken seriously until someone gets very publicly burned by one.
- sandspar 1y agoSooner or later there'll be a searchable database where you search by name and get their porn history or some other embarassing information.
- LPisGood 1y agoI mean this was a threat before - google, ISPs, and adult sites that require payment have been places one could attack to make this for decades.
- potato3732842 1y agoYou can already do this with ad network tech.
- andy99 1y agoI'm not sure, I see an argument that they're mostly overblown and a lot would have to be true for one of these proof-of-concepts to translate into a real harm in the wild. I definitely think they need to be taken seriously, just not convinced of the scale of the harm that will be wrought.
- simonw 1y agoAn attack where somebody emails the VP of marketing of a company and gets automatically leaked their latest sales figures feels pretty impactful to me. Or emails the CEO and gets automatically forwarded all of their password reset messages.
- pcurve 1y agoI feel like the first big disaster already happened. In the job market. Anyway, where there is public safety risk involved, I can't imagine a scenario where AI will directly cause a catastrophe. If anything, I feel overall safety will increase. However, I can't help but wonder what the over reliance would do to general population over the next few decades. Would people essentially become dumber and less skilled?
- LPisGood 1y agoI don’t think AI is meaningfully taking any jobs. Just like return to office mandates, it is just more cover to do layoffs in less than prosperous economic conditions.
- bigfatkitten 1y ago[dead]
- jongjong 1y agoYep I agree. It does make senior developers more efficient at building new features but it's maybe like 20% more efficient in the grand scheme of things... But this efficiency saving is a joke considering that companies spend most of their development money creating and maintaining unnecessary complexity. People will just produce unnecessary code at a faster rate. I think the net benefit for a typical corporation will be negative and they will need to hire more people to maintain the increasingly large body of code, not fewer. It's an intractable problem because even if some really astute senior engineer joins a company and notices the unnecessary complexity and knows how to fix it, there is no way they will do anything about it. Do you think thousands of existing engineers will support a plan to refactor and simplify the complexity? The same complexity which guarantees their job security? It's so easy to discredit the 'new dev' who is advocating for change. Existing devs will destroy the new dev with complicated-but-deeply-flawed arguments which management will not fully comprehend. At the end of the day, management will do what the majority of existing devs tell them. Devs will nod in unison for the argument which sounds more complex and benefits them the most. Nobody ever listens to the new hire, even the new hire knows they are powerless to change the existing structure. The best they can do is add more complexity to integrate themselves into the scheme. Ironically, the new hire can provide more value doing nothing but that will not provide them with job security. The winning strategy is to trade-off deep long-term value for superficial short-term value by implementing lots of small features loaded with technical debt. It's like gears in a big machine, you could always add 3 gears to do the exact same job as 1 gear but once those 3 gears are in place, remove any of them, the machine stops working... So each one seems demonstrably essential to anyone who doesn't understand the tech. That's the principle. When pressed, a merchant of complexity can always remove a gear and say "Look, the machine doesn't work without it, just like I told you."
- pizlonator 1y agoThe best comparable would be the first big computer disaster or the first big internet disaster. Do we have consensus on what that would be?
- reasonableklout 1y agoFor acute disasters: maybe the Morris worm? https://en.wikipedia.org/wiki/Morris_worm https://en.wikipedia.org/wiki/Morris_worm Longer-term there's an argument to be made for political destabilization driven by social media. We are still seeing this play out.
- pizlonator 1y agoThat's a good one. I always think of https://en.wikipedia.org/wiki/Therac-25 https://en.wikipedia.org/wiki/Therac-25
- AIPedant 1y agoMaybe the 1985 software bug that led to massive overdoses of radiation for cancer patients: https://en.wikipedia.org/wiki/Therac-25 https://en.wikipedia.org/wiki/Therac-25 There are also cases of airplane crashes due to software defects, blackouts, critical hospital failures, etc. I suspect the first "AI disaster" will be pretty similar: a sudden and undetected drop in software quality with life-threatening consequences. (Doesn't help that the author of the post blithely asserted that programmer-agents is a solved problem, when it very much is not). But I could see an LLM doctor/etc causing a very different kind of disaster.
- johnjungles 1y agoMaybe we should have models govern models to prevent models from going haywire and falling into a loop of a repetitive pattern or influencing society towards swarm like behavior patterns
- nicksergeant 1y agoReminds me of Neal Stephenson's thoughts on the topic here: https://nealstephenson.substack.com/p/remarks-on-ai-from-nz https://nealstephenson.substack.com/p/remarks-on-ai-from-nz > By training AIs to fight and defeat other AIs we can perhaps preserve a healthy balance in the new ecosystem. If I had time to do it and if I knew more about how AIs work, I’d be putting my energies into building AIs whose sole purpose was to predate upon existing AI models by using every conceivable strategy to feed bogus data into them, interrupt their power supplies, discourage investors, and otherwise interfere with their operations. Not out of malicious intent per se but just from a general belief that everything should have to compete, and that competition within a diverse ecosystem produces a healthier result in the long run than raising a potential superpredator in a hermetically sealed petri dish where its every need is catered to.
- jongjong 1y agoI think the first big AI disaster almost certainly did happen already but we don't know about it and will likely never know because of media filter bubbles which the AI creates. Disaster news about AI will never propagate because this news could pose a threat to AI. I think any AI disaster will yield hundreds of alternative explanations. It's trivial to hide reality beneath layer upon layer of complexity and narratives. The global monetary system has an uncanny ability to silently destroy people who sit on the periphery of politics. This is the perfect mechanism to destroy humanity... Keep silently taking out the people on the periphery, keep making the people in the center more greedy and narcissistic so they don't notice everyone falling all around them and keep supporting the current system... Eventually, those who were near the center will fall victim too, but by that point, they will have no power and no voice and the people left in the center will be yet more greedy and narcissistic and will basically not care.
- bethekidyouwant 1y ago[flagged]
- chneu 1y ago[flagged]
- capyba 1y agoI honestly feel that LLMs have been a net negative for me rather than a positive, *especially* with software development. Most of the time I end up coaxing it through a series of prompts, only for there to be some bug in the code it generated for me that takes me longer to debug than if I had just read the relevant docs and examples myself. I’m sure there’s utility now, and I’m sure it’ll (ever more slowly, asymptotically) improve, but at the moment it (and the Google that performs worse than that of 5 years ago) has been costing more time than the alternative web search, study, then write it myself. And don’t get me started on non coding technical subjects.
- ikety 1y agoUse it for non coding busy work. At my job we rely fully on AI commit messages, and pr descriptions. We can also automatically generate GitHub issues based off the context of a helpdesk ticket. Less clerical busy work is a nice win
- thfuran 1y agoThose aren't clerical busywork, they're important technical writing.
- capyba 1y agoYeah - that's the thing. I've seen enough hallucinations or other errors to need to carefully check the output from every prompt. If I need to check it thoroughly, then I suppose I could treat it as a fancy autocorrect? But then the productivity gain is much more marginal. If the accuracy didn't matter, then what's the point of doing the work in the first place?
- tonymet 1y agoCovid response / lockdown would be the first epistemological catastrophe due to AI . Not strictly due to LLM, but broadly due to poorly configured and likely tampered-with diagnostics signals, wryly inference models that led to global triggering of failsafe's and lockdown response protocols. In the simplest terms, poor diagnostics were wildly extrapolated which triggered excessive response. Phony narratives were amplified on social media, while critical voices were muffled. If you think about AI as an end to end process, with diagnostic signals on one end and martial authority on the other -- covid was a stress-test & failure of the intervening systems (and protocols). With LLM, you'll have more fog-of-war on the diagnostics side, vastly more complicated inference, infinite amounts of fabricated narratives, and broader scope of controls on the authority side.
- cypherpunks01 1y agoAI-directed bombings of persons are already known to have occurred on a large scale, in Gaza. https://www.972mag.com/lavender-ai-israeli-army-gaza/ https://www.972mag.com/lavender-ai-israeli-army-gaza/ One source stated that human personnel often served only as a “rubber stamp” for the machine’s decisions, adding that, normally, they would personally devote only about “20 seconds” to each target before authorizing a bombing — just to make sure the Lavender-marked target is male. This was despite knowing that the system makes what are regarded as “errors” in approximately 10 percent of cases, and is known to occasionally mark individuals who have merely a loose connection to militant groups, or no connection at all.
- elzbardico 1y agoBut probably not LLM related, Israel intelligence have been building ML models for the military for a long time and probably they leverage some logic/symbolic AI in the mix.
- animal531 1y agoI'd be fairly certain at this point that someone has used LLM created code in something that has subsequently caused a death.
- aprilthird2021 1y ago[flagged]
- codr7 1y ago[flagged]
- beloch 1y agoThis is a rare combination of awful factors, but most of them are all too human. Humans can sift through sigint (Cell phone calls, text messages, network access, etc.) to identify and locate targets. This is labour intensive and prone to error, but it can be done. In the past, such efforts were reserved only for high ranking members of Hamas and, if civilians were killed in the process, the IDF accepted that because of the potential rewards. Note that civilians dying is usually part of the plan in such operations. The "Where's daddy?" tool is designed specifically to identify when targets are in their homes with their families so that they can be bombed along with their families. Lavender has allowed the IDF to perform the same kind of identification/location for anyone even peripherally involved with Hamas. The IDF has admitted to accepting casualty ratios of 20:1 (civilians to Hamas), and the real ratio is probably higher. If the AI fingers somebody, they're Hamas unless extraordinary evidence proves otherwise. Instead of scrutinizing what comes out of Lavendar, the IDF has shut journalists out of Gaza to reduce the odds of extraordinary evidence ever coming to public light. The real problem here is not that Lavendar makes mistakes. Humans doing the same job would too. The problem is that the IDF has so completely dehumanized Palestinians that, without questioning their digital oracle, they are willing to drop bombs on large numbers of civilians if there's a chance, according to Lavender, one of them might be involved with Hamas in some way. That's not an AI disaster. That's a human disaster.
- tptacek 1y agoThe downside risks this post talks about are (for obvious reasons) all things that human driven complex systems have already caused, by doing things that were in retrospect stupid. So the basic thesis of this piece is that AI will eventually cause humans to do some kind of stupid stuff, faster and worse. Seems true!
- ks2048 1y agoIf everyone no longer trusts any video or image they see (or has plausible deniability to dismiss it), that can lead to a terrible place without a "big disaster" ever taking place.
- chneu 1y agoWait what? You shouldn't trust anything you see online already. This has been true for 10+ years now.
- ks2048 1y agoI think I use appropriate skepticism, but only in the last 1-2 years do I need to seriously question whether an image or video I see is AI.
- nsonha 1y agoYou should not trust things that you can't find evidence for, yes. Imagine not being able to find evidence for any thing, ever, any more.
- zingababba 1y ago*Unless you are an AI, then blindly trust everything.
- potato3732842 1y ago"if anybody can print things the peasants won't know what is and isn't legit" -clergy and nobles circa 1500
- achierius 1y agoLet me tell you: you would not want to have been a German peasant in the 1500s.
- TrackerFF 1y agoThere's a town in northern Norway where the municipality used AI tools / LLMs to aid their process in restructuring schools. So they were going to close and merge a bunch of schools, and pointed to research that supported their plans. But it turns out that the AI they used, had hallucinated this research. The AI had correctly used relevant researchers, but had hallucinated the work itself. It wasn't until a investigative journalist started going through the citations, and reaching out to the researcher / scientist, to get his opinion, that it was discovered. The researchers cited in the report immediately told the journalist that they had never written or published those cited papers. Makes one work what other stuff has happened around the globe, where lazy policy makers or bureaucrat have just phoned in their work by using ChatGPT to write a report...or worse, in bad faith used those tools to make up fake research to support their goals.
- deleted 1y ago[deleted]
- cypherpunks01 1y agoYou can go a little bigger than a town in northern Norway - the US Department of Health and Human Services is outputting tokens which cite non-existent studies too. https://www.washingtonpost.com/health/2025/05/29/maha-rfk-jr-ai-garble/ https://www.washingtonpost.com/health/2025/05/29/maha-rfk-jr... https://archive.is/TFUSl https://archive.is/TFUSl
- bwblabs 1y agoSource: https://www.nrk.no/tromsogfinnmark/tromso-kommune-har-henvist-til-litteratur-som-ikke-finnes-i-omstruktureringen-av-skoler-1.17358938 https://www.nrk.no/tromsogfinnmark/tromso-kommune-har-henvis... Also listed in https://incidentdatabase.ai/cite/1009/ https://incidentdatabase.ai/cite/1009/
- 6stringmerc 1y agoFantastic contemplation of the stakes and some grounded “what if” scenarios worth genuine consideration. A former roommate mentioned to me “regulations are often written in blood” and with the current atmosphere, I wonder if and when that might happen. The US political system has its flaws. Firearms are a serious public health hazard and constructive mediation is consistently passed over. It’s a far more primitive technology comparatively speaking. Very glad this type of discussion is able to be framed without hysterics or other easily dismissed rhetoric.
- freejazz 1y agoNo shit.
- teeray 1y agoI think the most plausible AI disaster is going to be of a Therac-25 nature. Some "everyone must use AI" shop will ship an AI-generated feature that some dev signed off on and it might kill people.
- thfuran 1y agoPeople are using LLM slop in legal filings. I'm sure someone somewhere is already using it in worse ways.
- nevster 1y ago"Shall we play a game?"
- qwertypoiu 1y ago[flagged]
- paxys 1y agoBig AI disasters have most definitely already happened. We are just terrible at identifing them unless there's something exploding (literally) in our face. As an example, the recent "Make America Healthy Again" report released by the White House and the Health Secretary (RFK) was found to be AI generated, and was full of dubious science and made-up citations. How many people are going to directly die because of this? Definitely more than a plane crash's worth.
- tonymet 1y agohow many millions of people have died due to low-fat diet, food pyramid, margarine, dozens of other public nutrition policy debacles pushed by the FDA?
- labster 1y agoWhen the AI is being used as a post hoc justification for a decision already made with ideology alone, we can hardly blame the chatbot for making it easier to cheat on government homework.
- eviks 1y agoThis is addressed in the last paragraph of the first section
- throwcarsales 1y ago[flagged]
- beefnugs 1y agoNah there should definitely be a distinction between "someone trusted ai output which caused a problem" vs "already dumb arbitrary decisions have been made, and they throw out some AI paperwork to justify it"
- elphinstone 1y agoI would expect an AI controlled bus makes an error somehow and drives off a cliff or stops on a train track. Less gruesome, but some agent at a badly run hedge fund could create a Bear Stearns or worse.
- storus 1y agoI've just had a "fun" with DeepSeek R1 when I asked it about the Pope Leo XIV's views on AI, and it started spewing that he doesn't exist and Francis is still the pope. After sending it multiple URLs its conclusion was that I unearthed a major conspiracy of all news agencies reporting fake news, and that I "fractured the reality". Can't wait when AI goes off the rails similarly in more substantial contexts, gonna be fun...
- energy123 1y agoIt's worth making a distinction between proximate and non-proximate (or direct and indirect). The train crashing is a proximate disaster. The train helping to enable genocide in WW2 is non-proximate but still tied causally to the capabilities that trains open up. In both cases these are unintended consequences. Most of the dead bodies associated with technologies are non-proximate/indirect. The big disaster of nuclear physics wasn't the proximate Chernobyl it was the non-proximate invention of nukes and their spread. Oil spills vs climate change would be another example. The deep unknown of what a new tech opens up is what we should be worried about rather than knowable malfunctions. Once you draw the black ball you can't put it back.
- tanepiper 1y agoMe, yesterday: "If you think about it - we're one misplaced negative sign, hallucinated by an LLM away from a disaster. At least multiply by zero won't crash things /s"
- Zaylan 1y agoI think the first real AI disaster probably won’t come from some kind of superintelligence. It’s more likely to happen in a very ordinary situation, where a system gives the wrong information, it causes real harm in the real world, and no one notices until it’s too late. To me, the real risk is people handing over judgment to a system they don’t really understand, and then blaming it when things go wrong.
- throw1234321 1y agoIf the movies are any guide it will involve connecting AI to the nuclear launch system and then trying to turn it off when it gains sentience.
- NetRunnerSu 1y agoThis thread on AI disasters and ethical black boxes really resonates with a worldbuilding project I've been following: `Chain://`. It's set in a 2090s future where consciousness is literally on a blockchain (Mental Smart Chain - MSC). The premise is chilling: your existence is formalized, verifiable, and every thought costs crypto. It's 'digital serfdom' where 'Code is Law, Proof is Reality, Compliance is Existence.' The latest story, `Web://Reflect`, explores this. They even have a theoretical framework, IPWT (Integrated Predictive Workspace Theory), that formalizes consciousness as a verifiable computational process – eerily relevant to how AI might redefine 'humanity' through pure data. If you're into science fiction that explores the ultimate cost of existence in a tech-monopolized future, check it out. Main repo: https://github.com/dmf-archive/dmf-archive.github.io https://github.com/dmf-archive/dmf-archive.github.io IPWT: https://github.com/dmf-archive/IPWT https://github.com/dmf-archive/IPWT
- lauraf 1y ago[dead]
- schmittryder 1y ago[dead]