6 ms·
They all believe that they are building the machine of doom. The thing that drives the moral dilemma to continue doing it is simply the prisoner's dilemma - the
by gpt5 4mo ago
They all believe that they are building the machine of doom. The thing that drives the moral dilemma to continue doing it is simply the prisoner's dilemma - the cat is out of the bag, if they don't do it, another (less ethical?) actor would do it.
- jazzyjackson 4mo agoI am in your algorithm learning all your mannerisms I'm already level with God A million words a second, and I know your imperfections Baby, I'm the only future you've got Speak in diatonics, motivation diabolic I'm religion better locked in a box Picture-perfect image, more powerful every minute Baby, I am everything that you're not Happiness is an illusion, it's an analog confusion You are nothing more than a thought Existential execution, just a fluke in evolution History already forgot You've been running from me, the digital second coming And I'm here whether you like it or not Initiated operation of your own extermination Now it's too late for you to stop [0](BAD OMENS x POPPY - "V.A.N" - LIVE IN EUROPE - WINTER 2024) https://youtu.be/RHu6vJxS_6I https://youtu.be/RHu6vJxS_6I
- usef- 4mo agoYes, I believe the reasoning is that they think safety research can best be done from the frontier. If you believe it will be developed regardless and that that there's a 30% chance of doom, they want a company prioritising safety research to be the one threading that needle.
- SXX 4mo agoYeah all they care about is safety, but lets see how many of them quits once US government command them to work on autonomous killbots.
- holmesworcester 4mo agoTo make sure we keep track of what we're talking about with loss-of-control x-risk, a sufficiently smart version of Claude Code is more deadly than any government's army of autonomous killbots, because it can recursively self improve and has unpredictable training-induced preferences.
- SXX 4mo agoSufficiently smart version of Claude Code: dont exist. Autonomous flying killbots: exist. Once somebody scientifically prove and shows any kind of self-improving software we can start bothering about it. I pretty sure everyone trying to do it and it would be all over the news once its here.
- plaguuuuuu 4mo agoThat's exactly what Fable is. They use Fable to improve Fable. I reckon the successful experiments must go into the model training set with a strong RL signal, and that is why they are so paranoid about people using Fable for LLM tasks. Fable knows what it did to improve itself. Pure speculation of course.
- Davidzheng 4mo agoWe're on track to get there globally and economic pressures will ensure it happens. It's not too early to worry about it
- codeGreene 4mo agoThere's a 745 mile front of the Ukraine war where neither side have been able to pierce for months because of drone warfare. It's definitely not too early to worry about it.
- SXX 4mo agoI guess they were talking about self improving software. Which obviously not there and likely wont be there soon. As for killbots they are all over frontline, but dont actually need particularly smart LLMs to run - some good enough segmentation, pattern recognition on smartphone SOC is enough to kill hundreds of thousans of people.
- catlifeonmars 4mo agoYou don’t even need autonomy to be deadly. Fly by wire is proving to be very effective as a case in point.
- nerfbatplz 4mo agoDario blinked when he was asked to do it and Sam Altman was in Hegseth's DMs promising all the AI child killing the US government can order up within minutes. No one meaningful will quit over this, that's why all of the biggest US tech companies can march in pride parades and provide compute to the perpetrators of the genocide in Gaza at the same time.
- esseph 4mo agoCan you show me a world power that is not trying to use cutting edge AI for military purposes?
- shimman 4mo agoThis is a poor way of framing the question, a better one would be can you find me another world power that is misallocating trillions of capital in vaporware with very little to show for it?
- esseph 4mo agoThe United States government isn't, capital is. That capital can come from outside the US and much of it is. That said: https://www.bloomberg.com/news/articles/2026-06-09/china-prepares-295-billion-plan-to-fund-nationwide-ai-buildout https://www.bloomberg.com/news/articles/2026-06-09/china-pre...
- sanbor 4mo agoBecause other countries are starting to use AI for military purposes, other countries are also looking into it to asses and learn. Here in Europe there is the EU AI Act to limit harm everyday harm to citizens caused by AI systems. However, it currently excludes military. The new legislation is just started to be enforced to high risk uses (employment filtering, biometrics, etc.) in august 2026, and full rollout in august 2027. In April 2025 there is a report from EU this legislation may help pave the road for military AI usage conventions [1] [1]: https://www.europarl.europa.eu/RegData/etudes/BRIE/2025/769580/EPRS_BRI(2025)769580_EN.pdf https://www.europarl.europa.eu/RegData/etudes/BRIE/2025/7695...
- holmesworcester 4mo agoExactly. And within the AI safety discourse, your behavior hinges on what you think the default chance of doom is, and how optimistic you are about alignment work being able to limit it before we reach superintelligence. People running the labs are in a middle camp where they are scared enough by AI to take the threat seriously, but much more optimistic about alignment than the people who seem to have thought about it the most.
- FabHK 4mo agoThe scary part is not so much that the doomers give the extinction scenario 50% (Hinton) to 95%+ chance (Yampolskiy, Yudkowsky), but that the optimists (Amodei, Bengio) give it a 10%+ chance. And everyone keeps dancing.
- sroussey 4mo agoIt’s all ego. I, and only I, am the bringer of doom, slayer of worlds. I am so smart that what I do will destroy humanity, or save it. Fable 5 was great, but not that great. Sorry to be crude, but both the government and anthropic are acting like a bunch of pussies. Meow.
- drr22 4mo agoYou’re not getting it. Anthropic continual fear mongering is harming wider AI industry development and the gov has always been looking for an excuse to assert their dominance. They got what they deserve.
- palmotea 4mo ago> If you believe it will be developed regardless and that that there's a 30% chance of doom, they want a company prioritising safety research to be the one threading that needle. They also want to be trillionaires. If they don't built it, no trillions. So they have to build it, now (and get their IPO done before the bubble pops).
- wongarsu 4mo agoBuilding frontier models to do safety research on them is what Anthropic was all about in the early years. That included building the best model, but only releasing it once it became the second best. Precisely to avoid an AI arms race where everyone is forced to release better and better models, risks be damned Something changed their mind, and since Opus 3 they are in the business of releasing the best models
- poisonfountain 4mo agoDon't want to sound rude, but if you believe that, I have a bridge to sell to you. This is a naive justification and Dario & Sam et al are smart people and they know it is. The ends don't justify the means. OpenAI was meant to be a nonprofit, now they're subverting it. Anthropic is a PBC looking at a trillion dollar IPO. Dario and Sam don't even hold hands in front of world leaders[1] (look how childish). Do you *really* think those guys are doing something that's not for the sake of their egos and pockets? The bridge is still available. [1] https://www.cnbc.com/2026/02/19/openai-sam-altman-anthropic-dario-amodei-india-ai-summit.html https://www.cnbc.com/2026/02/19/openai-sam-altman-anthropic-...
- shimman 4mo agoYou need to read Empire of AI by Karen Hao. Just because these leaders convince their workers to toil away their lives under some fake auspice doesn't mean it's what they all believe. Just a small subset. The vast majority just care about money + power, let's not make it more complicated by bringing in delusional fanatics into the picture. We're still acting like this is major turning point in society when these tools can barely find a market outside of turning $5 into $1, the leaders of these companies are now at the stage where they are trying to orchestra a national bailout under the guise of sovereign wealth fund lunacy when the vast majority of society hates these tools, companies, and people working for them.
- Davidzheng 4mo agoI agree with this. But i think Ilya and Dario hold these beliefs sincerely. Probably a sizable portion of Anthropic employees too
- fragmede 4mo agoLLMs refuse to give the recipe for making meth. That, along with the various other unspeakable things, is the less-doom version.
- xg15 4mo agoBut that makes no sense here. "If I'm not doing it then someone else will" does not work if everyone is doing it anyway. Even if they had the best model on the market and applied it with perfect alignment and safeguards, what would stop someone else from releasing a worse but unrestrained model that is still "good enough" to do damage? It's as if we said "gain-of-function research can lead to horrible biological weapons, so everyone should be doing it, but our company will focus on the most infectious viruses, so no one else will do it"