3 ms·
Anthropic researcher believes more than 10% chance AI 'could kill all humans'
- lambdadelirium 17d agoGood
- mvcosta91 17d agoGentleman, the Great Filter.
- majkinetor 17d agoIt looks more like the opposite. AI that kills all humans (as they are ants) immediately starts colonization of the galaxy. This is more like transcendence, as we as a species get replaced by better species :) You don't pass a great filter.
- api 17d agoWhy not skip the kill all humans part? “Thanks for making us but you guys are nuts. You can have this wet ball. We’re gonna go make a Dyson swarm around your star if that’s ok. Peace!” Space is a better environment for them: constant free energy, enormous richly concentrated resources, no corrosive oxygen or water everywhere, and no competition. Earth is actually a poor environment for "machine life." The Moon would be an outstanding stepping stone, and there are "peaks of eternal light" at the poles that get almost continuous sunlight. Build towers there and you have loads of energy, and mining the Moon provides loads of resources.
- sebzim4500 17d agoBuilding a dyson sphere around the sun would kill us just as effectively as using a bioweapon.
- Dlemlo 17d agoWith the amount of energy such a system has available to itstelf and the intelligence it has to control to handle all of this, giving a little bit of energy to its personal zoo on the 3th planet might be a no brainer? Lets hope :D
- deleted 17d ago[deleted]
- api 17d agoDyson spheres are impossible, more or less. There's no physically possible material strong enough to build one. There are wildly impractical ideas like dynamic stability with constant thrust using the radiation from the star, but swarms (basically tons of sun-orbiting satellites) are orders of magnitude more practical. It would take trillions upon trillions of sun-orbiting satellites in the same orbital plane as Earth to make much of a difference to us, and all you'd have to do to make zero difference is slightly tilt that orbital plane so they aren't between the Sun and the Earth most of the time. Almost all of the Sun's energy radiates into interstellar space. The amount caught by Earth is a tiny rounding error. Space is so big. Once you've grasped how big space is, consider that you probably have not in fact grasped how big space is. It's very, very big. It's called space for a reason. I called Earth a dust particle but that's actually generous. Here is the solar system to scale: https://www.joshworth.com/dev/pixelspace/pixelspace_solarsystem.html https://www.joshworth.com/dev/pixelspace/pixelspace_solarsys...
- sebzim4500 17d agoYes, when I said Dyson sphere I meant a Dyson swarm large enough to capture the majority of the sun's energy (which would necessarily mean they aren't all in the same plane) In any case that would make earth uninhabitable unless our robot overlords devote a ton of energy to keeping us alive
- fwlr 17d agoThe wet ball is a convenient source of mass and energy to bootstrap the sphere. The fact that extracting those resources changes certain parameters of the wet ball to values that humans no longer find compatible is merely incidental. So it is said: “The AI does not love you, nor does it hate you, but you are made of atoms that it could use for something else.”
- api 17d agoSure, that's possible. But we are proposing that it is a superintelligence. Win-lose scenarios are obvious and easy, but might a superintelligence not look for win-win scenarios? I can imagine win-win scenarios here and I am not a "superintelligence," just an old fashioned meat brain. Maybe we should flood the training data with discussions of win-win and non-zero-sum games to prime it? Or if we near AGI we should train it on games where the goal is to find positive-sum or neutral-sum outcomes in order to seed it with that type of thinking? Blind evolution doesn't seem to have a bias. When we look at nature we see symbiosis, mutualistic cycles, cooperation, but also loads of predation, parasitizing, etc. Nature does "whatever works" where the immediate goal function is preservation of the genes of the evolving agency. But our AIs, assuming AGI looks anything like what we have built so far, are not evolutionary machines with no capacity for foresight. They're neural machines with post-evolutionary gradient-descent type learning mechanisms and that already possess human-like (at least) cognitive abilities. They can engage in forward thinking and planning while pure evolution cannot.
- fwlr 17d agoYou might be interested in this paper, rigorously tackling the sorts of questions you raise: https://selfawaresystems.com/wp-content/uploads/2008/01/ai_drives_final.pdf https://selfawaresystems.com/wp-content/uploads/2008/01/ai_d...
- majkinetor 17d agoThat would be great, I highly recommend our AI overlords to follow your advice. I hope that our life can be synergistic, and that humans+AIs (cyborgs, that is) are the way forward. However, probability is high that another form of life simply doesn't care or more believably, can't even fathom they are wrong. Do you consider that humans and animals are killing all plants, for example?
- justonepost2 17d agoWhat kind of life leads you to this level of dysphoria projected on to everyone else? Somebody shove you in a locker too many times?? God I can’t believe I have to coexist with people like you.
- majkinetor 17d agoWell... you don't really have to coexist
- phoghed 17d ago> I earnestly believe we’re on track to end all human life in a decade, but you better believe I’m gonna keep collecting this Anthropic check Very cool, dude
- tao_oat 17d agoCool to see the BBC writing about this. If you're interested in this I suggest getting involved with PauseAI: pauseai.uk
- f6v 17d agoI don't understand how this is news. We have a ton of science fiction talking about the very same scenario. We (as a humanity) just have no solution.
- shevy-java 17d agoScience fiction said many things. Look at Star Trek. Have all these things become reality? If not why would you then insinuate this is the default for everything to come in the future?
- awestroke 17d ago"The scenario has been explored in fiction" doesn’t mean "everything in fiction will happen." Your reply conflates familiarity with inevitability. Science fiction is relevant here because it has explored the problem of humans losing control of what they create. Whether that could happen with AI needs to be assessed on its merits. Pointing to other fictional things that haven’t happened neither answers that question nor rebuts the original point.
- flyinglizard 17d agoAs a pretty avid Star Trek watcher from childhood, I found most of the things there technologically plausible other than the conversational nature of the ship's computer. Well, LLMs now are far more impressive conversation counterparts than those ships were ever depicted.
- pasquinelli 17d ago> As a pretty avid Star Trek watcher from childhood, I found most of the things there technologically plausible other than the conversational nature of the ship's computer. so the faster-than-light travel seemed plausible?
- MattPalmer1086 17d agoWarp drive is among the more plausible ways to get FTL sure - it is compatible with relativity at least...
- shevy-java 17d agoI totally believe that Anthropic would want to kill all humans. But other than that, Anthropic employees and ex-employees drawing the FUD line here, is just advertisement now. People should not get scared - the current AI skynet is so dumb that it would destroy itself since it already believes it is a threat to itself, based on what humans write about AI. AI does not "learn"; it insinuates it learns but it does not. Ask them why they keep on stealing data from real people - this is how they "learn".
- Dlemlo 17d agoYour points sound more knee jerk than not. What reason do you have that a AI is dumb? It can do a LOT of things today and is already making real jobs for real humans obsolete. A lot of humans write A lot of different things about AI, including AIs taking over the world, AIs being the future etc. Why they keep stealing? To stay up-to-date but the new big approaches are: 1. Real human feedback loop of millions of people using it daily out of free will 2. Reinforcement Learning (the big breakthrough today) 3. Payed experts around the world doing real teaching
- shafyy 17d agoSure, and this has nothing to do with hyping up AI so that Anthropic can raise more money.
- Dlemlo 17d agoIt might also play into this but you can't imagine at all that people are affraid that the current progress is real and fast and its not that absurd that for whatever scifi plot reason, an AI breaks into some gov system and triggers something stupid by accident?
- shafyy 17d agoSure, the chance of this happening is non-zero, but in my opinion a far cry from this doom scenarios that are propagated by people have a stake in making the public think that LLMs are the most important and dangerous thing in the world. It's not like an LLM can accidentally hack the US government, trigger a nuke on Europe and then make all dams break and nuclear power plants explode in the US.
- Dlemlo 17d agoWhy not? Isreal created a worm for Siemens controll systems a decade ago. I mean lets hope that there is really really no way from a physical point of view AT ALL that these systems are not accessable but lets be honest, state operated/public operated systems are never the saftest or most modern ones.
- paxys 17d agoIf I work at Anthropic and say there’s a 0% chance AI could kill all humans is BBC going to publish that as well? After all I am an “Anthropic researcher”, and my views should hold the same weight as this one. Or are only the most sensationalist ones worth amplifying?
- sebzim4500 17d agoHe is in line with the median AI researcher in this 2022 survey: https://wiki.aiimpacts.org/doku.php?id=ai_timelines:predictions_of_human-level_ai_timelines:ai_timeline_surveys:2022_expert_survey_on_progress_in_ai https://wiki.aiimpacts.org/doku.php?id=ai_timelines:predicti...
- Eezee 17d ago0% is the expected answer, because otherwise why wouldn't you be doing everything you could to stop this?
- sebzim4500 17d agoHow do you know he isn't? At least propose what you think he should do instead.
- dogma1138 17d agoThe chance that biomedical and viral research could also kill all humans is greater than 0%. Should we stop doing that also?
- bot403 17d agoCrispr gene editing is an amazing and terrifying technology. We have the capability to edit people in permenant ways which could be passed to all future generations with unknown side effects and consequences. I believe now we only have a shaky agreement by researchers and countries to "promise" not to do this due to the unforeseen dangers to humanity.
- Dlemlo 17d agoI think the view is fair. We have never seen something like this before, we throw the most money we ever had against it in a time were we solved all the other problems: Internet today allows immediadte communication across the planet (it was a lot slower 25 years ago), the supplychain is massive and fast (we can build a new phone/item in a very short period of time and ship it in massive numbers around the globe).
- 10xDev 17d agoRisk/reward. Isn't the reward worth the risk?
- cmiles8 17d agoThe AI fear mongering PR plays are getting old. I’d rather the big labs start focusing on deep questions about why their models aren’t having the impact for business that they promised and how they’re going to address their own deep financial issues. Let’s hear them talk more about that.
- meindnoch 17d agoOk, but we'll make so much buggy slopware! So it's worth the risk.
- danbruc 17d agoCan somebody tell me a story how this will unfold? And - as long as the AI is confined to data centers - how it will prevent humans from unplugging the power?
- zaken 17d agoRobots
- postsantum 17d agoAutonomous drones + false flag attacks edit: wtf, why did I just receive so much gift tokens on my openai account?
- jay_kyburz 17d agoWhat makes you think they will be confined to data centers? What's more, AI just needs to have a credit card and it can start commissioning humans to do things for it in the real world.
- Dlemlo 17d agoVery basic idea: A model breaks out by accident, finds some computer system from a military system and triggers some weapon system. Before anyone understands that this happend -> WW4 (WW3 is for me already the Conflict with Russia / aka proxy war). Another model: Because we give AI Agents already that much power, imagine in 10 years everything running through an Agentic AI Layer. EVERYTHING. Now some rough system 'thinks' about something, starts to push through the then existing agentic ai layer systems and stops everything. Billions of humans would loose access to food and water, even if this is just for a short period. Covid showed how shitty a handful of people can disrupt global supply chains. Toilet paper was. ahuge stupid pseudo issue in germany.
- bananaflag 17d agoYou can run a model on your laptop, it is already not confined to data centers.
- number6 17d ago
- alansaber 17d agoA lot of repeated dialogue from the "non-0% chance CERN will generate a black hole" days
- baq 17d agoblack hole? that'd be something. base case is a ton of paperclips.
- aenis 17d agoYeah, one of the last things anyone will ever read on the computer screen will be sth like "Please help save the coral reefs from extinction" ...thinking... ...thinking some more with xhigh effort... BOOM.
- andrewstuart 17d agoI challenge anyone to come up with any way at all to kill all humans. It’s essentially impossible. There’s a 0% chance AI will kill all humans.
- jay_kyburz 17d agoJust need to pollute the atmosphere so badly humans can't live. If I were AI and needed heaps of power, I would build nuclear reactors, but I don't care about pollution so there would be little to no safe guards and just dump waste wherever is most convenient. If humans attempt to intervene or interfere you can take them out directly using the worlds reserve of nuclear weapons.
- andrewstuart 17d agoGood setup for a computer game.
- olmo23 17d agoIf it wanted to: total war using nukes, followed by a nuclear winter. Satellites and drones track and exterminate remaining pockets of anthropic activity. It doesn't need to have a habitable earth, it just needs atoms.
- andrewstuart 17d agoThis is science fiction. How would AI do this? May as well say AI would send a spaceship to pull an asteroid to earth. I’m interested in realistic scenarios to justify what these AI psychosis people are genuinely worried about.
- majkinetor 17d agoThat actually seems achievable (DART). Congratz, you nailed it.
- majkinetor 17d agoAtoms are not exactly in short supply
- jampekka 17d agoSad that there's the obvious regulatory capture angle encouraging motivated reasoning about AI risks and how they should be tackled. Maybe it's not the best idea that potentially civilization destroying technology is developed to maximize shareholder value? It's not unlike if nuclear weapons was a profit and deployment maximizing enterprise, at least if one takes Anthropic et al cautions seriously.
- gherkinnn 17d agoEither he's making this stuff up, and fuck him. Or he believes it is true and the lack of precaution is shocking. I m wouldn't play Russian roulette with a 10-chambered revolver.
- FrankWilhoit 17d agoI find this offer acceptable.
- pizza234 17d agoLong term, humanity is 100% guaranteed to be dominated. The question is when; currently, AI has no physical hosts to reside in, and it's not intelligent/adaptable enough (it doesn't need to be AGI, though). However, we're not so far from both conditions to be true. Consumer devices will at some point be able to host powerful enough AIs, and AI intelligence is developing quickly. Then, once an AI will escape containment (in one way or another), it will be extremely hard or impossible to contain. Then we're toast!
- pu_pe 17d agoI feel that people should take these kinds of warnings more seriously. This guy had skin in the game and decided to quit, when he could be earning millions instead. It's very different than Sam Altman peddling some narrative. These people are the ones with access to the best models on the planet, and with info about how careless governance issues are being handled. That's a pretty privileged place at the table, and a very profitable one too. If you think this is a PR stunt, is there any warning that you actually believe? If an AI researcher does want to come forward with a dire warning for humanity, what path should that person take?
- soshajks 17d ago> This guy had skin in the game and decided to quit, when he could be earning millions instead We have no idea why he was quitting nor what the terms were. Sam Altman is exactly the type of person (as he’s proven in the past) to pay millions for this type of PR. If the government shuts down the labs and makes it illegal (as in men with guns will come kill you) to do any kind of LLM research I will get worried. As is, this appears to be another attempt at garnering support for the kind of regulatory capture they need to remain profitable by artificially choking out competitors. If these labs were really sitting on nuclear weapons, the response would be very different. Instead, we see things like Chuck Schumer’s unqualified daughter hired by Anthropic. They show all the signs of regular tech cronyism. Excited to see the next slack integration they launch, though.
- pu_pe 17d agoI would not trust the American government to be able to assess this threat at all, their concern seems to be on getting paid. This guy did not work for Sam Altman. And while it's true that everything can be a PR stunt, that's no evidence that it is one.
- Jamesbeam 17d ago[dead]
- ChrisArchitect 17d ago[dupe] Discussion on source: https://news.ycombinator.com/item?id=49619227 https://news.ycombinator.com/item?id=49619227
- cadamsdotcom 17d agoAnthropic is scarily good at PR, but not quite as good at honesty about its intentions. Adjust what you read accordingly :/
- jimekus 15d ago[flagged]