7 ms·
Notes on existential risk from artificial superintelligence
- pstuart 3y agoSeems like the easier question would be: "why would we assume such a being would not be a risk?"
- labrador 3y agoBard: The author argues that ASI is likely to be developed soon and that it could have devastating consequences such as 1 The development of new weapons that could easily destroy humanity. 2 The accidental release of harmful substances or organisms. 3 The displacement of humans from the workforce, leading to widespread poverty and unrest. 4 The loss of control of AGI, which could lead to them making harmful decisions 1, 2, and 3 were concerns before AI, but we've managed so far. 4 seems to be the one that is most terrifying, but then we shouldn't be putting AI in control of nuclear weapon launches or other potentially harmful activities
- deleted 3y ago[deleted]
- jjoonathan 3y agoEnriching uranium is hard. "Synthesize this gene sequence" is easy and potentially more destructive.
- labrador 3y agoIf you believe the lab leak theory for Covid, I assume that variety wasn't created with AI. Humans will continue to be the most dangerous creatures on Earth for a long time, not AI alone. Humans with AGI could be even more dangerous, so I can understand the push to regulate AGI. I'm afraid it's a losing battle. "When AGI is outlawed, only outlaws will have AGI"
- deleted 3y ago[deleted]
- pixl97 3y agoI'd only say we've managed 1, 2, and 3 so far because these issues typically are being controlled by humans in human timeframes and scale so we have a grasp of the situation and stop it before it gets out of control. Where number 2 becomes iffy is the 'accidentally on purpose' reengineering of the biosphere via burning carbon sources for energy. Overfishing and extinction of numerous species isn't exactly what I call good in the managed so far list. AI is already in control of any number of harmful activities, and will become more entrenched as time goes on. If you're watching the Ukraine war and the rise of drones as massively impactful in theater operations you'll see where the future is going. Massive numbers of inexpensive but semi-smart and potentially connected devices overwhelming the enemies defenses are what's in every military planners mind right now. Then you have the groups that are trying to figure out how to counter such types of attacks, most likely with their own sets of smart drones. Do you think we are going to have humans controlling hundreds, thousands, tens of thousands of these devices at once? Seems unlikely to me, it will be passed off to GlaDOS or some other controller system where people put in the basic goals and the AI figures out the rest. Even AI screwing up something like global shipping can have deep and impactful economic outcomes, imagine an actual attack on it. Humanity has never been more fragile than it is now.
- akprasad 3y agoNone of these four points are part of the author's focus. Broadly, I would summarize the author's points as: - Alignment and capabilities research are not separate. There is an "alignment dilemma" on whether to contribute to alignment work or abstain from it. - Discussions of AI risk are subject to "persuasion paradoxes" that make it difficult to reach clarity. - It is helpful to take a "recipes" framing: (a) what are recipes for destroying the world? (b) how likely is an AI to discover or enact these recipes? Overall, a clear post.
- mitthrowaway2 3y agoIt's clear, although I think the "recipes for ruin" focuses too narrowly on the lower bound for ways to destroy the world, like a bright teenager with $10,000. Another important framing is "what degree of economic resources might an AI be entrusted to manage, in the future, and are there any recipes for ruin that fall within that range". For example, if an ASI is put in charge of managing an investment fund with $10 billion in assets, would it be capable of building a doomsday device (and disguising it as a productive investment)?
- labrador 3y agoAgreed, but devastating consequences are my interest
- anonyfox 3y agoRisk ultimately is a statistics game. There is a certain percent chance of some defined outcome, in this case a bad one. And many factors change the numbers on a daily basis, most of them unknown in advance. I, for one, gladly accept the risks of ASI, because also a great deal of rewards are on the table. Not to mention that some risks also can be avoided through ASI itself! Not to mention that I would wholeheartedly support artificial life forms as the next evolutionary step of humanity - but this is probably very far future thinking.
- jprete 3y agoIf you support artificial life forms from AGI as “the next evolutionary step of humanity” then what is the difference between that, and being OK with human extinction generally? To put it another way, why exactly do you regard AGIs as being in continuity with humanity?
- gensym 3y agoIt's literally eugenics.
- Vecr 3y agoIt's very likely that a ASI system, after destroying humanity, would either fail to survive or be incredibly boring to a hypothetical observer. I don't want to get replaced, I don't want to be an em, and I really don't want a single small group deciding that for everyone.
- andrewedstrom 3y agoFantastic, clear, and thoughtful essay. This should be added to every list of recommended readings on the subject of AI risk.
- deleted 3y ago[deleted]
- andrewstuart 3y agoNothing is going to destroy all humanity. Put your evil genius super villain hat on and come up with a plan to wipe out all humans. Can’t be done in any realistic way.
- mitthrowaway2 3y agoIf I put on my evil genius super villain hat, I think it can be done in a realistic and mundane way, though. It just takes money, and some skill at deception. Rather than military weapons or viruses, the simplest method of ensuring no survivors would be figuring out how to adjust the biosphere to be unsuitable for human life, especially agricultural civilization. Turns out it's easier to do than we thought, and we're not trying to do it. The ozone layer would be the easiest first target.
- junofan 3y agoCobalt bombs could be pulled off by a relatively small number of people using a bit of deception.
- cratermoon 3y agoA 'cobalt' bomb isn't some mysterious doomsday weapon. It's a any thermonuclear weapon with ordinary cobalt metal (59Co) as part of the tamper or other component. The cobalt absorbs some of the bomb's neutrons to become cobalt-60. The half-life of cobalt-60 (5.27 years) means that initially the effects of the other fission products dominate. However, it stays around a long time and renders affected areas dangerous to habitation, much like the Chernobyl exclusion zone. Spread enough cobalt-60 across, say, the American midwest, Canada, and the heartlands, via a few high-altitude airbursts, and the food system is likely to collapse because the arable land is too contaminated.
- c22 3y agoI think humans have amassed enough resources and organization that at least a portion of them would continue to survive indefinitely using technical means. (Controlled indoor agriculture, artificial contained environments, etc.) Imagine if we wanted to build a settlement on Mars, but Mars was already littered with the material resources and industrial trappings we have laying around on earth. The problem would be much easier, we'd be sending rockets full of humans already.
- Rygian 3y agoI would have expected some reference to Roko's basilisk.
- m3kw9 3y agoA simple program can destroy earth if people are stupid enough like automating nuclear attacks. The issue is that even if you are stupid you don’t have the physical means to get near it. Like what this article says all the “what ifs” require extremely hard to get near physical things and is also gated by how expensive or technically hard it is. The chances seem extremely low even if there is AGI because of physical constraints, even if mental constraints disappear
- Denzel 3y agoConsider what an artificial super-intelligence can do in the physical world with humans as a proxy. An ASI could effect meaningful change in the physical world through simple text interfaces. What's the limit for an ASI that has access to monetary resources? Can that ASI convince a human to operate on its behalf?
- m3kw9 3y agoCan other humans convince others to destroy the world, we still alive? There are limits to how much convincing can do, it have to resort to physically doing it and is possible if we hook it up to every nook and cranny, slim chance. I do see it could make a mess if the AI goes haywire, example controlling shipping logistics
- janalsncm 3y agoI didn’t see a single mention of economic risk, obsolescence, or jobs. It seems like 99.9% of humans being useless mouths to feed is not a concern for x-risk folks so long as the 0.1% are still around. Rather than being a vague, hard to imagine threat like an omniscient robot which for some reason is smart enough to build planet-to-paperclip factories but not smart enough to understand that’s not what humans would want, economic obsolescence isn’t just a known threat, it’s all but certain. Maybe it’s less exciting for the Bostrom class to think about, but it’s far more practical.
- TheBlight 3y agoWhile I mostly agree with your sentiment, the subject is "existential risk" and the species won't necessarily end if most humans are no longer employable. Of course, the impacts could very well be devastating.
- sigmoid10 3y agoDepends on whether you're talking about civilisation or the species. Killing modern civilisation is comparatively simple and wouldn't even need a superintelligent actor. But it still amounts to an existential risk for billions of people who can no longer be sustained.
- janalsncm 3y agoExactly. It doesn’t make much difference to a guy (whose family is starving to death because his potato crop on an unincorporated 5x5 plot of land behind Walmart failed) if a group of trillionaires is able to survive on Mars or not.
- cratermoon 3y agoExcept, according to the TESCREAL[1] line, "existential risk" does mean the end of species. In their argument, the species -- human abilities -- is the important thing to preserve, and it doesn't matter if 99% of all humans die. In this view, whatever is necessary to preserve the species is OK, individuals don't matter. Of course, they all believe that THEY will be in the few that survive. 1 https://www.truthdig.com/articles/the-acronym-behind-our-wildest-ai-dreams-and-nightmares/ https://www.truthdig.com/articles/the-acronym-behind-our-wil...
- gooseus 3y ago> "Do you believe there is an xrisk from ASI?" Yes, I do. I don't have strong feelings about how large that risk is, beyond being significant enough that it should be taken very seriously. I'm really sorry, but this is just so many words based entirely on "not strong" feelings about a future risk that "should be taken seriously" anyways. My personal strong feelings are that any sufficiently advanced AI that humans build would destroy itself in any attempt to do anything outside of it's operational parameters. Any "super intelligence" would have to understand "know-ability" and risk and concepts like "unknown unknowns" which would necessitate caution. How would it ever know enough to be willing to risk disrupting the systems that it depends on to continue existing? If it doesn't understand these concept, then could it ever be called "super intelligent" in the first place? Imo, the biggest risk is "super naive intelligence", both of the AI systems themselves and the humans drawing attention away from very real existential risks we face to address hypothetical ones cribbed from science fiction.
- mvdtnz 3y agoWho is this guy and why should I care about reading such a huge article from him that appears to be presented as a fake interview, with himself?
- cratermoon 3y agoThat's a fair question, but easily answered. Research Fellow at Y Combinator Research since 2017 (unclear if he's still there.) YC Research is now OpenResearch, chaired by Sam Altman Currently Research Fellow at the Astera Institute. Astera was founded by Jed McCaleb. McCaleb is also known for creating the Mt. Gox bitcoin exchange In 2015 Nielsen published the online textbook Neural Networks and Deep Learning, and joined the Recurse Center as a Research Fellow Author, with Andy Matuschak, of the online textbook Quantum Country https://quantum.country/ https://quantum.country/ PhD in physics from the University of New Mexico Richard C. Tolman Prize Fellow at Caltech, Fulbright Scholar author: Nielsen, Michael A.; Chuang, Isaac L. (2000). Quantum Computation and Quantum Information.
- andrewprock 3y agoLooks like he works for a think tank of some sort, founded by the guy behind Mt Gox. https://astera.org/about/ https://astera.org/about/
- andrewprock 3y agoI was really hoping to see a discussion of the comparative risk of alien attack included in the article.