3 ms·
Yes, actually, there is. (I have no idea why Yudkowsky didn’t present it, other than “Twitter isn’t a good platform for this kind of question in the first place
by fwlr 4y ago
Yes, actually, there is. (I have no idea why Yudkowsky didn’t present it, other than “Twitter isn’t a good platform for this kind of question in the first place, and ever since his Time essay and his appearance on the Fridman podcast, his Twitter timeline has been deluged by disagreeable people, making it an even worse place for this kind of question”.)
The canonical source for ~all arguments for AGI ruin is Omohundro’s Basic AI Drives: https://selfawaresystems.files.wordpress.com/2008/01/ai_drives_final.pdf https://selfawaresystems.files.wordpress.com/2008/01/ai_driv... (pdf)
It argues that several basic drives will arise in any goal-directed intelligent agent. The relevant drives to the AGI ruin argument are that it will protect its goals from arbitrary edits, it will want to survive, and it will want to acquire resources (please do read the paper; I am summarizing its conclusions and not its arguments, which it makes significant effort to ground in first principles and basic decision theory to make them as general as possible - for example, it argues that a “drive to survive“ will manifest even in the explicit absence of any kind of self-preservation rule or “survival instinct“).
The general base argument for the risk of AGI ruin could thus be summarized as:
Humans depend on certain configurations of matter and energy to continue to exist; effective AGIs are likely to reconfigure that matter and energy in ways incompatible with humans, not because “they hate us”, but because 1. most configurations of matter and energy are incompatible with humans, and 2. reconfiguring matter and energy is how goal-directed intelligent agents achieve their goals.
All of the individually unlikely AGI will kill us in this way scenarios are just specific instantiations of this general argument (e.g. Clippy will kill us all because we are made of matter that could be rearranged to form paperclips, or an intelligent server farm will kill us all by freezing the whole Earth because it determined that its processors would run more efficiently at -10C.)
- yeck 4y agoThanks. While I don't personally need to be convinced of the existential risks for AGI/ASI, I was also genuinely interested in know what might serve as a good, clear "canonical" argument for why the risk is present. Definitely going to keep this link on standby.