4 ms·
"The assumption that any intelligent agent will want to recursively self-improve, let alone conquer the galaxy, to better achieve its goals makes unwarranted as
by apsec112 10y ago
"The assumption that any intelligent agent will want to recursively self-improve, let alone conquer the galaxy, to better achieve its goals makes unwarranted assumptions about the nature of motivation."
This isn't just an unreflective assumption. The argument is laid out in much more detail in "The Basic AI Drives" (Omohundro 2008, https://selfawaresystems.files.wordpress.com/2008/01/ai_drives_final.pdf https://selfawaresystems.files.wordpress.com/2008/01/ai_driv...), which is expanded on in a 2012 paper (http://www.nickbostrom.com/superintelligentwill.pdf http://www.nickbostrom.com/superintelligentwill.pdf).
- tlb 10y agoCertainly the assumption that every intelligent agent will want to recursively self-improve is unwarranted. But it only takes one intelligent agent that wants to self-improve for the scary thing to happen.
- coldtea 10y ago>But it only takes one intelligent agent that wants to self-improve for the scary thing to happen. Only if all sorts of other conditions (several of which are mentioned in the post) also apply. Merely "wanting to self-improve" is not enough.
- flukus 10y agoWouldn't a constant desire to self improve mean a constant desire for more energy? That would bring it into conflict with other beings that want that energy.
- flukus 10y agoI wonder if an AI would favor a recursive or genetic approach? If it's the latter then an AI could see value in death.
- trishume 10y agoWhy wouldn't it if it is able to? It doesn't have to "want" to self-improve, it only has to want anything that it could do better if it was smarter. All it needs is the ability, the lack of an overwhelming reason not to, and a basic architecture of optimizing towards a goal. If you knew an asteroid would hit the earth 1 year from now, and you had the ability to push a button and become 100,000x smarter, I would hope your values would lead you to push the button because it gives you the best chance of saving the world.
- SilasX 10y agoBecause there are tradeoffs. Whatever it's goal is, some of those "drives" (instrumental values) will be more effective for reaching that goal over the timespan that it cares about. Sometimes "accumulating more resources" is the most effective way. Sometimes "better understanding what problem I'm trying to solve" is the most effective way". Sometimes "resisting attempts to subvert my mind" is the most effective way. And yes, sometimes "becoming better at general problem solving" (self improvement of one's intelligence) is the most effective way. But there's no guarantee that any one of those will be a relevant bottleneck in any particular domain, so there's no guarantee an agent will pursue all of those drives.
- trishume 10y agoAgreed. But if the goal is something like "build the largest number of paperclips", recursive self improvement is going to be a phenomenally good way to achieve that, unless it is already intelligent enough to be able to tile the universe with paperclips. Either way we don't care if it self improved or not, that's just the seemingly most likely path, we just care if it is overwhelmingly more powerful than us. The only thing that stops me from recursively self improving is that I'm not able to. If I could it would be a fantastic way to do good things that as an altruistic human I want to do. Like averting crises (climate change, nuclear war), minimizing poverty and misery, etc...
- Applejinx 10y ago'build the largest number of paperclips' is a nihilistic, unintelligent goal.
- coldtea 10y agoSo? Just because it's in some paper doesn't mean much. There are tons of BS string theory papers for example.