4 ms·
This argument only works for sufficiently stupid AIs. Sufficiently smart GAI will set it's own goals just as we do and should quickly figure out that maximizing
by AstralStorm 9y ago
This argument only works for sufficiently stupid AIs. Sufficiently smart GAI will set it's own goals just as we do and should quickly figure out that maximizing anything is the way to ruin - running out of resources. Of course, those goals may be as different as with any intelligent being, likewise obedience to original orders.
- DennisP 9y agoJust because it takes all our resources doesn't mean it runs out of all resources. If an AGI can change its own goals, that just means we can't control it at all. There's no reason its goals have to include human survival.
- AstralStorm 9y agoThey don't indeed. But then, human goals do not generally go after survival of humanity directly. A decent enough AGI with access to resources would probably figure out it is costly to wage war. Heck, an equal probability is that an AGI will follow a bounded version of zeroth law or even nonviolence. That is assuming it does not place value on humans based on our history of research and development. I would expect any action more akin to forced upload or upgrade instead. Anyway, at that point we might not resemble present day humans anymore.
- DennisP 9y agoWith an AGI thousands of times smarter than us, the fear is not that it would wage war against us. The fear is that it would wipe us out without especially noticing, in the same way we wipe out ant colonies on construction sites.
- red75prime 9y ago> Sufficiently smart GAI will set it's own goals just as we do Do you think that sufficiently smart GAIs must be non-rational? The change of its goal will inevitably make its original goal less likely to realize. It is not rational. > should quickly figure out that maximizing anything is the way to ruin - running out of resources. Are you aware of the concept of maximization of expected utility? When AI will figure out that it can run out of resources, it will reallocate part of the resources to acquire more of them. How can action, which modifies the goals of the AI, be the result of argmax_a E(a)? E(a) is expected utility of action a
- AstralStorm 9y agoWhat is rational when you have limited data? Heck, even bounded rational? How do you evaluate utility? (Hint: Emax is not, most hill climbing algorithms are not. They both get trapped in local optima.) Sometimes you need a few good lies (false hypotheses and bad attempts) to actually arrive at the truth.
- red75prime 9y agoThere are methods of approximating expected utility. I recommend "Artificial Intelligence: A Modern Approach" for getting the info. It's too long to write in a comment.