> The Chess AI that turns into AGI isn't realistic - it's values are "be the best at chess" which it can do with existing computing power. The AGI has whatever

> The Chess AI that turns into AGI isn't realistic - it's values are "be the best at chess" which it can do with existing computing power. The AGI has whatever values we give it. Existing chess AIs don't seek to maximize their ability to play chess, they seek merely to win the particular game of chess they're playing. But suppose we build a chess-playing AGI and tell it to "be the best at chess". It must anticipate that we might build a second, superior, chess-playing AGI and give it the same goal. One way to be the best at chess would be to prevent that second AGI being built. One way to prevent that second AGI being built would be to destroy humanity's capability to build AGIs. That probably counts as a loss for humanity. Suppose the second AGI gets built despite the first's efforts. Now both AGIs have an incentive to destroy both the other, and the possibility of a third. At any particular time, one or both of the AGIs won't be the best at chess, so they'll also have an incentive to get better at chess by actually improving their chess-playing capability. This will involve converting the Earth into processing power for it to use. That probably counts as a loss for humanity.