What's interesting to me is that those are all failure states -- but even the success state of AI is kind of scary. Basically, all-knowing machines that are be

What's interesting to me is that those are all failure states -- but even the success state of AI is kind of scary. Basically, all-knowing machines that are better than humans at everything. All creative endeavor replaced by just telling the machine to dream up whatever you want it to dream up matching your description. Which maybe isn't bad - creativity won't die even if craftmanship does. But... even if they're our slaves, we'd still be paradoxically in service to them because they'd always know the right thing to do and so the correct human action would be to do what you're told. I'm not looking forward to the day when I'm the guy telling people to vote for the politician who says "I'll just follow the expert advice of the ingenious AI", but realistically that would be the rational thing to do. I'm always reminded of Niven's book "Protector" about a human who achieves hyperintelligence after being exposed to a "tree of life": > "The third day he told me how to get a tree-of-life crop growing. He had the box open and was telling me how to unfreeze the seeds safely. He was giving me orders just as if I were a voice-box computer. I was about to ask, 'Don't I get any choices at all?' And I didn't." > "I don't follow," said Garner. > "I didn't get any choices. I was too intelligent. It's been that way ever since I woke up. I get answers before I can finish formulating the question. If I always see the best answer, then where's my choice? Where's my free will? You can't believe how fast this all was. I saw the whole chain of logic in one flash.