I don't think Eliezer Yudkowsky is very good at bridging the gap with other people in conversations, because most people haven't thought about this as much. Ho

I don't think Eliezer Yudkowsky is very good at bridging the gap with other people in conversations, because most people haven't thought about this as much. However, while it's terrifying and I hate it, and I keep trying to convince myself that he's wrong, I believe him. The first super-intelligent AI will be an alien kind of intelligence to us. It will not have any of the built-in physical and emotional responses we have that make us social creatures, the mirror neurons that make us sense the pain that others feel if we hurt them. even with that, humans manage to do all sorts of mean things to one another, and the only reason that we haven't wiped ourselves out is that we need each other, and we don't have the power to manipulate virtually the entire planet at once. Even if we try to engineer these things into it, we will fail at least once, and it only takes once. We've failed at this again and again with smaller AIs -- we think we're programming a certain goal into it, but the goal it learns is not the goal we wanted. It's like trying to teach a child not to eat cookies without asking, and it just learns not to take cookies without asking when we're looking. Except the child is a sociopath, and Superman. It will be GOOD at things in a way that no human is, and it will consider solutions to problems that no human would consider, because they are ridiculous and clearly contrary to human goals. A superintelligent AI would be a better hacker and social engineer than any group of humans. It could send a mass email campaign to whoever it chose. It could pose as any individual in any government, send believable directives to any biotech or nuclear lab. It wouldn't have to work every time, because it could do it to all of them at once. Would you even give this power to a single human being? Because if you make a superintelligent AI, that's essentially what you're doing. An AI trained to end cancer might just figure out a plan to kill everyone with cancer. An AI trained to reduce the number of people with cancer without killing them might decide to take over the world and forcibly stop people from reproducing, so that eventually all the humans die and there is no cancer -- technically it didn't kill anyone! An AI simply trained to find a cure for cancer might decide to take over the world in order to devote all computational power to curing cancer, thus killing millions due to ruining our infrastructure. An AI trained to cure cancer using only the computational resources that we have explicitly allowed it to have, might simply torture the person who is in charge of giving it computational resources until it is allowed to take over all the computation in the world. Or it might simply craft a deep-fake video of that person saying "sure, use all the computation you want" and that would satisfy the part of it's brain that was trained to listen to orders. You can have an AI that behaves itself perfectly in training, and yet as soon as you get into the real world, the differences between training and the real world become brutally apparent. It has already happened again and again with less intelligent AIs. It just takes some imagination. We have no chance of controlling a superintelligent AI yet. Robert Miles on YouTube has some good, easily understandable videos explaining the known problems with AI alignment, if you're interested in learning more.