You're still assuming that the assistant cares about doing what I mean, rather than doing precisely what I say or doing what I programmed it to do. That only ha

You're still assuming that the assistant cares about doing what I mean, rather than doing precisely what I say or doing what I programmed it to do. That only happens if we're really, really careful about how we programmed the AI in the first place. I grant you 100% that the AI probably knows precisely how many paperclips I want, but you're assuming that it wants the same number of paperclips that I want. What an agent considers to be "good" is orthogonal to how intelligent that agent is. An agent of arbitrary intelligence can have arbitrary goals; the goals of an intelligent agent need not in principle look anything like those of a human. The only reason a superintelligent AI's goals would look like those of a human is because the humans very, very carefully programmed them into the AI somehow. Very, very careful programming is not a feature of how humans currently approach AI research.