I'd argue that AI itself isn't the initial threat, because, as you say, the first AI capable of causing harm will likely be one whose goals are set by humans, w

I'd argue that AI itself isn't the initial threat, because, as you say, the first AI capable of causing harm will likely be one whose goals are set by humans, who already have a bunch of harmful desires. Indeed, we already are seeing machine-learning systems used by large corporations and governments for nefarious ends. But long term, assuming we manage to get past that problem and limit the goals that humans choose to set for their AIs (such as the obviously common goal of "don't destroy humanity"), there is the danger that the AI develops both its own goals and the ability to understand its place in the world enough to work towards those goals. This is the problem of "alignment" in AI safety research. https://en.wikipedia.org/wiki/AI_control_problem#Alignment https://en.wikipedia.org/wiki/AI_control_problem#Alignment