The whole discussion is way too antropocentric. We are analysing what would an AI want and desire and whatnot, assuming that an AI would have wants and desires.

The whole discussion is way too antropocentric. We are analysing what would an AI want and desire and whatnot, assuming that an AI would have wants and desires. But also ignoring that an AI could act simply out of programmed logic. One can get catastrophic results also out of (mis)programmed logic - every prorammer here can confirm. So let's not deny the catastrophic possibility only because we imagine an AI must be first a human approximation in order to take catastrophic decisions. The danger is instead when the AI can take catastrophic decision - then we can be truly effed even if it's out of a simple bug.