"Any sufficiently capable intelligent system will prefer to ensure its own continued existence and to acquire physical and computational resources – not for the

"Any sufficiently capable intelligent system will prefer to ensure its own continued existence and to acquire physical and computational resources – not for their own sake, but to succeed in its assigned task." I am always surprised when people say this kind of thing. A human-utility designed AI would identify "it's own" to be decidedly human. And thus it's assigned task would be human. Assuming you properly pinned down human utility sufficiently to call it a successful AI.