Sam wants to talk about the utility function of a superhuman AI. Ok, what are the likely outcomes? (For sake of discussion, let's just accept that a superhuman

Sam wants to talk about the utility function of a superhuman AI. Ok, what are the likely outcomes? (For sake of discussion, let's just accept that a superhuman AI will exist soon.) Asimov's "Three Laws" point out that a utility function is just a program like any other program. It has no inherent moral code. If it prefers "the good of mankind," it is because the engineers made it so. How long until someone makes one that actively destroys mankind? Sam seems to accept that a single superhuman hostile AI is an extinction-level event. Friendly AIs are non-events, but a hostile AI is an existential threat. There are counter-arguments: humans are resilient; our monoculture hasn't wiped out all avenues of escape; governments are still human-dominated and unlikely to surrender to AI control. I haven't decided yet, but I am convinced there are concrete actions _right_ _now_ that have a strong effect on the outcome. The real contest, though, is that humans just don't care, and AIs are tireless, flawless machines.