If you want to prevent this, you simply have to show that the probability for that extinction scenario is lower than the baseline where we start to care. Lets

If you want to prevent this, you simply have to show that the probability for that extinction scenario is lower than the baseline where we start to care. Lets take "big asteroid impact" as baseline because that is a credible risk and somewhat feasible to quantify: Probability is somewhere under 1 in a million over a human lifetime, and we barely care (=> we do care enough to pay for probe missions investigating possible mitigations!). So the following requirements: 1) Humanity creates one or more AI agents with strictly superhuman cognitive abilities within the century 2) AI acquires power/means to effect human extinction 3) AI decides against coexistence with humans Only need 1% probability each to exceed that probability bound. And especially 1) and 3) seem significantly more likely than 1% to me, so the conclusion would be that we should worry about AI extinction risks...