I agree. But, so far we have come closest to destroying the world when nation states are stockpiling weapons of war, threatening mutually assured destruction, w

I agree. But, so far we have come closest to destroying the world when nation states are stockpiling weapons of war, threatening mutually assured destruction, with their arsenals kept on a hair trigger. To me, that seems likely to keep being the main danger scenario in the future. It would still be a "mistake", but not the "oops what does this button do" sort of mistake, but rather the "I could have sworn they fired first" sort of mistake. That is why I think the main goal of AI safety should be how to defend against intentionally malicious AI. We should be thinking about drone swarms and hacker-AIs rather than paper clips.