I think the biggest issue with the thought experiment is that there's never going to be one person sitting there that can press a button that irreversibly dooms

I think the biggest issue with the thought experiment is that there's never going to be one person sitting there that can press a button that irreversibly dooms humanity. There's never even an attempt at a logical sequence of events that leads from the moment of that breach to our complete doom. It's not inconceivable but I feel like these fears need to be grounded in a plausible "here's how it could actually happen." What prevents the combined abilities of the human race from shutting it down? AI isn't magic, it is still confined to the pretty fragile and narrow bounds of digital technology.