For the record I think if we saw the emergence of strong AI, that could actually match the full range of human capabilities, including independently setting and

For the record I think if we saw the emergence of strong AI, that could actually match the full range of human capabilities, including independently setting and pursuing objectives to fulful a long-term fitness function, it would be the greatest existential threat that humanity has ever faced, far exceeding that posed by nuclear weapons and environmental collapse through pollution and over-consumption of natural resources. But the scenario envisioned is not why I would consider it such a threat. It's the AI that integrates in society that is dangerous, not some cartoonish unstoppable rogue program, or group of programs, that single-handedly plow through all defenses.