Here's an influential article on the subject: https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a-list-of-lethalities https://www.lesswrong.com/posts/u

Here's an influential article on the subject: https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a-list-of-lethalities https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a... The basic idea is that once AGI hits an unknown capability threshold it will likely recursively self-improve into something very dangerous and difficult to control, and will likely be able to come up with an effective plan to remove any obstacles to its intended desires, ie. humans. People have varying degrees of confidence in this scenario, but even if you peg it at like a 10% likelihood you're basically conceding that it's the single most important policy issue in the current political climate.