According to a survey summarized in this post from 2021 (https://www.astralcodexten.com/p/updated-look-at-long-term-ai-risks https://www.astralcodexten.com/p/up

According to a survey summarized in this post from 2021 (https://www.astralcodexten.com/p/updated-look-at-long-term-ai-risks https://www.astralcodexten.com/p/updated-look-at-long-term-a...), “people who work in ‘AI safety and governance’” “mostly assign “low” probability (~10%) that unaligned AI will result in human extinction”. They more commonly worry about other scenarios that are still very bad for humanity. To give one example, an AI could be instructed to reduce the crime rate, but rather than preventing people from committing crimes, the AI could decide it’s easier to prevent all victims from reporting crimes to anybody. That would remove the threat of law enforcement that currently prevents many people from turning to crime. Such a society probably wouldn’t cause human extinction, but would still be horrific and worth preventing. If you are skeptical that AI that is trapped in software could gain any significant real-world power (not necessarily the power to kill all of humanity), https://slatestarcodex.com/2015/04/07/no-physical-substrate-no-problem/ https://slatestarcodex.com/2015/04/07/no-physical-substrate-... describes some possible mechanisms.