As you are talking about extinction level events, I'm not very confident that if humans have ultimate say over nuclear weapons, we will continue to not end our

As you are talking about extinction level events, I'm not very confident that if humans have ultimate say over nuclear weapons, we will continue to not end our species with them. It might in fact be a good idea to establish an AGI overlord which watches over humans and enforces nuclear non proliferation policies. If you look through history, it's full with war, genocide, and similar. Human societies are bound for change, and while it's been a peaceful few decades in which we had the nuclear button, it's basically ensured that we'll press it in the next 10 thousand years. How will technological civilization become million years old if not with the help of an AGI that enforces basic rules like "don't nuke each other"?