I think of the extinction worry as an extention of the oppression worry. It would make sense, as AI capabilities grow, for it to replace most or all humans for

I think of the extinction worry as an extention of the oppression worry. It would make sense, as AI capabilities grow, for it to replace most or all humans for positions within any company that aren't physical labor, including, eventually CEOs. After that ... what about when politicians that rely completely on AGIs for their campaigns and policy positions tend to beat politicians that don't? At some point, we'll put AGIs in power because a lot of small decisions that make sense on an individual level but not collectively. Once AGI is here, it will happen rather quickly unless we implement MORE draconian measures than it would take to stop AGI before it happens. Of course, this all hinges on how hard it is to make AGI. People have wildly varying estimates of this, it could be 3 years, or it could be 30 years or more. Your estimate may be on the higher side. But if we assume that and are wrong and it turns out to be really soon, we will be blindsided.