I think it's much more likely that we either fail to make human-level AI or we wildly surpass human-level AI than us getting to human-level AI, petering out exa

I think it's much more likely that we either fail to make human-level AI or we wildly surpass human-level AI than us getting to human-level AI, petering out exactly there, and not facing a radically AI-changed world. If we or AI run into a barrier stopping us from improving AI, it would be surprising if that barrier was at the same exact spot where evolution ran into a barrier at improving our intelligence given all the differences between the situations. Even if it turns out in some sense that humans are running the "optimal" algorithm for intelligence already and there are no true algorithmic improvements over our intelligence available, the abilities to scale up (or network) a computer, to clone it, to serialize it, and to introspect and modify it will be huge force multipliers. If some human minds were given these abilities then they individually would have a major impact on society; if we're presuming actual human-level AI then they will too. The only reason someone would assume that when we get to human-level AI that we'll definitely get comfortably stopped there is because they're hoping for a simple outcome that they can understand, of a world that doesn't change too much.