Humans can't recursively self-improve in the sense AI adherents mean it, because we are stuck in our meat brains and can't redesign them or move to faster hardw

Humans can't recursively self-improve in the sense AI adherents mean it, because we are stuck in our meat brains and can't redesign them or move to faster hardware (yet!, they'll say). What's being smuggled under the philosophical carpet here is exactly what you point out. There's no guarantee that an AI would be a humorless utilitarian rationalist obsessed with self-improvement. It's also possible that general intelligence is a tradeoff against something else (like existential despair, or a tendency to veer off into writing extropian fanfic) and entities hundreds, or thousands, or tens of times smarter than human beings can't exist. Scio me nihil scire.