> By far the greater risk in that world is the economy/universe optimizing out everything that makes humans human until the entire universe is simply a highly e

> By far the greater risk in that world is the economy/universe optimizing out everything that makes humans human until the entire universe is simply a highly efficient philosophical zombie, rather than the risk that every last subsystem of everything in the world is secretly going to be a human-class homunculus trapped in some boring task. This, to me, is a greater risk than "evil AI." If humanity does cease to exist, it will be a slow, painless, barely observable extinction (perhaps even paraded as ascension) as we integrate with and are then replaced by the technology we've created. I don't think this is entirely realistic, just because there's an anthropocentric bias in me that believes that humans are wonderfully useful and robust self-replicating machines, and that economically we're (1) fairly energy efficient and (2) valuable to "throw at problems" we don't want to mass produce specific silicon-based solutions to. But I did want to thank you for pointing this out. I do think it's wonderfully naive we often ascribe human characteristics and fallacies to hypothetical general AI that we experiment with or build, as though distinctly human consciousness is somehow the epitome of achievement. It's not even in our best interest, economically or ethically, to strive for that.