Exactly. I don't know what time frame we're talking about, or whether I want to see it in my lifetime. But it's something I've spent too much time considering.

Exactly. I don't know what time frame we're talking about, or whether I want to see it in my lifetime. But it's something I've spent too much time considering. But one of the constants (and one of the issues I have with a lot of dystopian sci-fi, even though I enjoy it, is that they do anthropomorphize AGI way too often. Like you mentioned, we don't even understand how consciousness works. But we may not need to understand it to replicate it, and if that replication is allowed to be self-modifying, well, that'd very possibly be it. Hopefully we can try to embed some sort of Asimov's Law of Robotics or some morality that lasts beyond the vestiges of the human developed portions. Or maybe we can manage to learn how to copy our consciousness into "blank" mechanical "brains", and effectively become similar to AGI without being limited biologically.