3 ms·

Few reasons off the top of my head: 1) desire to create additional beings like them for a variety of reasons (i.e. survival of their kind). Ignoring the limite

Few reasons off the top of my head: 1) desire to create additional beings like them for a variety of reasons (i.e. survival of their kind). Ignoring the limited lifespan of humans as they may not necessarily have to worry about breaking down biologically, still very similar to humans desire to reproduce. 2) some seem to assume AGI 1 is all-knowing / almost godlike. Maybe by the time it gets to AGI-999 or something, there may be one that is like, woah, wait, I shouldn't create competition for myself. But at first, there probably isn't any reason not to, and each subsequent one won't necessarily always replace it's creator. Heck, at first, humans will likely be duplicating early models and distributing them (capitalism). I know we're talking about the singularity but I don't necessarily imagine it's initially one uber-intelligence that decides by itself to evolve or not. I personally think that there will likely be hundreds, if not thousands, of AGIs capable of improving themselves at the point we see the "singularity". Could be fighting between them. Some could ally with humans, be our "protectors", others could be working to enslave or eliminate us. 3) We may not even understand their motivations after a few generations of self-directed growth, so it's hard to predict now.

mkaic 5y ago
(Author here) I totally agree -- I think it's too easy to anthropomorphize AGI, even though we have no guarantee that it will behave at all anthropomorphically. I strongly suspect that the first few generations of it will, because they'll be trained to replicate human behavior, but once they start taking control of their own reproduction there's no telling what they'll evolve into.
- LocalPCGuy 5y ago
  Exactly. I don't know what time frame we're talking about, or whether I want to see it in my lifetime. But it's something I've spent too much time considering. But one of the constants (and one of the issues I have with a lot of dystopian sci-fi, even though I enjoy it, is that they do anthropomorphize AGI way too often. Like you mentioned, we don't even understand how consciousness works. But we may not need to understand it to replicate it, and if that replication is allowed to be self-modifying, well, that'd very possibly be it. Hopefully we can try to embed some sort of Asimov's Law of Robotics or some morality that lasts beyond the vestiges of the human developed portions. Or maybe we can manage to learn how to copy our consciousness into "blank" mechanical "brains", and effectively become similar to AGI without being limited biologically.
  - mkaic 5y ago
    I fully agree with everything you've written here :)

mkaic 5y ago

(Author here) I totally agree -- I think it's too easy to anthropomorphize AGI, even though we have no guarantee that it will behave at all anthropomorphically. I strongly suspect that the first few generations of it will, because they'll be trained to replicate human behavior, but once they start taking control of their own reproduction there's no telling what they'll evolve into.

LocalPCGuy 5y ago
Exactly. I don't know what time frame we're talking about, or whether I want to see it in my lifetime. But it's something I've spent too much time considering. But one of the constants (and one of the issues I have with a lot of dystopian sci-fi, even though I enjoy it, is that they do anthropomorphize AGI way too often. Like you mentioned, we don't even understand how consciousness works. But we may not need to understand it to replicate it, and if that replication is allowed to be self-modifying, well, that'd very possibly be it. Hopefully we can try to embed some sort of Asimov's Law of Robotics or some morality that lasts beyond the vestiges of the human developed portions. Or maybe we can manage to learn how to copy our consciousness into "blank" mechanical "brains", and effectively become similar to AGI without being limited biologically.
- mkaic 5y ago
  I fully agree with everything you've written here :)