> While this concept is pretty general, replication is central to it. I don't know what you mean otherwise. I think it would be nice to have another word for i

> While this concept is pretty general, replication is central to it. I don't know what you mean otherwise. I think it would be nice to have another word for it. But you can imagine that changing state of something or not is the same as replicating it, possibly with a modification. The superintelligence will have to be economic somehow, and at some point decide which parts will be build more (replicated), which parts will stay and which part will be removed (the fact that they can be physically reused doesn't change things). That's when the "evolution" happens. > So there's an AI that sometimes upgrades its parts (software modules?). How does this lead to parts that are "interested" in their own survival? Well it needs to detect which parts to upgrade and how, without understanding them fully (otherwise they wouldn't be needed). This can be influenced by behavior of those parts, so the parts that can successfully pretend to be useful will be kept, regardless of their actual usefulness. > Sure. But how do you specify "stop and think" formally in an algorithm, and will we figure that out before we construct a super intelligence or after? Right now all of our ML algorithms pursue optima. I think we figure that out when we figure AGI. We humans can reflect on our goals, our current ML algorithms can't, and I think that's precisely why they are not general. In other words, AGI will need to be capable to become "bored". I don't think that system that can never become bored with doing something can be truly generally intelligent. (It has to do with economics, IMHO.)