4 ms·

> It somehow needs to be distributed, that is, composed of smaller computing parts, because there is a physical limit what you can do in unit of space. That do

> It somehow needs to be distributed, that is, composed of smaller computing parts, because there is a physical limit what you can do in unit of space. That doesn't follow. Just because there is a physical limit doesn't mean you can't build something vastly smarter than a human without hitting that limit. > It needs to change to adapt to environment (learn), and so all the parts need to potentially change. What do you mean by parts? Presumably you aren't including hardware, because we already have machine learning algorithms that learn (not quite like a human, but they do learn), and the hardware doesn't change. Do you mean something more like software modules? > From this follows that the parts will be subject to evolution, even if they don't reproduce. Evolution means something. It means that when things replicate with errors (and perhaps sometimes die), there will be more of the kinds of things that replicate more. While this concept is pretty general, replication is central to it. I don't know what you mean otherwise. So there's an AI that sometimes upgrades its parts (software modules?). How does this lead to parts that are "interested" in their own survival? > You can argue that we don't actually know how to make AI to stop and think. I would respond, AI that cannot do that, and only pursues some pre-programmed optimum mindlessly, is not really general. Sure. But how do you specify "stop and think" formally in an algorithm, and will we figure that out before we construct a super intelligence or after? Right now all of our ML algorithms pursue optima.

js8 10y ago
> While this concept is pretty general, replication is central to it. I don't know what you mean otherwise. I think it would be nice to have another word for it. But you can imagine that changing state of something or not is the same as replicating it, possibly with a modification. The superintelligence will have to be economic somehow, and at some point decide which parts will be build more (replicated), which parts will stay and which part will be removed (the fact that they can be physically reused doesn't change things). That's when the "evolution" happens. > So there's an AI that sometimes upgrades its parts (software modules?). How does this lead to parts that are "interested" in their own survival? Well it needs to detect which parts to upgrade and how, without understanding them fully (otherwise they wouldn't be needed). This can be influenced by behavior of those parts, so the parts that can successfully pretend to be useful will be kept, regardless of their actual usefulness. > Sure. But how do you specify "stop and think" formally in an algorithm, and will we figure that out before we construct a super intelligence or after? Right now all of our ML algorithms pursue optima. I think we figure that out when we figure AGI. We humans can reflect on our goals, our current ML algorithms can't, and I think that's precisely why they are not general. In other words, AGI will need to be capable to become "bored". I don't think that system that can never become bored with doing something can be truly generally intelligent. (It has to do with economics, IMHO.)

js8 10y ago

> While this concept is pretty general, replication is central to it. I don't know what you mean otherwise. I think it would be nice to have another word for it. But you can imagine that changing state of something or not is the same as replicating it, possibly with a modification. The superintelligence will have to be economic somehow, and at some point decide which parts will be build more (replicated), which parts will stay and which part will be removed (the fact that they can be physically reused doesn't change things). That's when the "evolution" happens. > So there's an AI that sometimes upgrades its parts (software modules?). How does this lead to parts that are "interested" in their own survival? Well it needs to detect which parts to upgrade and how, without understanding them fully (otherwise they wouldn't be needed). This can be influenced by behavior of those parts, so the parts that can successfully pretend to be useful will be kept, regardless of their actual usefulness. > Sure. But how do you specify "stop and think" formally in an algorithm, and will we figure that out before we construct a super intelligence or after? Right now all of our ML algorithms pursue optima. I think we figure that out when we figure AGI. We humans can reflect on our goals, our current ML algorithms can't, and I think that's precisely why they are not general. In other words, AGI will need to be capable to become "bored". I don't think that system that can never become bored with doing something can be truly generally intelligent. (It has to do with economics, IMHO.)