3 ms·

If the AI is actually human-level and is made by humans, then it will be capable of working on itself. Assuming the evolution of human intelligence was mainly l

If the AI is actually human-level and is made by humans, then it will be capable of working on itself. Assuming the evolution of human intelligence was mainly limited by biological factors that don't apply to AI (like that brains can't be too big to go through the birth canal, that a body needs to optimize how much energy it gives to the brain vs other parts, etc), then if we hit human-level AI then we're going to be able to go past that. If humans are able to improve AI at any speed, then the AI would be able to improve itself exponentially faster because it will benefit from its own improvements.

TheOtherHobbes 4y ago
Why would anyone assume that AI won't suffer from either physical limitations or abstract ones? We can barely make a GPU that doesn't melt, and we certainly can't make reliable non-trivial software. If you increase complexity by a few orders of magnitude those problems not only don't go away, they become far worse.
- AgentME 4y ago
  I think it's much more likely that we either fail to make human-level AI or we wildly surpass human-level AI than us getting to human-level AI, petering out exactly there, and not facing a radically AI-changed world. If we or AI run into a barrier stopping us from improving AI, it would be surprising if that barrier was at the same exact spot where evolution ran into a barrier at improving our intelligence given all the differences between the situations. Even if it turns out in some sense that humans are running the "optimal" algorithm for intelligence already and there are no true algorithmic improvements over our intelligence available, the abilities to scale up (or network) a computer, to clone it, to serialize it, and to introspect and modify it will be huge force multipliers. If some human minds were given these abilities then they individually would have a major impact on society; if we're presuming actual human-level AI then they will too. The only reason someone would assume that when we get to human-level AI that we'll definitely get comfortably stopped there is because they're hoping for a simple outcome that they can understand, of a world that doesn't change too much.

TheOtherHobbes 4y ago

Why would anyone assume that AI won't suffer from either physical limitations or abstract ones? We can barely make a GPU that doesn't melt, and we certainly can't make reliable non-trivial software. If you increase complexity by a few orders of magnitude those problems not only don't go away, they become far worse.

AgentME 4y ago
I think it's much more likely that we either fail to make human-level AI or we wildly surpass human-level AI than us getting to human-level AI, petering out exactly there, and not facing a radically AI-changed world. If we or AI run into a barrier stopping us from improving AI, it would be surprising if that barrier was at the same exact spot where evolution ran into a barrier at improving our intelligence given all the differences between the situations. Even if it turns out in some sense that humans are running the "optimal" algorithm for intelligence already and there are no true algorithmic improvements over our intelligence available, the abilities to scale up (or network) a computer, to clone it, to serialize it, and to introspect and modify it will be huge force multipliers. If some human minds were given these abilities then they individually would have a major impact on society; if we're presuming actual human-level AI then they will too. The only reason someone would assume that when we get to human-level AI that we'll definitely get comfortably stopped there is because they're hoping for a simple outcome that they can understand, of a world that doesn't change too much.