> The idea that an exponentially self-improving being will arise seems unlikely when nature has been trying to do that for eons. I would say that it is already

> The idea that an exponentially self-improving being will arise seems unlikely when nature has been trying to do that for eons. I would say that it is already successful, if you look at the curve of intelligence and complexity growth. It took eons to get anything resembling life at all, and then a small slice of that to get animals, and an even smaller to get humans. I think one key idea here, though, is that the evolution of intelligence is in fact the evolution of intelligence-producing systems. In other words, nature is producing systems that are better and better at "self-improvement", and I think proponents of "intelligence explosion" overestimate how much self-improvement could be improved via AI. Personally, I suspect indefinite self-improvement is flatly impossible because that's akin to saying that there exists an incremental learning algorithm that systemically finds global minima. As AI research suggests, though, such algorithms likely don't exist at all: if you want to get truly optimal AI there is a point where nothing in your current state is salvageable and you just gotta restart from zero. Improvement would still be exponential, but not to an extent that it couldn't be dealt with.