You need to read the statement from the context of "in principle" vs "in practice". When, in principle, no human has the capacity to understand or control a thi

You need to read the statement from the context of "in principle" vs "in practice". When, in principle, no human has the capacity to understand or control a thing. A forest fire is a great thought experiment - we can, in principle, control and understand any forest fire. Given enough time and resources, no forest fire will ever exceed our theoretical capacity to control it. In principle, the technological rate of progress will exceed humanity's capacity to understand or control it. It will become a thing which happens to us, as opposed to us doing things. Intelligence is one of those "we know it when we see it" problems. It literally encompasses the entirety of human cognition. Moravec described AI capabilities in terms of human equivalence, avoiding the need for any specific and well defined unit of measurement. If and when AI capabilities exceed human capabilities in any and every possible capacity, AI will be more intelligent than humans, and we can be certain that AGI is achieved. When we run out of "but machines can't do X as well as a human!" statements, we'll have achieved software that, when reproduced, optimized, and tasked en masse, will represent superintelligence. There are multiple paths to get there. There are dozens, perhaps hundreds, of books covering the topic in depth. There's no "exponentially more time than the last" issue with AI development. The halting problem can't be solved - that's the point of the halting problem. We have sufficient understanding of computation and intelligence to say that there is nothing about the laws of physics, or the way biological brains work that even remotely suggests that anything brains do is impossible, or even particularly difficult, for silicon chips to achieve. The bottleneck in reproducing brains in silicon right now is data. Because of ethical and technical challenges, we can't gain access to the low level details of living brain tissue, but the science of connectome mapping, neurochemistry, and real-time electrical activity tracking continues to improve, and we will absolutely have mapped the entirety of the human brain down to the molecular level at some point in the next century. Given that information, we'll be able to recreate important structures in software, and produce functional equivalents to things like cortical neural columns and minicolumns, and we'll gain access to Hebbian dynamics that govern the wiring and growth of synapses, allowing us to recreate the rules that comprise human intelligence. That's the upper bound on when we're going to achieve artificial general intelligence - it may take a decade or two after we've got the map, but it's likely to happen a long time before we get the complete map, because much of what the brain is doing is based on highly repeated structures distributed throughout the neocortex - figure out how they work, how their behavior is coordinated, and how that produces learning and behavior, and you crack the problem of intelligence.