"The idea of exponential growth, which seems like an important underpinning of the singularity theory. Nothing in nature as far as I am aware, grows exponential

"The idea of exponential growth, which seems like an important underpinning of the singularity theory. Nothing in nature as far as I am aware, grows exponentially." Actually a lot of things grow exponentially, at least for a while. Consider zygotes, every cell division period they double. That said, AI is exactly not "natural". A major point a lot of folks miss thinking about this is AI is amenable to engineering. This means that AIs can be intentionally designed to not have the limitations of natural systems. For instance, even today we know how to make effectively infallible, very fast memories of arbitrary size. The other thing that makes strong AIs seem likely to improve exponentially is the idea that the first strong AIs could be focused on creating better strong AIs - and so on. Since AIs may also be replicated at will, a whole lot of distraction-free, singleminded intellect will be brought to bear on the next-gen AI design, and there is no reason that process couldn't be repeated indefinitely. "At some point the laws of physics will contrain further advances, and we will encounter the upper half of the s-curve." That might be true, however we also might be at IQ 1,000 (or IQ 1,000,000) AIs by then... "The idea of AI in a box. It's the idea that anything we would call intelligence can evolve divorced from a high-bandwidth, real-time sensory apparatus, and divorced from a way to directly manipulate its environment." This could be approached from many directions, from teleoperated robotics to VR. "The idea that more intelligence always makes a significant difference. If we look at thinking disciplines where computers are already better than the best humans (chess, heads-up poker, recognizing certain kinds of objects etc.), the differences are small. If the best human's decisions are 99% optimal, say, a computer's may be 99.9% or 99.999% optimal. The point being that a computer can never makes decisions that are better than 100% optimal." This seems a poor argument. I'd argue that greater intelligence introduces qualitative differences. For instance, the capabilities of a human are far beyond those of a dog. The same can be said of dog versus a grasshopper. Even the difference between an Albert Einstein and an average person seems profound. I believe it's quite impossible for even an IQ 200 human to imagine the capabilities of an IQ 1,000 AI. What does a dog think about the reasoning capacity of a human?