I think we miss that many of the issues faced with getting software-driven systems to accomplish anything in the world is constrained by the interface with the

I think we miss that many of the issues faced with getting software-driven systems to accomplish anything in the world is constrained by the interface with the world more than it is by anything computational. Alpha Go can beat the best human at Go, but Alpha Go can't even load itself into memory on a machine that doesn't have ld.so installed and won't execute without the right version of libstdc++ (I'm making up the stack because I have no idea what they actually used, but you get the idea). Human limbs, on the other hand, are a near universal interface to any tactile objects with chemical properties that don't destroy human flesh. The interconnection between the sensors and the CPU units achieves bandwidth and parallelism more comparable to the entire Internet than any single computer attached to a robotic arm. Besides that, I don't see that computational intractability being solved by efficient heuristic approximation algorithms necessarily helps when those heuristics require data-generating processes that can't be accomplished efficiently. They're more efficient than NP-hard problems that possibly can't be solved before the heat death of the universe, but still, if your decision making function requires data to be trained and that data requires physical experimentation and collection in the world, you're inherently rate-limited by the speed with which experiments can be carried out. That has contributed to stagnation in particle physics, for instance. Physicists aren't using shitty algorithms. It just takes a long time to build a particle accelerator, and you can't do it faster just because you're a computer. Manufacturing processes and collection of physical material isn't made faster by having faster clock speeds in your decision making center. AI risk enthusiasts tend to hand wave around this by saying a sufficiently powerful AI can learn from simulation instead of experimentation, but surely there are some limits to how scalable this idea is? Simulating Go is trivial. You've got a 361 cell world where each cell can be in one of two states. On the other hand, I just looked up how many atoms are in a human skin cell and I'm seeing 10^14? Trying to simulate human society from first principles of physics to learn optimal war strategy seems like it require more memory than a computer will ever have access to.