4 ms·
"An individual brain cannot implement recursive intelligence augmentation. An overwhelming amount of evidence points to this simple fact: a single human brain,
by pmoriarty 9y ago
"An individual brain cannot implement recursive intelligence augmentation. An overwhelming amount of evidence points to this simple fact: a single human brain, on its own, is not capable of designing a greater intelligence than itself. This is a purely empirical statement: out of billions of human brains that have come and gone, none has done so. Clearly, the intelligence of a single human, over a single lifetime, cannot design intelligence, or else, over billions of trials, it would have already occurred."
The same argument could have been about human flight (or any other invention). Billions of human beings over millions of years had come and gone, and yet none had been able to fly. Until they did.
There's also a point to be made about humans not having adequate tools for introspection or self-modification. We can not simply look in our brains/minds and read our source code, nor easily tweak it to see what would happen without great risks to our lives. A computer could. Furthermore, a computer could potentially run billions or trillions such experiments in the course of a single human lifetime.
"no human, nor any intelligent entity that we know of, has ever developed anything smarter than itself."
What about humans simply having smarter children? In many ways, humanity's creation of AI could be seen as analogous to giving birth to a child that's smarter than its parents.
"Beyond contextual hard limits, even if one part of a system has the ability to recursively self-improve, other parts of the system will inevitably start acting as
bottlenecks. Antagonistic processes will arise in response to recursive self-improvement and squash it ... a brain with smarter parts will have more trouble coordinating
them ... Exponential progress, meet exponential friction."
The problem is the author is looking at human-level problems with human-level intelligence. A superintelligence may have no such problems, or be able to easily resolve them. Once over the initial speed bump of creating such an intelligence it could be smooth sailing from then on. There's no way to tell, really, what its limits will be until it exists. Besides, even if it can't exponentially increase its own intelligence forever, even a relatively slight increase of intelligence over humanity could be a massive gamechanger.
- breuleux 9y ago> We can not simply look in our brains/minds and read our source code, nor easily tweak it to see what would happen without great risks to our lives. A computer could. That is not a given. The neural network type of AI, which is so far the most promising avenue, is about as opaque as a human brain. It is far from obvious that even a superintelligent AI could understand what its own brain does, let alone modify it in a way that has positive implications for itself. Hell, it's not even a given the AI could read its "source code" either. We often say we shouldn't anthropomorphize AI, but we shouldn't how-current-computers-work-ize it either. Future AI might not actually run on computers that have a CPU, a GPU and RAM that can be read freely. The ability to read or copy source code isn't cost-less, and several considerations may lead AI hardware designers to axe it. One consideration would be leveraging analog processes, which might be orders of magnitude more efficient than digital ones, at the cost of exactness and reproducibility. Another would be to make circuits denser by removing the clock and all global communication architecture, meaning that artificial neurons would only be connected to each other, and there would be no way to read their state externally without a physical probe.
- pmoriarty 9y ago"That is not a given. The neural network type of AI, which is so far the most promising avenue, is about as opaque as a human brain. It is far from obvious that even a superintelligent AI could understand what its own brain does, let alone modify it in a way that has positive implications for itself." What it could do is experiment. It could rewire the network, add or take away nodes, change functions, etc. This could potentially be done and evaluated by it at an incredibly fast rate compared to humans. Regarding the difficulties of it reading its own source code, even the hypothetical ones you cite are many orders of magnitude smaller than those faced by humans in reading their own, as long as the AI is not itself running on a biological substrate.
- breuleux 9y ago> What it could do is experiment. It could rewire the network, add or take away nodes, change functions, etc. This could potentially be done and evaluated by it at an incredibly fast rate compared to humans. "Potentially," but let's not get ahead of ourselves here. There are two ways a brain could be "rewired." It could first be rewired by a local process, that is to say, a group of neurons decide how they are to be connected to each other, without influence from faraway neurons except through the normal propagation of signals through the brain. That's how biological brains do it. Or it could be rewired by a global process, a coordinator that can look at the big picture and make minute changes. That would be your suggestion. My contention is that the latter method involves a lot of hardware overhead: you basically need a global network connected to each and every neuron which can probe their state in addition to the local wiring that lets neurons communicate with their neighbours. You need space for this bus, so neurons need to be further apart than they otherwise would, which means that signals have to travel further and the brain will think slower. Nor is "rewire the network, add or take away nodes, change functions, etc." necessarily an effective strategy. First you need to identify what to change, which is like finding a needle inside a haystack, then you need to figure out in what way to change it, which is also difficult, then you need to test whether it had the intended effect, and more importantly, whether there were harmful side-effects. Whether humans can do it or not is not relevant: What is relevant is whether this process is efficient enough to beat the baseline local learning method. It is not clear that it would be. > Regarding the difficulties of it reading its own source code, even the hypothetical ones you cite are many orders of magnitude smaller than those faced by humans in reading their own, as long as the AI is not itself running on a biological substrate. What makes you think these are smaller difficulties? On the contrary, if you imagine that the AI is built as a very dense, solid 3D circuit, and you need to read the value of a neuron at the center, you might have a much harder time jamming a probe in there than you would injecting one in a squishy human brain. You would need to build it in such a fashion that it can be probed easily, but that may require making the circuit twice as big and therefore slower. Furthermore, in the presence of local update rules, which is likely to be the case, your "source code" is changing all the time, even as you read it, so your self-knowledge is constantly out of date. There is a synchronization issue here.