4 ms·

> We can not simply look in our brains/minds and read our source code, nor easily tweak it to see what would happen without great risks to our lives. A computer

> We can not simply look in our brains/minds and read our source code, nor easily tweak it to see what would happen without great risks to our lives. A computer could. That is not a given. The neural network type of AI, which is so far the most promising avenue, is about as opaque as a human brain. It is far from obvious that even a superintelligent AI could understand what its own brain does, let alone modify it in a way that has positive implications for itself. Hell, it's not even a given the AI could read its "source code" either. We often say we shouldn't anthropomorphize AI, but we shouldn't how-current-computers-work-ize it either. Future AI might not actually run on computers that have a CPU, a GPU and RAM that can be read freely. The ability to read or copy source code isn't cost-less, and several considerations may lead AI hardware designers to axe it. One consideration would be leveraging analog processes, which might be orders of magnitude more efficient than digital ones, at the cost of exactness and reproducibility. Another would be to make circuits denser by removing the clock and all global communication architecture, meaning that artificial neurons would only be connected to each other, and there would be no way to read their state externally without a physical probe.

pmoriarty 9y ago
"That is not a given. The neural network type of AI, which is so far the most promising avenue, is about as opaque as a human brain. It is far from obvious that even a superintelligent AI could understand what its own brain does, let alone modify it in a way that has positive implications for itself." What it could do is experiment. It could rewire the network, add or take away nodes, change functions, etc. This could potentially be done and evaluated by it at an incredibly fast rate compared to humans. Regarding the difficulties of it reading its own source code, even the hypothetical ones you cite are many orders of magnitude smaller than those faced by humans in reading their own, as long as the AI is not itself running on a biological substrate.
- breuleux 9y ago
  > What it could do is experiment. It could rewire the network, add or take away nodes, change functions, etc. This could potentially be done and evaluated by it at an incredibly fast rate compared to humans. "Potentially," but let's not get ahead of ourselves here. There are two ways a brain could be "rewired." It could first be rewired by a local process, that is to say, a group of neurons decide how they are to be connected to each other, without influence from faraway neurons except through the normal propagation of signals through the brain. That's how biological brains do it. Or it could be rewired by a global process, a coordinator that can look at the big picture and make minute changes. That would be your suggestion. My contention is that the latter method involves a lot of hardware overhead: you basically need a global network connected to each and every neuron which can probe their state in addition to the local wiring that lets neurons communicate with their neighbours. You need space for this bus, so neurons need to be further apart than they otherwise would, which means that signals have to travel further and the brain will think slower. Nor is "rewire the network, add or take away nodes, change functions, etc." necessarily an effective strategy. First you need to identify what to change, which is like finding a needle inside a haystack, then you need to figure out in what way to change it, which is also difficult, then you need to test whether it had the intended effect, and more importantly, whether there were harmful side-effects. Whether humans can do it or not is not relevant: What is relevant is whether this process is efficient enough to beat the baseline local learning method. It is not clear that it would be. > Regarding the difficulties of it reading its own source code, even the hypothetical ones you cite are many orders of magnitude smaller than those faced by humans in reading their own, as long as the AI is not itself running on a biological substrate. What makes you think these are smaller difficulties? On the contrary, if you imagine that the AI is built as a very dense, solid 3D circuit, and you need to read the value of a neuron at the center, you might have a much harder time jamming a probe in there than you would injecting one in a squishy human brain. You would need to build it in such a fashion that it can be probed easily, but that may require making the circuit twice as big and therefore slower. Furthermore, in the presence of local update rules, which is likely to be the case, your "source code" is changing all the time, even as you read it, so your self-knowledge is constantly out of date. There is a synchronization issue here.
michaelmrose 9y ago
The ability to make a billion identical somethings would seem to be more valuable than any possible improvement you could make. The other trade offs you are imagining seem to be worse than any possible gain.
- breuleux 9y ago
  Straight copying is not a robust way to scale: any vulnerability found in a single copy will be found in all of them, so you end up with a very brittle system. As far as copying goes, you want a smarter scheme than that, where e.g. only truly useful information is transferred and the rest is rebuilt around that. The breadth of easy information access current computers allow is good for the really naive approaches we currently use, but probably overkill relative to more efficient schemes.

pmoriarty 9y ago

"That is not a given. The neural network type of AI, which is so far the most promising avenue, is about as opaque as a human brain. It is far from obvious that even a superintelligent AI could understand what its own brain does, let alone modify it in a way that has positive implications for itself." What it could do is experiment. It could rewire the network, add or take away nodes, change functions, etc. This could potentially be done and evaluated by it at an incredibly fast rate compared to humans. Regarding the difficulties of it reading its own source code, even the hypothetical ones you cite are many orders of magnitude smaller than those faced by humans in reading their own, as long as the AI is not itself running on a biological substrate.

breuleux 9y ago
> What it could do is experiment. It could rewire the network, add or take away nodes, change functions, etc. This could potentially be done and evaluated by it at an incredibly fast rate compared to humans. "Potentially," but let's not get ahead of ourselves here. There are two ways a brain could be "rewired." It could first be rewired by a local process, that is to say, a group of neurons decide how they are to be connected to each other, without influence from faraway neurons except through the normal propagation of signals through the brain. That's how biological brains do it. Or it could be rewired by a global process, a coordinator that can look at the big picture and make minute changes. That would be your suggestion. My contention is that the latter method involves a lot of hardware overhead: you basically need a global network connected to each and every neuron which can probe their state in addition to the local wiring that lets neurons communicate with their neighbours. You need space for this bus, so neurons need to be further apart than they otherwise would, which means that signals have to travel further and the brain will think slower. Nor is "rewire the network, add or take away nodes, change functions, etc." necessarily an effective strategy. First you need to identify what to change, which is like finding a needle inside a haystack, then you need to figure out in what way to change it, which is also difficult, then you need to test whether it had the intended effect, and more importantly, whether there were harmful side-effects. Whether humans can do it or not is not relevant: What is relevant is whether this process is efficient enough to beat the baseline local learning method. It is not clear that it would be. > Regarding the difficulties of it reading its own source code, even the hypothetical ones you cite are many orders of magnitude smaller than those faced by humans in reading their own, as long as the AI is not itself running on a biological substrate. What makes you think these are smaller difficulties? On the contrary, if you imagine that the AI is built as a very dense, solid 3D circuit, and you need to read the value of a neuron at the center, you might have a much harder time jamming a probe in there than you would injecting one in a squishy human brain. You would need to build it in such a fashion that it can be probed easily, but that may require making the circuit twice as big and therefore slower. Furthermore, in the presence of local update rules, which is likely to be the case, your "source code" is changing all the time, even as you read it, so your self-knowledge is constantly out of date. There is a synchronization issue here.

michaelmrose 9y ago

The ability to make a billion identical somethings would seem to be more valuable than any possible improvement you could make. The other trade offs you are imagining seem to be worse than any possible gain.

breuleux 9y ago
Straight copying is not a robust way to scale: any vulnerability found in a single copy will be found in all of them, so you end up with a very brittle system. As far as copying goes, you want a smarter scheme than that, where e.g. only truly useful information is transferred and the rest is rebuilt around that. The breadth of easy information access current computers allow is good for the really naive approaches we currently use, but probably overkill relative to more efficient schemes.