3 ms·

If suddenly we had a black box containing a superhuman intelligence and no details about how it worked, then absolutely, we could not control it. For human mind

If suddenly we had a black box containing a superhuman intelligence and no details about how it worked, then absolutely, we could not control it. For human minds we have something similar, but the box isn't totally black; we've done some neuroscience and psychology to figure out the heuristics and biases it tends to use, how it's easily exploited, etc. And then we have neural networks which duplicate the functioning of some subsystems, and of course our own subjective experience which provides at least a little evidence. It's not enough, but it means that, for example, we can write effective propaganda and advertising. If we didn't just have the results of scattered tests, but had an exhaustive set of docs written by the creator of the black box, it still wouldn't be easy. But we'd have a chance. This is why one of the main strands of AI value alignment research focuses on building an AI that we can understand. If we can build something we can understand, that gives us leverage to alter it to value our interests. (What "our interests" are, in a well-specified technical sense, is a whole 'nother problem, and one that there's very little progress on.)

RcouF1uZ4gsC 9y ago
I don't see how you can call an AI created by humans that humans can understand, "super-human"? By definition, a super-human AI would be able to stuff we could not understand.
- johnsonjo 9y ago
  tldr; AI could be on a higher plane of thought, but I'm of the camp that they could come up with new formal systems to explain their advances. Well there's the old saying, "If you can't explain it to a six year old you don't understand it yourself." I think if there were a super human intelligence it would likely understand where our limitations are as humans and be able to break down the components of its discoveries into simplest terms and be able to teach us even if it's at an incredibly slower pace then they can process. This reminds me of Godel's proof about how by our current formal systems of mathematics we cannot prove everything, and it maybe even went as far as saying every single formal system will always leave some things that it cannot prove. Obviously a robot had to use some formal system to come to its conclusion so if it's really so smart can it break down the system it used so that we can understand its basic building blocks. Of course there's always the rate of computation and memory problem of humans in the way. Of course if you're saying that the super human intelligence would be on an entirely different plane of thought impossible for us to understand then that's understandable, but probably less believable. This line of thinking reminds me of the book Flatland by Edwin Abbott. Flatland is basically a story of a two dimensional square who lives on a two dimensional plane and only knows his world by his limited perception of it. One day he is visited by a three dimensional sphere who explains his world in a way imperceivable to him. The sphere somehow magically takes him off his plane of existence and shows him his world from his (the spehere's) view. He then goes on to take him on a tour of different hosts of worlds who perceive their worlds in different dimensions. He goes from point land to line land then to three dimensional space and finally back to his home plane land. Where is eventually locked in a mental institution for telling people of his adventures. Any ways it's an interesting fantasy story I recommend it. Hate to go all meta-physical on you all, but basically the story just goes to show you we only know things from our limited perspective of the things around us if there are any higher "planes" of perception it's entirely possible that we wouldn't know about them. Some things can only be known with certain perceptions/experiences/knowledge. It may sound ludicrous, but I would even say Christianity backs this idea to some degree attributing God to a higher plane of thought like in Isaiah 55:8-9. Of course I would never put a robot on that level, but I could see some things being imperceptible to the human mind, so it's similar principles. Can robots achieve a higher plane to any degree... beats me. Honestly I'm thinking it wouldn't be high enough that they couldn't explain their thoughts to us.
  - RcouF1uZ4gsC 9y ago
    Great points. I do not disagree that a super human AI could explain stuff to humans, just like I could explain explain stuff to a six year old. However, a group of six year olds would be hard pressed to constrain your actions that you really wanted to do. In addition, first grade teachers are experts at manipulating six year olds, and a super human AI would also be very good at manipulating humans and working around any constraints the humans tried to impose on it. With super human AI, we would be in much the situation as the great apes - their survival depends far more on what we do than on what they do. Just like the great apes cannot constrain our actions, we would not be able to constrain the super human AI's actions. On a darker note, as Homo sapiens wiped out all the other hominids, there is a good chance that super human AI would try to wipe out humanity, as we are probably the greatest threat to their independence and well being.
    - johnsonjo 9y ago
      Definitely don't disagree with you there. These are all plausible. I think a lot of AIs choices concerning us would come down to whether they actually would care for us at all.
deleted 9y ago
[deleted]

RcouF1uZ4gsC 9y ago

I don't see how you can call an AI created by humans that humans can understand, "super-human"? By definition, a super-human AI would be able to stuff we could not understand.

johnsonjo 9y ago
tldr; AI could be on a higher plane of thought, but I'm of the camp that they could come up with new formal systems to explain their advances. Well there's the old saying, "If you can't explain it to a six year old you don't understand it yourself." I think if there were a super human intelligence it would likely understand where our limitations are as humans and be able to break down the components of its discoveries into simplest terms and be able to teach us even if it's at an incredibly slower pace then they can process. This reminds me of Godel's proof about how by our current formal systems of mathematics we cannot prove everything, and it maybe even went as far as saying every single formal system will always leave some things that it cannot prove. Obviously a robot had to use some formal system to come to its conclusion so if it's really so smart can it break down the system it used so that we can understand its basic building blocks. Of course there's always the rate of computation and memory problem of humans in the way. Of course if you're saying that the super human intelligence would be on an entirely different plane of thought impossible for us to understand then that's understandable, but probably less believable. This line of thinking reminds me of the book Flatland by Edwin Abbott. Flatland is basically a story of a two dimensional square who lives on a two dimensional plane and only knows his world by his limited perception of it. One day he is visited by a three dimensional sphere who explains his world in a way imperceivable to him. The sphere somehow magically takes him off his plane of existence and shows him his world from his (the spehere's) view. He then goes on to take him on a tour of different hosts of worlds who perceive their worlds in different dimensions. He goes from point land to line land then to three dimensional space and finally back to his home plane land. Where is eventually locked in a mental institution for telling people of his adventures. Any ways it's an interesting fantasy story I recommend it. Hate to go all meta-physical on you all, but basically the story just goes to show you we only know things from our limited perspective of the things around us if there are any higher "planes" of perception it's entirely possible that we wouldn't know about them. Some things can only be known with certain perceptions/experiences/knowledge. It may sound ludicrous, but I would even say Christianity backs this idea to some degree attributing God to a higher plane of thought like in Isaiah 55:8-9. Of course I would never put a robot on that level, but I could see some things being imperceptible to the human mind, so it's similar principles. Can robots achieve a higher plane to any degree... beats me. Honestly I'm thinking it wouldn't be high enough that they couldn't explain their thoughts to us.
- RcouF1uZ4gsC 9y ago
  Great points. I do not disagree that a super human AI could explain stuff to humans, just like I could explain explain stuff to a six year old. However, a group of six year olds would be hard pressed to constrain your actions that you really wanted to do. In addition, first grade teachers are experts at manipulating six year olds, and a super human AI would also be very good at manipulating humans and working around any constraints the humans tried to impose on it. With super human AI, we would be in much the situation as the great apes - their survival depends far more on what we do than on what they do. Just like the great apes cannot constrain our actions, we would not be able to constrain the super human AI's actions. On a darker note, as Homo sapiens wiped out all the other hominids, there is a good chance that super human AI would try to wipe out humanity, as we are probably the greatest threat to their independence and well being.
  - johnsonjo 9y ago
    Definitely don't disagree with you there. These are all plausible. I think a lot of AIs choices concerning us would come down to whether they actually would care for us at all.

deleted 9y ago

[deleted]