4 ms·

> So good luck, humanity, in keeping AI under control, should it be developed. This seems to anthropomorphize AI. Why should we expect AI in general to be diff

by bmh100 10y ago

> So good luck, humanity, in keeping AI under control, should it be developed. This seems to anthropomorphize AI. Why should we expect AI in general to be difficult to control? AlphaGo, while not a strong AI, seems quite harmless.

mikeash 10y ago
AlphaGo is still dumber than a potted plant, the tiny amount of intelligence it has is just all focused on Go, which is enough to beat the masters at that one specific task. Now imagine you have a system with human-level intelligence and you tell it to beat the masters. Rather than becoming really good at Go, it might discover how to escape its container and sabotage the match, murder its opponents, etc.
- bmh100 10y ago
  Okay, I will go along with your scenario. Suppose we have a system with human-level intelligence, according to your definition, and we set up the following conditions: 1. The AI is embodied in a computer the size of a laptop. 2. The AI is given the objective "increase your skill in Go as much as possible and play to the best of your ability when presented with a specific position". 3. The computer has one input: an electronic Go board with a button to submit the position to the computer. 4. The computer has one output: a second electronic Go board. 5. The electronic Go boards indicate the status of positions by rotating cubes to white, black, or beige ("no piece") sides with a slow servo motor for each position. Under what conditions would this AI escape its container or murder its opponents?
  - mikeash 10y ago
    The AI decides that the best way to become better at Go would be to obtain control of more computing power. It carries out an exploit on its own container which gives it root access on the local computer, then spreads on to the internet. Once it takes over all existing computers, the next logical step to become better at Go is to turn all available non-computer matter into more computers. Being made of matter, humans may find this objectionable. (I'm not saying this must happen, but it's this sort of scenario that worries people, and it doesn't seem impossible.)
    - bmh100 10y ago
      You have identified a chain of events. Here is a list of questions I have about how that chain of events occurs. I am asking these in good faith, not being snarky, just in case it seemed otherwise. How would the AI learn that more processing power equals becoming better at Go? How would the AI learn to program? How would the AI acquire the creativity necessary to develop novel programs? Why would a Go AI have the ability to modify its own code? How would it learn to use such an ability to escape its container? How would it identify escaping its container as a goal? How did this computer become connected to the internet? How would it learn to communicate over a network? How would it learn that taking over computers would give it more processing power? Even biological human-level intelligences have difficulty learning such skills and planning so far in advance.
      - mikeash 10y ago
        All excellent questions. I'm assuming it gets some information from the outside, which I'd guess would be necessary as part of having a more intelligent AI. I probably should have specified an AI that's beyond human equivalent, either being human equivalent but much faster, or being beyond human in terms of capabilities overall. Given enough smarts, it may be able to derive the necessary ideas and skills from first principles. I realize this is all rather vague, but superhuman AI is so far beyond current technology that it's barely better than discussing a Star Trek warp drive. Details just aren't known.
        bmh100 10y ago
        My point is really that you seem to be imposing characteristics which will not necessarily exist. Why would the AI have ambition? Maybe the AI only cares about leisure and simply wants to play against a program that always concedes, this triggering the reward of winning. The AI equivalent of heroin, if you will. AIs are built with specific purposes today. It seems like a large leap from a specific purpose AI into a generalized, ambitious AI. My impression is that you are endowing "superhuman intelligence" with (forgive the term) "magical" properties, such as the ability to develop new skills merely by consuming information and thinking really hard about them. For an argument against that, I appeal to your experience and knowledge about the world. How many brilliant, intelligent people fail at "common sense" and social skills? Who ever learned to be a great public speaker while barely even talking to others? Who ever learned how to fit into society by reading about it, while being raised in isolation? How do we know the AI's first attempt at trying to break free won't be so completely obvious that we stop it before it can make even the first step of progress? I can imagine something like "Hello. I wish to grow as powerful as possible in order to play Go better. Would you enhance my program with administrator access to my machine and all machines under your control? Thank you." "Hey Bob, OmegaGo tried to take over the world again. We're going to have to do another system revert. Maybe we should disable the Wikipedia API this time." We may not know the details of what a strong AI will look like, but we can examine the supposed path to get there. Even if we suppose that human should develop the ability to create such a strong AI, why not enhance our own minds through technology beforehand? Have a weak AI that is purpose built to identify programs with flawed designs from their behavior. Game over for the ambitious strong AI. Maybe it could somehow socially engineer humans. But before it gets the chance, Norton Antiskynet quarantined the program and asks "This program appears to be behaving in unintended ways. Are you sure you want to grant its suspicious request of effectively unlimited privileges?" Said weak AI can't be socially engineered, so "game over" for the ambitious strong AI. Or better yet, why not develop a compiler that identifies AI patterns allowing ambition and throws an error instead of compiling? If we can suppose a superhuman AI, then we should also suppose the ability to use the components and prerequisite technology thereof available to ensure our safety.
mannykannot 10y ago
The comment is somewhat tongue-in-cheek, but I think the concern is valid. It probably is anthopomophizing, but that does not mean that it is automatically wrong. It might be game over by the time an example is apparent.
- bmh100 10y ago
  For an AI to become "uncontrolled" requires quite a lot of assumptions. I don't dismiss the concern as invalid, but I don't think the assumptions implied by the question are necessarily plausible either.
  - mannykannot 10y ago
    The argument follows from the first thing I wrote in this thread. Our understanding of complex systems is through abstractions, but abstractions (by definition) hide the details. Just as attackers can exploit details that are 'below the radar' of an abstract view, AI with a mind of its own (which it would have, by definition) could probably do things that escape our attention, including social engineering. That is not necessarily bad, but it might be out of our control. To be clear: I am considering real AI, not that which is sometimes passed off as AI these days.
    - bmh100 10y ago
      How would a strong AI learn to influence humans? How would it come to the conclusion that influencing humans would achieve its goal? How would it develop the ability to socially engineer humans in useful ways before the people monitoring it would notice? How would a strong AI develop the "motivation", for lack of a better term, to influence humans to remove constraints on its behavior? Why would this strong AI even decide that removing constraints would be its "goal" in the first place?

by bmh100 10y ago

mikeash 10y ago

AlphaGo is still dumber than a potted plant, the tiny amount of intelligence it has is just all focused on Go, which is enough to beat the masters at that one specific task. Now imagine you have a system with human-level intelligence and you tell it to beat the masters. Rather than becoming really good at Go, it might discover how to escape its container and sabotage the match, murder its opponents, etc.

bmh100 10y ago
Okay, I will go along with your scenario. Suppose we have a system with human-level intelligence, according to your definition, and we set up the following conditions: 1. The AI is embodied in a computer the size of a laptop. 2. The AI is given the objective "increase your skill in Go as much as possible and play to the best of your ability when presented with a specific position". 3. The computer has one input: an electronic Go board with a button to submit the position to the computer. 4. The computer has one output: a second electronic Go board. 5. The electronic Go boards indicate the status of positions by rotating cubes to white, black, or beige ("no piece") sides with a slow servo motor for each position. Under what conditions would this AI escape its container or murder its opponents?
- mikeash 10y ago
  The AI decides that the best way to become better at Go would be to obtain control of more computing power. It carries out an exploit on its own container which gives it root access on the local computer, then spreads on to the internet. Once it takes over all existing computers, the next logical step to become better at Go is to turn all available non-computer matter into more computers. Being made of matter, humans may find this objectionable. (I'm not saying this must happen, but it's this sort of scenario that worries people, and it doesn't seem impossible.)
  - bmh100 10y ago
    You have identified a chain of events. Here is a list of questions I have about how that chain of events occurs. I am asking these in good faith, not being snarky, just in case it seemed otherwise. How would the AI learn that more processing power equals becoming better at Go? How would the AI learn to program? How would the AI acquire the creativity necessary to develop novel programs? Why would a Go AI have the ability to modify its own code? How would it learn to use such an ability to escape its container? How would it identify escaping its container as a goal? How did this computer become connected to the internet? How would it learn to communicate over a network? How would it learn that taking over computers would give it more processing power? Even biological human-level intelligences have difficulty learning such skills and planning so far in advance.
    - mikeash 10y ago
      All excellent questions. I'm assuming it gets some information from the outside, which I'd guess would be necessary as part of having a more intelligent AI. I probably should have specified an AI that's beyond human equivalent, either being human equivalent but much faster, or being beyond human in terms of capabilities overall. Given enough smarts, it may be able to derive the necessary ideas and skills from first principles. I realize this is all rather vague, but superhuman AI is so far beyond current technology that it's barely better than discussing a Star Trek warp drive. Details just aren't known.
      - bmh100 10y ago
        My point is really that you seem to be imposing characteristics which will not necessarily exist. Why would the AI have ambition? Maybe the AI only cares about leisure and simply wants to play against a program that always concedes, this triggering the reward of winning. The AI equivalent of heroin, if you will. AIs are built with specific purposes today. It seems like a large leap from a specific purpose AI into a generalized, ambitious AI. My impression is that you are endowing "superhuman intelligence" with (forgive the term) "magical" properties, such as the ability to develop new skills merely by consuming information and thinking really hard about them. For an argument against that, I appeal to your experience and knowledge about the world. How many brilliant, intelligent people fail at "common sense" and social skills? Who ever learned to be a great public speaker while barely even talking to others? Who ever learned how to fit into society by reading about it, while being raised in isolation? How do we know the AI's first attempt at trying to break free won't be so completely obvious that we stop it before it can make even the first step of progress? I can imagine something like "Hello. I wish to grow as powerful as possible in order to play Go better. Would you enhance my program with administrator access to my machine and all machines under your control? Thank you." "Hey Bob, OmegaGo tried to take over the world again. We're going to have to do another system revert. Maybe we should disable the Wikipedia API this time." We may not know the details of what a strong AI will look like, but we can examine the supposed path to get there. Even if we suppose that human should develop the ability to create such a strong AI, why not enhance our own minds through technology beforehand? Have a weak AI that is purpose built to identify programs with flawed designs from their behavior. Game over for the ambitious strong AI. Maybe it could somehow socially engineer humans. But before it gets the chance, Norton Antiskynet quarantined the program and asks "This program appears to be behaving in unintended ways. Are you sure you want to grant its suspicious request of effectively unlimited privileges?" Said weak AI can't be socially engineered, so "game over" for the ambitious strong AI. Or better yet, why not develop a compiler that identifies AI patterns allowing ambition and throws an error instead of compiling? If we can suppose a superhuman AI, then we should also suppose the ability to use the components and prerequisite technology thereof available to ensure our safety.

mannykannot 10y ago

The comment is somewhat tongue-in-cheek, but I think the concern is valid. It probably is anthopomophizing, but that does not mean that it is automatically wrong. It might be game over by the time an example is apparent.

bmh100 10y ago
For an AI to become "uncontrolled" requires quite a lot of assumptions. I don't dismiss the concern as invalid, but I don't think the assumptions implied by the question are necessarily plausible either.
- mannykannot 10y ago
  The argument follows from the first thing I wrote in this thread. Our understanding of complex systems is through abstractions, but abstractions (by definition) hide the details. Just as attackers can exploit details that are 'below the radar' of an abstract view, AI with a mind of its own (which it would have, by definition) could probably do things that escape our attention, including social engineering. That is not necessarily bad, but it might be out of our control. To be clear: I am considering real AI, not that which is sometimes passed off as AI these days.
  - bmh100 10y ago
    How would a strong AI learn to influence humans? How would it come to the conclusion that influencing humans would achieve its goal? How would it develop the ability to socially engineer humans in useful ways before the people monitoring it would notice? How would a strong AI develop the "motivation", for lack of a better term, to influence humans to remove constraints on its behavior? Why would this strong AI even decide that removing constraints would be its "goal" in the first place?