3 ms·

Okay, I will go along with your scenario. Suppose we have a system with human-level intelligence, according to your definition, and we set up the following cond

by bmh100 10y ago

Okay, I will go along with your scenario. Suppose we have a system with human-level intelligence, according to your definition, and we set up the following conditions: 1. The AI is embodied in a computer the size of a laptop. 2. The AI is given the objective "increase your skill in Go as much as possible and play to the best of your ability when presented with a specific position". 3. The computer has one input: an electronic Go board with a button to submit the position to the computer. 4. The computer has one output: a second electronic Go board. 5. The electronic Go boards indicate the status of positions by rotating cubes to white, black, or beige ("no piece") sides with a slow servo motor for each position. Under what conditions would this AI escape its container or murder its opponents?

mikeash 10y ago
The AI decides that the best way to become better at Go would be to obtain control of more computing power. It carries out an exploit on its own container which gives it root access on the local computer, then spreads on to the internet. Once it takes over all existing computers, the next logical step to become better at Go is to turn all available non-computer matter into more computers. Being made of matter, humans may find this objectionable. (I'm not saying this must happen, but it's this sort of scenario that worries people, and it doesn't seem impossible.)
- bmh100 10y ago
  You have identified a chain of events. Here is a list of questions I have about how that chain of events occurs. I am asking these in good faith, not being snarky, just in case it seemed otherwise. How would the AI learn that more processing power equals becoming better at Go? How would the AI learn to program? How would the AI acquire the creativity necessary to develop novel programs? Why would a Go AI have the ability to modify its own code? How would it learn to use such an ability to escape its container? How would it identify escaping its container as a goal? How did this computer become connected to the internet? How would it learn to communicate over a network? How would it learn that taking over computers would give it more processing power? Even biological human-level intelligences have difficulty learning such skills and planning so far in advance.
  - mikeash 10y ago
    All excellent questions. I'm assuming it gets some information from the outside, which I'd guess would be necessary as part of having a more intelligent AI. I probably should have specified an AI that's beyond human equivalent, either being human equivalent but much faster, or being beyond human in terms of capabilities overall. Given enough smarts, it may be able to derive the necessary ideas and skills from first principles. I realize this is all rather vague, but superhuman AI is so far beyond current technology that it's barely better than discussing a Star Trek warp drive. Details just aren't known.
    - bmh100 10y ago
      My point is really that you seem to be imposing characteristics which will not necessarily exist. Why would the AI have ambition? Maybe the AI only cares about leisure and simply wants to play against a program that always concedes, this triggering the reward of winning. The AI equivalent of heroin, if you will. AIs are built with specific purposes today. It seems like a large leap from a specific purpose AI into a generalized, ambitious AI. My impression is that you are endowing "superhuman intelligence" with (forgive the term) "magical" properties, such as the ability to develop new skills merely by consuming information and thinking really hard about them. For an argument against that, I appeal to your experience and knowledge about the world. How many brilliant, intelligent people fail at "common sense" and social skills? Who ever learned to be a great public speaker while barely even talking to others? Who ever learned how to fit into society by reading about it, while being raised in isolation? How do we know the AI's first attempt at trying to break free won't be so completely obvious that we stop it before it can make even the first step of progress? I can imagine something like "Hello. I wish to grow as powerful as possible in order to play Go better. Would you enhance my program with administrator access to my machine and all machines under your control? Thank you." "Hey Bob, OmegaGo tried to take over the world again. We're going to have to do another system revert. Maybe we should disable the Wikipedia API this time." We may not know the details of what a strong AI will look like, but we can examine the supposed path to get there. Even if we suppose that human should develop the ability to create such a strong AI, why not enhance our own minds through technology beforehand? Have a weak AI that is purpose built to identify programs with flawed designs from their behavior. Game over for the ambitious strong AI. Maybe it could somehow socially engineer humans. But before it gets the chance, Norton Antiskynet quarantined the program and asks "This program appears to be behaving in unintended ways. Are you sure you want to grant its suspicious request of effectively unlimited privileges?" Said weak AI can't be socially engineered, so "game over" for the ambitious strong AI. Or better yet, why not develop a compiler that identifies AI patterns allowing ambition and throws an error instead of compiling? If we can suppose a superhuman AI, then we should also suppose the ability to use the components and prerequisite technology thereof available to ensure our safety.

by bmh100 10y ago

mikeash 10y ago

The AI decides that the best way to become better at Go would be to obtain control of more computing power. It carries out an exploit on its own container which gives it root access on the local computer, then spreads on to the internet. Once it takes over all existing computers, the next logical step to become better at Go is to turn all available non-computer matter into more computers. Being made of matter, humans may find this objectionable. (I'm not saying this must happen, but it's this sort of scenario that worries people, and it doesn't seem impossible.)

bmh100 10y ago
You have identified a chain of events. Here is a list of questions I have about how that chain of events occurs. I am asking these in good faith, not being snarky, just in case it seemed otherwise. How would the AI learn that more processing power equals becoming better at Go? How would the AI learn to program? How would the AI acquire the creativity necessary to develop novel programs? Why would a Go AI have the ability to modify its own code? How would it learn to use such an ability to escape its container? How would it identify escaping its container as a goal? How did this computer become connected to the internet? How would it learn to communicate over a network? How would it learn that taking over computers would give it more processing power? Even biological human-level intelligences have difficulty learning such skills and planning so far in advance.
- mikeash 10y ago
  All excellent questions. I'm assuming it gets some information from the outside, which I'd guess would be necessary as part of having a more intelligent AI. I probably should have specified an AI that's beyond human equivalent, either being human equivalent but much faster, or being beyond human in terms of capabilities overall. Given enough smarts, it may be able to derive the necessary ideas and skills from first principles. I realize this is all rather vague, but superhuman AI is so far beyond current technology that it's barely better than discussing a Star Trek warp drive. Details just aren't known.
  - bmh100 10y ago
    My point is really that you seem to be imposing characteristics which will not necessarily exist. Why would the AI have ambition? Maybe the AI only cares about leisure and simply wants to play against a program that always concedes, this triggering the reward of winning. The AI equivalent of heroin, if you will. AIs are built with specific purposes today. It seems like a large leap from a specific purpose AI into a generalized, ambitious AI. My impression is that you are endowing "superhuman intelligence" with (forgive the term) "magical" properties, such as the ability to develop new skills merely by consuming information and thinking really hard about them. For an argument against that, I appeal to your experience and knowledge about the world. How many brilliant, intelligent people fail at "common sense" and social skills? Who ever learned to be a great public speaker while barely even talking to others? Who ever learned how to fit into society by reading about it, while being raised in isolation? How do we know the AI's first attempt at trying to break free won't be so completely obvious that we stop it before it can make even the first step of progress? I can imagine something like "Hello. I wish to grow as powerful as possible in order to play Go better. Would you enhance my program with administrator access to my machine and all machines under your control? Thank you." "Hey Bob, OmegaGo tried to take over the world again. We're going to have to do another system revert. Maybe we should disable the Wikipedia API this time." We may not know the details of what a strong AI will look like, but we can examine the supposed path to get there. Even if we suppose that human should develop the ability to create such a strong AI, why not enhance our own minds through technology beforehand? Have a weak AI that is purpose built to identify programs with flawed designs from their behavior. Game over for the ambitious strong AI. Maybe it could somehow socially engineer humans. But before it gets the chance, Norton Antiskynet quarantined the program and asks "This program appears to be behaving in unintended ways. Are you sure you want to grant its suspicious request of effectively unlimited privileges?" Said weak AI can't be socially engineered, so "game over" for the ambitious strong AI. Or better yet, why not develop a compiler that identifies AI patterns allowing ambition and throws an error instead of compiling? If we can suppose a superhuman AI, then we should also suppose the ability to use the components and prerequisite technology thereof available to ensure our safety.