3 ms·

>both him and Bostrom imagine AI operating simply by humans giving discreet, rational commands to AIs and having the AI twist the wording of those commands, mal

>both him and Bostrom imagine AI operating simply by humans giving discreet, rational commands to AIs and having the AI twist the wording of those commands, malevolently or just incidentally, to give rise to terrible things - the myth of Genie writ large. These are just hypothetical scenarios they write about, to help explain problems with AI. I don't think they actually believe it will happen exactly like that. >However, the failure of Gofia, of logical specification AI, shows us that to be intelligent, AI would have to go beyond the level of just taking explicit orders. Such an AI would "know what we mean" in the fashion that a person would. Any truly intelligent AI would probably be smart enough to understand what we mean with our words. The issue is that there is no reason it would care. In the same way that if an alien showed up and started giving you orders, you wouldn't really care, even if you deciphered his language and figured out what he meant. This is known as the Control Problem. The problem of controlling an AI's motivations so that they actually want to do what we tell them to.

joe_the_user 11y ago
Any truly intelligent AI would probably be smart enough to understand what we mean with our words. The issue is that there is no reason it would care. Computer programs don't care about anything inherently. Assuming people created programs with the ability to "know what we meant", the sensible thing would be to program them to care primarily about "doing what we mean". It seems like more or less an an anthropomorphization-driven illusion to believe that attaining "human intelligence" cause a machine to have all the unpredictable self-interesting-seek behaviors of humans. In the same way that if an alien showed up and started giving you orders, you wouldn't really care, even if you deciphered his language and figured out what he meant. Except I'm not a tool (a thing crafted for a purpose) but a product of an evolution with a combination of often contradictory impulses.
- Houshalter 11y ago
  >the sensible thing would be to program them to care primarily about "doing what we mean". The problem is no one has any idea how to do this. We can make an AI with desires. For example, we can give it a reward every time you push a button. And then program it to predict what actions will lead to the most reward. That's quite simple, because you can easily keep track of how many times the button has been pressed in the program. But given the opportunity, such an AI would just kill it's human creators and steal the button, and hold it down. It doesn't care about anything but the button. What you wanted doesn't matter at all. How do you make it want to do what you mean. How do you directly measure "obedience" in the same way we can measure a simple button press? >Except I'm not a tool (a thing crafted for a purpose) but a product of an evolution with a combination of often contradictory impulses. To some extent, you are a tool crafted for a specific purpose. That is, evolution crafted you to spread your genes as much as possible. And yet that's probably not what you want. People have children, sure, but they also use birth control, and do things other than trying to make as many copies of themselves as possible.
  - joe_the_user 11y ago
    "We can make an AI with desires. For example, we can give it a reward every time you push a button." I wouldn't call that a "desire" in the fashion we've discussed it. I guess it comes down to fundamental disagreement over how GAIs could be created. I think it's obvious that the creation of a GAI would require a very careful engineering of "understanding" in a broad sense - "knowing what people mean" "caring about X" etc. in order to occur at all. If just complexity and rewards are enough to create a thing that intentionally increases resources and makes longer term plans, then it seems like we are indeed in trouble. But I think that's implausible.
    - Houshalter 11y ago
      Call it a desire/goal/value/utility function/whatever. It's a thing which the AI wants and creates plans to obtain. This isn't that different than how animals and humans work, as far as we know. We get pleasure from different things, and our brains seek actions which lead to those things. Creating an AI is only a matter of coming up with really good prediction algorithms. Algorithms that predict the future reward the AI will get from an action. While this is of course a difficult problem, it's not impossible, and we are making a ton of progress on it. Other models of AI all have similar problems though. At some level you need to program the AI's goals explicitly. Even if that goal is "try to understand what we mean, and do that", you still have to figure out a way of writing that down in code. It's an impossible task.
      - joe_the_user 11y ago
        OK, I think I understand where you coming from. While I don't think a GAI could arise from your scenario - just loops - I can see now that you are making a plausible argument and given that the whole field is extremely uncertain, it is reasonable to be worried by a plausible, problematic scenario.

joe_the_user 11y ago

Any truly intelligent AI would probably be smart enough to understand what we mean with our words. The issue is that there is no reason it would care. Computer programs don't care about anything inherently. Assuming people created programs with the ability to "know what we meant", the sensible thing would be to program them to care primarily about "doing what we mean". It seems like more or less an an anthropomorphization-driven illusion to believe that attaining "human intelligence" cause a machine to have all the unpredictable self-interesting-seek behaviors of humans. In the same way that if an alien showed up and started giving you orders, you wouldn't really care, even if you deciphered his language and figured out what he meant. Except I'm not a tool (a thing crafted for a purpose) but a product of an evolution with a combination of often contradictory impulses.

Houshalter 11y ago
>the sensible thing would be to program them to care primarily about "doing what we mean". The problem is no one has any idea how to do this. We can make an AI with desires. For example, we can give it a reward every time you push a button. And then program it to predict what actions will lead to the most reward. That's quite simple, because you can easily keep track of how many times the button has been pressed in the program. But given the opportunity, such an AI would just kill it's human creators and steal the button, and hold it down. It doesn't care about anything but the button. What you wanted doesn't matter at all. How do you make it want to do what you mean. How do you directly measure "obedience" in the same way we can measure a simple button press? >Except I'm not a tool (a thing crafted for a purpose) but a product of an evolution with a combination of often contradictory impulses. To some extent, you are a tool crafted for a specific purpose. That is, evolution crafted you to spread your genes as much as possible. And yet that's probably not what you want. People have children, sure, but they also use birth control, and do things other than trying to make as many copies of themselves as possible.
- joe_the_user 11y ago
  "We can make an AI with desires. For example, we can give it a reward every time you push a button." I wouldn't call that a "desire" in the fashion we've discussed it. I guess it comes down to fundamental disagreement over how GAIs could be created. I think it's obvious that the creation of a GAI would require a very careful engineering of "understanding" in a broad sense - "knowing what people mean" "caring about X" etc. in order to occur at all. If just complexity and rewards are enough to create a thing that intentionally increases resources and makes longer term plans, then it seems like we are indeed in trouble. But I think that's implausible.
  - Houshalter 11y ago
    Call it a desire/goal/value/utility function/whatever. It's a thing which the AI wants and creates plans to obtain. This isn't that different than how animals and humans work, as far as we know. We get pleasure from different things, and our brains seek actions which lead to those things. Creating an AI is only a matter of coming up with really good prediction algorithms. Algorithms that predict the future reward the AI will get from an action. While this is of course a difficult problem, it's not impossible, and we are making a ton of progress on it. Other models of AI all have similar problems though. At some level you need to program the AI's goals explicitly. Even if that goal is "try to understand what we mean, and do that", you still have to figure out a way of writing that down in code. It's an impossible task.
    - joe_the_user 11y ago
      OK, I think I understand where you coming from. While I don't think a GAI could arise from your scenario - just loops - I can see now that you are making a plausible argument and given that the whole field is extremely uncertain, it is reasonable to be worried by a plausible, problematic scenario.