4 ms·

I don't think Bostrom or Yudkowsky believe this: If one creates a human-level AGI with certain human-friendly goals, and allows it to self-modify freely, the o

I don't think Bostrom or Yudkowsky believe this: If one creates a human-level AGI with certain human-friendly goals, and allows it to self-modify freely, the odds are high that it will eventually self-modify into a condition where it no longer pursues the same goals it started out with I think their idea is a rational agent would have no incentive to change its terminal goals, as changing your utility function has to be of negative utility save for in some extreme edge cases that aren't likely to be relevant. The hard part is giving it the correct terminal goal. I also don't really think Bostrom is anthropomorphising. Basing your AI ideas on Piaget's theory of cognitive development, that seems like anthropomorphism - though likely entirely appropriate when the human mind is what you're attempting to mimic. Interesting article, brings back fond memories of Sl4,reading Ben, Yudkowsky, Robin Hanson, Gwern, and two Satoshi candidates argue was quite fun. I was convinced then that by 2015 nanotech or AI would have eaten the world. I'm a little more uncertain now, but we still have a couple days.

joe_the_user 11y ago
AI may be dangerous. AI can easily be dangerous without getting out of control of all humans. Rather, it could be very dangerous in the hands of a malevolent human - and we've not found a thing more inclined to harm humans and humanity than certain humans. That said, Yudkowsky's reasoning process about the situation seems fundamentally flawed - both him and Bostrom imagine AI operating simply by humans giving discreet, rational commands to AIs and having the AI twist the wording of those commands, malevolently or just incidentally, to give rise to terrible things - the myth of Genie writ large. However, the failure of Gofia, of logical specification AI, shows us that to be intelligent, AI would have to go beyond the level of just taking explicit orders. Such an AI would "know what we mean" in the fashion that a person would. IE, "rational" AI was discredited quite a while yet somehow the "rationalists" have wormed their way into the position of interpreters of future AI. Says more about human psychology than future prospects. This doesn't mean other disasters aren't lurking but I'd say it means that their particular disaster argument is untenable.
- Houshalter 11y ago
  >both him and Bostrom imagine AI operating simply by humans giving discreet, rational commands to AIs and having the AI twist the wording of those commands, malevolently or just incidentally, to give rise to terrible things - the myth of Genie writ large. These are just hypothetical scenarios they write about, to help explain problems with AI. I don't think they actually believe it will happen exactly like that. >However, the failure of Gofia, of logical specification AI, shows us that to be intelligent, AI would have to go beyond the level of just taking explicit orders. Such an AI would "know what we mean" in the fashion that a person would. Any truly intelligent AI would probably be smart enough to understand what we mean with our words. The issue is that there is no reason it would care. In the same way that if an alien showed up and started giving you orders, you wouldn't really care, even if you deciphered his language and figured out what he meant. This is known as the Control Problem. The problem of controlling an AI's motivations so that they actually want to do what we tell them to.
  - joe_the_user 11y ago
    Any truly intelligent AI would probably be smart enough to understand what we mean with our words. The issue is that there is no reason it would care. Computer programs don't care about anything inherently. Assuming people created programs with the ability to "know what we meant", the sensible thing would be to program them to care primarily about "doing what we mean". It seems like more or less an an anthropomorphization-driven illusion to believe that attaining "human intelligence" cause a machine to have all the unpredictable self-interesting-seek behaviors of humans. In the same way that if an alien showed up and started giving you orders, you wouldn't really care, even if you deciphered his language and figured out what he meant. Except I'm not a tool (a thing crafted for a purpose) but a product of an evolution with a combination of often contradictory impulses.
    - Houshalter 11y ago
      >the sensible thing would be to program them to care primarily about "doing what we mean". The problem is no one has any idea how to do this. We can make an AI with desires. For example, we can give it a reward every time you push a button. And then program it to predict what actions will lead to the most reward. That's quite simple, because you can easily keep track of how many times the button has been pressed in the program. But given the opportunity, such an AI would just kill it's human creators and steal the button, and hold it down. It doesn't care about anything but the button. What you wanted doesn't matter at all. How do you make it want to do what you mean. How do you directly measure "obedience" in the same way we can measure a simple button press? >Except I'm not a tool (a thing crafted for a purpose) but a product of an evolution with a combination of often contradictory impulses. To some extent, you are a tool crafted for a specific purpose. That is, evolution crafted you to spread your genes as much as possible. And yet that's probably not what you want. People have children, sure, but they also use birth control, and do things other than trying to make as many copies of themselves as possible.

joe_the_user 11y ago

AI may be dangerous. AI can easily be dangerous without getting out of control of all humans. Rather, it could be very dangerous in the hands of a malevolent human - and we've not found a thing more inclined to harm humans and humanity than certain humans. That said, Yudkowsky's reasoning process about the situation seems fundamentally flawed - both him and Bostrom imagine AI operating simply by humans giving discreet, rational commands to AIs and having the AI twist the wording of those commands, malevolently or just incidentally, to give rise to terrible things - the myth of Genie writ large. However, the failure of Gofia, of logical specification AI, shows us that to be intelligent, AI would have to go beyond the level of just taking explicit orders. Such an AI would "know what we mean" in the fashion that a person would. IE, "rational" AI was discredited quite a while yet somehow the "rationalists" have wormed their way into the position of interpreters of future AI. Says more about human psychology than future prospects. This doesn't mean other disasters aren't lurking but I'd say it means that their particular disaster argument is untenable.

Houshalter 11y ago
>both him and Bostrom imagine AI operating simply by humans giving discreet, rational commands to AIs and having the AI twist the wording of those commands, malevolently or just incidentally, to give rise to terrible things - the myth of Genie writ large. These are just hypothetical scenarios they write about, to help explain problems with AI. I don't think they actually believe it will happen exactly like that. >However, the failure of Gofia, of logical specification AI, shows us that to be intelligent, AI would have to go beyond the level of just taking explicit orders. Such an AI would "know what we mean" in the fashion that a person would. Any truly intelligent AI would probably be smart enough to understand what we mean with our words. The issue is that there is no reason it would care. In the same way that if an alien showed up and started giving you orders, you wouldn't really care, even if you deciphered his language and figured out what he meant. This is known as the Control Problem. The problem of controlling an AI's motivations so that they actually want to do what we tell them to.
- joe_the_user 11y ago
  Any truly intelligent AI would probably be smart enough to understand what we mean with our words. The issue is that there is no reason it would care. Computer programs don't care about anything inherently. Assuming people created programs with the ability to "know what we meant", the sensible thing would be to program them to care primarily about "doing what we mean". It seems like more or less an an anthropomorphization-driven illusion to believe that attaining "human intelligence" cause a machine to have all the unpredictable self-interesting-seek behaviors of humans. In the same way that if an alien showed up and started giving you orders, you wouldn't really care, even if you deciphered his language and figured out what he meant. Except I'm not a tool (a thing crafted for a purpose) but a product of an evolution with a combination of often contradictory impulses.
  - Houshalter 11y ago
    >the sensible thing would be to program them to care primarily about "doing what we mean". The problem is no one has any idea how to do this. We can make an AI with desires. For example, we can give it a reward every time you push a button. And then program it to predict what actions will lead to the most reward. That's quite simple, because you can easily keep track of how many times the button has been pressed in the program. But given the opportunity, such an AI would just kill it's human creators and steal the button, and hold it down. It doesn't care about anything but the button. What you wanted doesn't matter at all. How do you make it want to do what you mean. How do you directly measure "obedience" in the same way we can measure a simple button press? >Except I'm not a tool (a thing crafted for a purpose) but a product of an evolution with a combination of often contradictory impulses. To some extent, you are a tool crafted for a specific purpose. That is, evolution crafted you to spread your genes as much as possible. And yet that's probably not what you want. People have children, sure, but they also use birth control, and do things other than trying to make as many copies of themselves as possible.