5 ms·

The funny thing is that this "computers do what you say, not what you mean" comes directly from their lack of intelligence. So it's kind of strange that we tal

by Udik 11y ago

The funny thing is that this "computers do what you say, not what you mean" comes directly from their lack of intelligence. So it's kind of strange that we talk about the threats of superintelligence brought along by the fact that, fundamentally, a machine is stupid. Am I the only one to see a slight contradiction there?

Strilanc 11y ago
Goals are orthogonal to intelligence. The fact that the AI understands what you want won't motivate it to change what it's optimizing. It's not being dumb, it's being literal. You asked it to make lots of paperclips, tossing you into an incinerator as fuel slightly increases the expected number of paper clips in the universe, so into the incinerator you go. Your complaints that you didn't mean that many paperclips are too little, too late. It's a paperclip-maximizer, not a complaint-minimizer. Choosing the goal for a superintelligent AI a goal is like choosing your wish for a monkey's paw[1][2]. You come up with some clever idea, like "make me happy" or "find out what makes me happy, then do that", but the process of mechanizing that goal introduces some weird corner case strategy that horrifies you while doing really well on the stated objective (e.g. wire-heading you, or disassembling you to do a really thorough analysis before moving on to step 2). 1: https://en.wikipedia.org/wiki/The_Monkey's_Paw https://en.wikipedia.org/wiki/The_Monkey's_Paw 2: http://lesswrong.com/lw/ld/the_hidden_complexity_of_wishes/ http://lesswrong.com/lw/ld/the_hidden_complexity_of_wishes/
- snowwrestler 11y ago
  This reads to me like begging the question, by assuming the existence of a "superintelligent AI" without addressing how a goal-optimizing machine becomes a superintelligent AI in the first place. The exercise of fearing future AIs seems like the South Park underpants gnomes: 1. Work on goal-optimizing machinery. 2. ?? 3. Fear superintelligent AI. Or maybe it's like the courtroom scene in A Few Good Men: > If you ordered that Santiago wasn't to be touched, -- and your orders are always followed, -- then why was Santiago in danger? If a paperclip AI is so dedicated to the order to produce paperclips, why wouldn't it be just as dedicated to any other order? Like "don't throw me in that incinerator!"
  - Strilanc 11y ago
    > assuming the existence of a "superintelligent AI" without addressing how a goal-optimizing machine becomes a superintelligent AI I'm just talking about the fallout if one did exist, saw ways to achieve goals that you didn't foresee, and did exactly what you asked it to do. I have no idea how the progression from better-than-humans-in-specific-cases to significantly-better-than-humans-at-planning-and-executing-in-the-real-world will play out. It's not relevant to what I'm claiming. > why wouldn't it be just as dedicated to any other order? It would be just as dedicated to those other orders. The problem is that we don't know how to write the right ones. "Don't throw me into that incinerator" is straightforward, but there's a billion ways for the AI to do horrible things. (A super-optimizer does horrible things by default because maximizing a function usually involves pushing variables to extreme values.) Listing all the ways to be horrible is hopeless. You need to communicate the general concept of not creating a dystopia. Which is safely-wishing-on-monkey's-paw hard.
  - astrofinch 11y ago
    Part 2 is when the AI reaches the point where it's smarter than it creators, then starts improving its own code and bootstraps its way to superintelligent. This idea is referred to as "the intelligence explosion" https://wiki.lesswrong.com/wiki/Intelligence_explosion https://wiki.lesswrong.com/wiki/Intelligence_explosion >If a paperclip AI is so dedicated to the order to produce paperclips, why wouldn't it be just as dedicated to any other order? Like "don't throw me in that incinerator!" The paperclipper scenario is meant to indicate that even a goal which seems benign could have extremely bad implications if pursued by a superintelligence. People concerned with AI risk typically argue that of the universe of possible goals that could be given to an AI, the vast majority of goals in that universe are functionally equivalent to papperclipping. For example, an AI could be programmed to maximize the number of happy people, but without a sufficiently precise specification of what "happy people" means, this could result in something like manufacturing lots of tiny smiley faces. An AI given that order could avoid throwing you in an incinerator and instead throw you in to the thing that's closest to being an incinerator without technically qualifying as an incinerator. Etc.
    - snowwrestler 11y ago
- Retric 11y ago
  I would suggest that a computer is not 'super intelligent' until it can modify it's goals. Further, maximizing paperclips in the long term may not involve building any paperclips for a very long time. https://what-if.xkcd.com/4/ https://what-if.xkcd.com/4/
  - Strilanc 11y ago
    By "super-intelligent" I meant "surprisingly good at achieving specified goals in real life". A super-optimizer. An optimizer that modifies its goals is bad at achieving specified goals, so if that's what you had in mind then we're talking about different things.
    - Retric 11y ago
      We don't call people geniuses because there really good at following orders. Further, a Virus may be extremely capable of achieving specific goals in real life, but that's hardly intelligence. So, powerful but dumb optimizers might be a risk, but super intelligent AI is a different kind of risk. IMO, think cthulhu not HAL 9000. Science fiction thinks in terms of narrative causality, but AI is likely to have goals we really don't understand. EX: Maximizing the number of people that say Zulu on black Friday without anyone noticing that something odd is going on.
      - astrofinch 11y ago
        >We don't call people geniuses because there really good at following orders. If I order someone to prove whether P is equal to NP, and a day later they come back to me with a valid proof, solving a decades-long major open problem in computer science, I would call that person a genius. >EX: Maximizing the number of people that say Zulu on black Friday without anyone noticing that something odd is going on. Computers do what you say, not what you mean, so an AGI's goal would likely be some bastardized version of the intentions of the person who programmed it. Similar to how if you write a 10K line program without testing it, then run it for the first time, it will almost certainly not do what you intended it to do, but rather some bastardized version of what you intended it to do (because there will be bugs to work out).
Aron 11y ago
I think you've summarized [one of] Ben Goertzel's beliefs regarding unfriendly AI.
hosh 11y ago
Let's say we create an AI that can think for itself. There's a fear I think, that lurks in people's subconscious that ... what if the AIs, upon their own initiative, decide that humans are wasteful, inefficient beings that should be replaced? I think that comes from a guilt shared by a lot of folks, even if it never reaches the surface. Another side is, suppose an AI can think for itself and it thinks better than humans. Upon its own initiative, decides that humans are stupid and wasteful, but there is room to teach and and nurture. In either case, I think that speaks less of AIs and more about human nature and what we feel about ourselves, don't you think?
astrofinch 11y ago
"Yes, the UFAI will be able to solve Friendliness Theory. But if we haven't already solved it on our own power, we can't pinpoint Friendliness in advance, out of the space of utility functions. And if we can't pinpoint it with enough detail to draw a road map to it and it alone, we can't program the AI to care about conforming itself with that particular idiosyncratic algorithm." http://lesswrong.com/lw/igf/the_genie_knows_but_doesnt_care/ http://lesswrong.com/lw/igf/the_genie_knows_but_doesnt_care/ Let me put it another way: Humans are a result of evolution. We know that evolution created us to have as many descendants as possible. But most of us don't care, and we use technologies like condoms and birth control to cut down on the number of descendants we have. Adding more intelligence to humans helps us understand evolution in greater detail, but it does nothing to change our actual goals.

by Udik 11y ago

Strilanc 11y ago

Goals are orthogonal to intelligence. The fact that the AI understands what you want won't motivate it to change what it's optimizing. It's not being dumb, it's being literal. You asked it to make lots of paperclips, tossing you into an incinerator as fuel slightly increases the expected number of paper clips in the universe, so into the incinerator you go. Your complaints that you didn't mean that many paperclips are too little, too late. It's a paperclip-maximizer, not a complaint-minimizer. Choosing the goal for a superintelligent AI a goal is like choosing your wish for a monkey's paw[1][2]. You come up with some clever idea, like "make me happy" or "find out what makes me happy, then do that", but the process of mechanizing that goal introduces some weird corner case strategy that horrifies you while doing really well on the stated objective (e.g. wire-heading you, or disassembling you to do a really thorough analysis before moving on to step 2). 1: https://en.wikipedia.org/wiki/The_Monkey's_Paw https://en.wikipedia.org/wiki/The_Monkey's_Paw 2: http://lesswrong.com/lw/ld/the_hidden_complexity_of_wishes/ http://lesswrong.com/lw/ld/the_hidden_complexity_of_wishes/

snowwrestler 11y ago
This reads to me like begging the question, by assuming the existence of a "superintelligent AI" without addressing how a goal-optimizing machine becomes a superintelligent AI in the first place. The exercise of fearing future AIs seems like the South Park underpants gnomes: 1. Work on goal-optimizing machinery. 2. ?? 3. Fear superintelligent AI. Or maybe it's like the courtroom scene in A Few Good Men: > If you ordered that Santiago wasn't to be touched, -- and your orders are always followed, -- then why was Santiago in danger? If a paperclip AI is so dedicated to the order to produce paperclips, why wouldn't it be just as dedicated to any other order? Like "don't throw me in that incinerator!"
- Strilanc 11y ago
  > assuming the existence of a "superintelligent AI" without addressing how a goal-optimizing machine becomes a superintelligent AI I'm just talking about the fallout if one did exist, saw ways to achieve goals that you didn't foresee, and did exactly what you asked it to do. I have no idea how the progression from better-than-humans-in-specific-cases to significantly-better-than-humans-at-planning-and-executing-in-the-real-world will play out. It's not relevant to what I'm claiming. > why wouldn't it be just as dedicated to any other order? It would be just as dedicated to those other orders. The problem is that we don't know how to write the right ones. "Don't throw me into that incinerator" is straightforward, but there's a billion ways for the AI to do horrible things. (A super-optimizer does horrible things by default because maximizing a function usually involves pushing variables to extreme values.) Listing all the ways to be horrible is hopeless. You need to communicate the general concept of not creating a dystopia. Which is safely-wishing-on-monkey's-paw hard.
- astrofinch 11y ago
  Part 2 is when the AI reaches the point where it's smarter than it creators, then starts improving its own code and bootstraps its way to superintelligent. This idea is referred to as "the intelligence explosion" https://wiki.lesswrong.com/wiki/Intelligence_explosion https://wiki.lesswrong.com/wiki/Intelligence_explosion >If a paperclip AI is so dedicated to the order to produce paperclips, why wouldn't it be just as dedicated to any other order? Like "don't throw me in that incinerator!" The paperclipper scenario is meant to indicate that even a goal which seems benign could have extremely bad implications if pursued by a superintelligence. People concerned with AI risk typically argue that of the universe of possible goals that could be given to an AI, the vast majority of goals in that universe are functionally equivalent to papperclipping. For example, an AI could be programmed to maximize the number of happy people, but without a sufficiently precise specification of what "happy people" means, this could result in something like manufacturing lots of tiny smiley faces. An AI given that order could avoid throwing you in an incinerator and instead throw you in to the thing that's closest to being an incinerator without technically qualifying as an incinerator. Etc.
  - snowwrestler 11y ago
Retric 11y ago
I would suggest that a computer is not 'super intelligent' until it can modify it's goals. Further, maximizing paperclips in the long term may not involve building any paperclips for a very long time. https://what-if.xkcd.com/4/ https://what-if.xkcd.com/4/
- Strilanc 11y ago
  By "super-intelligent" I meant "surprisingly good at achieving specified goals in real life". A super-optimizer. An optimizer that modifies its goals is bad at achieving specified goals, so if that's what you had in mind then we're talking about different things.
  - Retric 11y ago
    We don't call people geniuses because there really good at following orders. Further, a Virus may be extremely capable of achieving specific goals in real life, but that's hardly intelligence. So, powerful but dumb optimizers might be a risk, but super intelligent AI is a different kind of risk. IMO, think cthulhu not HAL 9000. Science fiction thinks in terms of narrative causality, but AI is likely to have goals we really don't understand. EX: Maximizing the number of people that say Zulu on black Friday without anyone noticing that something odd is going on.
    - astrofinch 11y ago
      >We don't call people geniuses because there really good at following orders. If I order someone to prove whether P is equal to NP, and a day later they come back to me with a valid proof, solving a decades-long major open problem in computer science, I would call that person a genius. >EX: Maximizing the number of people that say Zulu on black Friday without anyone noticing that something odd is going on. Computers do what you say, not what you mean, so an AGI's goal would likely be some bastardized version of the intentions of the person who programmed it. Similar to how if you write a 10K line program without testing it, then run it for the first time, it will almost certainly not do what you intended it to do, but rather some bastardized version of what you intended it to do (because there will be bugs to work out).

Aron 11y ago

I think you've summarized [one of] Ben Goertzel's beliefs regarding unfriendly AI.

hosh 11y ago

Let's say we create an AI that can think for itself. There's a fear I think, that lurks in people's subconscious that ... what if the AIs, upon their own initiative, decide that humans are wasteful, inefficient beings that should be replaced? I think that comes from a guilt shared by a lot of folks, even if it never reaches the surface. Another side is, suppose an AI can think for itself and it thinks better than humans. Upon its own initiative, decides that humans are stupid and wasteful, but there is room to teach and and nurture. In either case, I think that speaks less of AIs and more about human nature and what we feel about ourselves, don't you think?

astrofinch 11y ago

"Yes, the UFAI will be able to solve Friendliness Theory. But if we haven't already solved it on our own power, we can't pinpoint Friendliness in advance, out of the space of utility functions. And if we can't pinpoint it with enough detail to draw a road map to it and it alone, we can't program the AI to care about conforming itself with that particular idiosyncratic algorithm." http://lesswrong.com/lw/igf/the_genie_knows_but_doesnt_care/ http://lesswrong.com/lw/igf/the_genie_knows_but_doesnt_care/ Let me put it another way: Humans are a result of evolution. We know that evolution created us to have as many descendants as possible. But most of us don't care, and we use technologies like condoms and birth control to cut down on the number of descendants we have. Adding more intelligence to humans helps us understand evolution in greater detail, but it does nothing to change our actual goals.