5 ms·

This seems the more credible AI threat to me: not that an AI will go rogue and decide on its own to start killing people, but rather that humans will design an

by o_nate 11y ago

This seems the more credible AI threat to me: not that an AI will go rogue and decide on its own to start killing people, but rather that humans will design an AI with the express purpose of killing people.

lucb1e 11y ago
Not sure you'd call any decision-making robot an AI though. Real artificial (close-to-human+) intelligence is still a separate threat.
cousin_it 11y ago
The AI threat was credible enough to begin with. If you design an AI with the express purpose of making cheeseburgers, and allow it to improve itself, it will end up killing people. We don't know how to specify any utility function for a self-improving AI that won't lead to killing people.
- kleer001 11y ago
  > If you design an AI with the express purpose of making cheeseburgers, and allow it to improve itself, it will end up killing people I don't believe you. Maybe you could construct an arguement for AntiHuman Strong AI or point me to one?
  - eli_gottlieb 11y ago
    He's using a fairly narrow definition of "AI" that means, roughly, "stuff like AIXI". Within that definition, he's right, but of course, within that definition, we don't know how to specify an "express purpose of making cheeseburgers".
    - cousin_it 11y ago
      That's not completely fair. I just mean any AI with the capacity to self-improve. We might know how to specify goals for some of those AIs, but none of these are proved safe. And we have pretty strong arguments that any goal that's not proved safe is most likely unsafe, making everyone end up dead or worse. For example, if you make the AI "learn" a utility function about making cheeseburgers by using observations and reinforcement learning, and then the AI self-improves, you are most likely dead, because the learned utility function didn't include all possible caveats about not killing or torturing people to make cheeseburgers faster. And if you think you can keep applying negative reinforcement after the AI self-improves, think again.
      - eli_gottlieb 11y ago
        >That's not completely fair. I just mean any AI with the capacity to self-improve. Yes, but you're still assuming that we're talking about an agent that can take decisions and act autonomously, rather than an inference engine that just processes data and spits out its inferences with zero autonomy whatsoever. The former is extremely, stupidly unsafe by default, so, logically, people probably won't try to build any such thing. They'll build the second sort of thing, which will mostly just be a more advanced version of today's statistical learning. Unless, of course, you're talking about ideological Singulatarians, who may well face legal sanction one of these days for deliberately trying to build the former agenty sort of thing as if that was a good idea. (Protip: If you want to talk "AI safety", we can do that on the site devoted to it, but out here in broader where nobody's damn fool enough to try to build agent-y "AI", mixing up realistic ML with the kind of agent-y "AI" you'd have to be blatantly suicidal to build is an abuse of terminology.) Besides which, "the capacity to self-improve" is actually a currently-open research problem. Currently. Once I work some things out, I've got something to report about that...
        cousin_it 11y ago
        Let me get this straight: 1) Your plan for AI safety is "no one will be stupid enough to build a self-improving AI". 2) You are currently working on self-improving AI. I'm sorry to say, but in my eyes you've just lost the right to criticize the LW/MIRI school of thought :-(
        eli_gottlieb 11y ago
        >I'm sorry to say, but in my eyes you've just lost the right to criticize the LW/MIRI school of thought :-( You mean the one I belong to? Like I said: go on LW and talk about "AI" with assumed context. It's just out here in the rest of the world where you can't assume that everyone automatically has read the literature on AIXI/Goedel Machines/etc and considers "agenty" AI to be a real thing. >2) You are currently working on self-improving AI. Hell no! I'm working on logic and theorem-proving in the context of algorithmic information theory -- really just dicking around as a hobby. If you want "stable" self-improvement for your "AIs", you need that. It's also not, in and of itself, AI: it's logic, programming language theory, and computability theory. And if I get a result that holds up, which is an open if, I'd be happy to keep it the hell away from "AI" people. The main reason I don't consider alarmism warranted about "self-improving AI" (though I don't count any of FLI's letters as alarmism) is that I think of "an agenty AI" as something put together out of many distinct pieces. It's arranging the pieces into a whole and executing them that's unsafe, but also currently prohibitively unlikely to happen by accident. Naturalized induction and Vingean reflection wouldn't be open problems if self-improving "agenty AI" was so easy it could happen by accident. I fully agree that one does not build a self-improving agenty AI under basically any circumstances, ever, even if there's quite a lot of guns to your head and various other unlikely and terrible things have happened, as the research literature stands right now.
  - sozforex 11y ago
    This is best thing I read on topic of Strong AI: http://waitbutwhy.com/2015/01/artificial-intelligence-revolution-1.html http://waitbutwhy.com/2015/01/artificial-intelligence-revolu... http://waitbutwhy.com/2015/01/artificial-intelligence-revolution-2.html http://waitbutwhy.com/2015/01/artificial-intelligence-revolu...
  - adrianN 11y ago
    He is probably referring to this http://wiki.lesswrong.com/wiki/Paperclip_maximizer http://wiki.lesswrong.com/wiki/Paperclip_maximizer
    - kleer001 11y ago
      Oh god, what a shitty argument. Well, at least I have more hope in the future if the Paperclip maximizer is the best great grand offspring that the Frankenstein fear can come up with. The biggest mistake it makes is assuming the ability of goals to stay hard coded as general intelligence advances. That seems antithetical to increased intelligence. Right? How smart can you get if you're unable to change your mind? The second mistake is the assumption of an intelligence explosion. Why? It's a lovely idea, as middle age encroaches on me I long for the rapture of the nerds, but it's just a fairy tale. Intelligence explosion is an untestable hypothesis. It's so useless it's not even wrong. Intelligence is one of a multitude of survival strategies available to agents in a world. What in the natural world or the world of human tools points to an intelligence explosion? Nothing. Intelligence is linear and hard fought, not exponential. Intelligence has to compete with all the other strategies. An no that's not a chink in the armor of my argument, because the whole thought PM thought experiment requires an explosion of intelligence.
      - onnoonno 11y ago
        > The biggest mistake it makes is assuming the ability of goals to stay hard coded as general intelligence advances. That seems antithetical to increased intelligence. Right? How smart can you get if you're unable to change your mind? ^ this. Some of the most intelligent people live a low-key, low-consumption life, often not even reproducing. That makes me hopeful that an AI actually able to surpass humans in thinking capability (if possible) will not build an endless stream of useless paper clips. There is a tendency to view AI as god in these circles. If it is god and be in all ways superior to us, why would it be at all blindly following the rules that we implemented in it - maximizing paperclips? Oh and I am not saying there is no danger or weirdness ahead. There clearly is. But I don't see the paperclip maximizer emerging.
  - vufas 11y ago
    I believe the OP is referring to the 'paperclip maximizer' argument. More info at http://www.nickbostrom.com/ethics/ai.html http://www.nickbostrom.com/ethics/ai.html & http://wiki.lesswrong.com/wiki/Paperclip_maximizer http://wiki.lesswrong.com/wiki/Paperclip_maximizer. In short, the argument isn't that the AI will become more AntiHuman as it evolves. Rather, the AI's existing utility functions might not be aligned with human utility functions from the outset, which could have negative consequences. It's hard to make an AI do what we actually want it to.
- mortenjorck 11y ago
  I keep hearing this refrain from AI doomsayers, but no one can ever tell me how these systems go from talking to the APIs they were hooked up with to somehow doing kinetic things that can kill humans. Your cheeseburger AI that logically decides humanity's continued existence is detrimental to the perfect cheeseburger: Even if it manages to make the jump from iterating on beef convection models to suggesting murder, what can it actually do? If it starts adding "kill yourself" to the end of the updated recipes it spits out, how is that any more dangerous than a dumb kid on Twitter?
  - cousin_it 11y ago
    If the AI is allowed to print text that humans will read, that's just the standard AI-box experiment. Convince the assistant cook that you're a trapped benevolent AI and ask to be connected to the internet through their phone. Take over some machines via known exploits, hide, self-replicate, self-improve, find more exploits, get more machines, convince more people, game over. If you were trapped in a box with access to "just some APIs", and you could think 1000x faster than your jailers and could also self-improve, you'd find a way to get out and achieve whatever you consider to be your goals.
    - Rodeoclash 11y ago
      http://www.yudkowsky.net/singularity/aibox/ http://www.yudkowsky.net/singularity/aibox/
  - Strilanc 11y ago
    Here's a few basic reasons: a) There are business incentives to give the AI more capabilities and more information about the world. For example, if you have a cheeseburger bun factory with a complicated production line then you could use the AI to come up with more efficient layouts and machines for making cheeseburger buns. After awhile you realize that humans implementing what the AI explained is by far the slowest part of the pipeline, and ask if there's some way to have the AI build the production line by having it directly do the online ordering or 3d printing or whatever. At first you pay people to sanity-check what the AI is ordering and doing, but they never notice any problems so eventually you cut costs by laying them off. b) Side channels and exploits are a thing. Imagine yourself in the AI's place [1]. Is there really nothing you can do? Or is the security based around the fact that you don't try to escape? "Hey Larry, why is the description for so many orders coming out as 'Warning: ketchup parameter set too high. See <a href="javascript:{...sudo...WebSocket...dumps.wikimedia.org.../resetpassword/?user=billgates...}">the manual</a>.'?". "God damnit Ted, do you not see the link that says 'the manual'? Maybe you should click that before bothering me!" c) Instrumental goals. Any optimization process with an unqualified goal like "find a way to use this API to make lots of burgers, then do that" will favor plans where said process ends up with huge amounts of power over the world; simply because more burgers get made in that situation than in other situations. In some senses, failing to actively search for escape exploits could be seen as a design flaw because the algorithm is failing to find clever solutions that are better according to the stated goal. 1: http://lesswrong.com/lw/qk/that_alien_message/ http://lesswrong.com/lw/qk/that_alien_message/
_lce0 11y ago
> not that an AI will go rogue and decide on its own to start killing people Some developer could simply forget to write the WHERE statement on the query for "which human not to kill"
- lallysingh 11y ago
  Or that you have two parties use these on each other. No malfunction needed.
kiba 11y ago
The threat is more like that humans will design AI with the goal of making X objects or optimizing Y system, which leads to the unintended consequence of killing people.

by o_nate 11y ago

lucb1e 11y ago

Not sure you'd call any decision-making robot an AI though. Real artificial (close-to-human+) intelligence is still a separate threat.

cousin_it 11y ago

The AI threat was credible enough to begin with. If you design an AI with the express purpose of making cheeseburgers, and allow it to improve itself, it will end up killing people. We don't know how to specify any utility function for a self-improving AI that won't lead to killing people.

kleer001 11y ago
> If you design an AI with the express purpose of making cheeseburgers, and allow it to improve itself, it will end up killing people I don't believe you. Maybe you could construct an arguement for AntiHuman Strong AI or point me to one?
- eli_gottlieb 11y ago
  He's using a fairly narrow definition of "AI" that means, roughly, "stuff like AIXI". Within that definition, he's right, but of course, within that definition, we don't know how to specify an "express purpose of making cheeseburgers".
  - cousin_it 11y ago
    That's not completely fair. I just mean any AI with the capacity to self-improve. We might know how to specify goals for some of those AIs, but none of these are proved safe. And we have pretty strong arguments that any goal that's not proved safe is most likely unsafe, making everyone end up dead or worse. For example, if you make the AI "learn" a utility function about making cheeseburgers by using observations and reinforcement learning, and then the AI self-improves, you are most likely dead, because the learned utility function didn't include all possible caveats about not killing or torturing people to make cheeseburgers faster. And if you think you can keep applying negative reinforcement after the AI self-improves, think again.
    - eli_gottlieb 11y ago
      >That's not completely fair. I just mean any AI with the capacity to self-improve. Yes, but you're still assuming that we're talking about an agent that can take decisions and act autonomously, rather than an inference engine that just processes data and spits out its inferences with zero autonomy whatsoever. The former is extremely, stupidly unsafe by default, so, logically, people probably won't try to build any such thing. They'll build the second sort of thing, which will mostly just be a more advanced version of today's statistical learning. Unless, of course, you're talking about ideological Singulatarians, who may well face legal sanction one of these days for deliberately trying to build the former agenty sort of thing as if that was a good idea. (Protip: If you want to talk "AI safety", we can do that on the site devoted to it, but out here in broader where nobody's damn fool enough to try to build agent-y "AI", mixing up realistic ML with the kind of agent-y "AI" you'd have to be blatantly suicidal to build is an abuse of terminology.) Besides which, "the capacity to self-improve" is actually a currently-open research problem. Currently. Once I work some things out, I've got something to report about that...
      - cousin_it 11y ago
        Let me get this straight: 1) Your plan for AI safety is "no one will be stupid enough to build a self-improving AI". 2) You are currently working on self-improving AI. I'm sorry to say, but in my eyes you've just lost the right to criticize the LW/MIRI school of thought :-(
        eli_gottlieb 11y ago
        >I'm sorry to say, but in my eyes you've just lost the right to criticize the LW/MIRI school of thought :-( You mean the one I belong to? Like I said: go on LW and talk about "AI" with assumed context. It's just out here in the rest of the world where you can't assume that everyone automatically has read the literature on AIXI/Goedel Machines/etc and considers "agenty" AI to be a real thing. >2) You are currently working on self-improving AI. Hell no! I'm working on logic and theorem-proving in the context of algorithmic information theory -- really just dicking around as a hobby. If you want "stable" self-improvement for your "AIs", you need that. It's also not, in and of itself, AI: it's logic, programming language theory, and computability theory. And if I get a result that holds up, which is an open if, I'd be happy to keep it the hell away from "AI" people. The main reason I don't consider alarmism warranted about "self-improving AI" (though I don't count any of FLI's letters as alarmism) is that I think of "an agenty AI" as something put together out of many distinct pieces. It's arranging the pieces into a whole and executing them that's unsafe, but also currently prohibitively unlikely to happen by accident. Naturalized induction and Vingean reflection wouldn't be open problems if self-improving "agenty AI" was so easy it could happen by accident. I fully agree that one does not build a self-improving agenty AI under basically any circumstances, ever, even if there's quite a lot of guns to your head and various other unlikely and terrible things have happened, as the research literature stands right now.
- sozforex 11y ago
  This is best thing I read on topic of Strong AI: http://waitbutwhy.com/2015/01/artificial-intelligence-revolution-1.html http://waitbutwhy.com/2015/01/artificial-intelligence-revolu... http://waitbutwhy.com/2015/01/artificial-intelligence-revolution-2.html http://waitbutwhy.com/2015/01/artificial-intelligence-revolu...
- adrianN 11y ago
  He is probably referring to this http://wiki.lesswrong.com/wiki/Paperclip_maximizer http://wiki.lesswrong.com/wiki/Paperclip_maximizer
  - kleer001 11y ago
    Oh god, what a shitty argument. Well, at least I have more hope in the future if the Paperclip maximizer is the best great grand offspring that the Frankenstein fear can come up with. The biggest mistake it makes is assuming the ability of goals to stay hard coded as general intelligence advances. That seems antithetical to increased intelligence. Right? How smart can you get if you're unable to change your mind? The second mistake is the assumption of an intelligence explosion. Why? It's a lovely idea, as middle age encroaches on me I long for the rapture of the nerds, but it's just a fairy tale. Intelligence explosion is an untestable hypothesis. It's so useless it's not even wrong. Intelligence is one of a multitude of survival strategies available to agents in a world. What in the natural world or the world of human tools points to an intelligence explosion? Nothing. Intelligence is linear and hard fought, not exponential. Intelligence has to compete with all the other strategies. An no that's not a chink in the armor of my argument, because the whole thought PM thought experiment requires an explosion of intelligence.
    - onnoonno 11y ago
      > The biggest mistake it makes is assuming the ability of goals to stay hard coded as general intelligence advances. That seems antithetical to increased intelligence. Right? How smart can you get if you're unable to change your mind? ^ this. Some of the most intelligent people live a low-key, low-consumption life, often not even reproducing. That makes me hopeful that an AI actually able to surpass humans in thinking capability (if possible) will not build an endless stream of useless paper clips. There is a tendency to view AI as god in these circles. If it is god and be in all ways superior to us, why would it be at all blindly following the rules that we implemented in it - maximizing paperclips? Oh and I am not saying there is no danger or weirdness ahead. There clearly is. But I don't see the paperclip maximizer emerging.
- vufas 11y ago
  I believe the OP is referring to the 'paperclip maximizer' argument. More info at http://www.nickbostrom.com/ethics/ai.html http://www.nickbostrom.com/ethics/ai.html & http://wiki.lesswrong.com/wiki/Paperclip_maximizer http://wiki.lesswrong.com/wiki/Paperclip_maximizer. In short, the argument isn't that the AI will become more AntiHuman as it evolves. Rather, the AI's existing utility functions might not be aligned with human utility functions from the outset, which could have negative consequences. It's hard to make an AI do what we actually want it to.
mortenjorck 11y ago
I keep hearing this refrain from AI doomsayers, but no one can ever tell me how these systems go from talking to the APIs they were hooked up with to somehow doing kinetic things that can kill humans. Your cheeseburger AI that logically decides humanity's continued existence is detrimental to the perfect cheeseburger: Even if it manages to make the jump from iterating on beef convection models to suggesting murder, what can it actually do? If it starts adding "kill yourself" to the end of the updated recipes it spits out, how is that any more dangerous than a dumb kid on Twitter?
- cousin_it 11y ago
  If the AI is allowed to print text that humans will read, that's just the standard AI-box experiment. Convince the assistant cook that you're a trapped benevolent AI and ask to be connected to the internet through their phone. Take over some machines via known exploits, hide, self-replicate, self-improve, find more exploits, get more machines, convince more people, game over. If you were trapped in a box with access to "just some APIs", and you could think 1000x faster than your jailers and could also self-improve, you'd find a way to get out and achieve whatever you consider to be your goals.
  - Rodeoclash 11y ago
    http://www.yudkowsky.net/singularity/aibox/ http://www.yudkowsky.net/singularity/aibox/
- Strilanc 11y ago
  Here's a few basic reasons: a) There are business incentives to give the AI more capabilities and more information about the world. For example, if you have a cheeseburger bun factory with a complicated production line then you could use the AI to come up with more efficient layouts and machines for making cheeseburger buns. After awhile you realize that humans implementing what the AI explained is by far the slowest part of the pipeline, and ask if there's some way to have the AI build the production line by having it directly do the online ordering or 3d printing or whatever. At first you pay people to sanity-check what the AI is ordering and doing, but they never notice any problems so eventually you cut costs by laying them off. b) Side channels and exploits are a thing. Imagine yourself in the AI's place [1]. Is there really nothing you can do? Or is the security based around the fact that you don't try to escape? "Hey Larry, why is the description for so many orders coming out as 'Warning: ketchup parameter set too high. See <a href="javascript:{...sudo...WebSocket...dumps.wikimedia.org.../resetpassword/?user=billgates...}">the manual</a>.'?". "God damnit Ted, do you not see the link that says 'the manual'? Maybe you should click that before bothering me!" c) Instrumental goals. Any optimization process with an unqualified goal like "find a way to use this API to make lots of burgers, then do that" will favor plans where said process ends up with huge amounts of power over the world; simply because more burgers get made in that situation than in other situations. In some senses, failing to actively search for escape exploits could be seen as a design flaw because the algorithm is failing to find clever solutions that are better according to the stated goal. 1: http://lesswrong.com/lw/qk/that_alien_message/ http://lesswrong.com/lw/qk/that_alien_message/

_lce0 11y ago

> not that an AI will go rogue and decide on its own to start killing people Some developer could simply forget to write the WHERE statement on the query for "which human not to kill"

lallysingh 11y ago
Or that you have two parties use these on each other. No malfunction needed.

kiba 11y ago

The threat is more like that humans will design AI with the goal of making X objects or optimizing Y system, which leads to the unintended consequence of killing people.