3 ms·

The friendly AI goal alignment problem is a pretty interesting part of this and there’s interesting work going on there. The basic idea is how do you effective

The friendly AI goal alignment problem is a pretty interesting part of this and there’s interesting work going on there. The basic idea is how do you effectively align a general intelligence’s goal with humans so that it uses its intelligence to solve problems in a way that aligns with our own human utility functions. If humans figure out general AI before goal alignment it may have outcomes we don’t want. http://lesswrong.com/lw/ld/the_hidden_complexity_of_wishes/ http://lesswrong.com/lw/ld/the_hidden_complexity_of_wishes/ https://youtu.be/EUjc1WuyPT8 https://youtu.be/EUjc1WuyPT8

skybrian 9y ago
Maybe it would be better to stick to narrow AI. There's no reason a self-driving car or stock-picking algorithm or even a cancer research algorithm needs to be general AI.
- fossuser 9y ago
  In some sense we got lucky with nuclear weapons because they require enriched uranium to make which is hard to get. A self improving general AI may just be be a matter of an intuitive leap, but nothing that’s hard to turn on once you understand it. This property would make it hard to suppress. The upside is that if we can figure out the goal alignment problem we may be able to fix the coordination issues humans have and have a much better end state than all of us burning out and going extinct on our planet. The downside is some variant of the world turning into paper clips. http://slatestarcodex.com/2014/07/30/meditations-on-moloch/ http://slatestarcodex.com/2014/07/30/meditations-on-moloch/ https://wiki.lesswrong.com/wiki/Paperclip_maximizer https://wiki.lesswrong.com/wiki/Paperclip_maximizer
  - skybrian 9y ago
    If it's that easy, we're doomed. Even if goal alignment works, you can't make people use it. And even if they did, whose goals would they have it align with? The goals of evil humans are plenty bad enough.
    - fossuser 9y ago
      In the short term it would still probably take a lot of computing power. If you can figure out the goal alignment problem (or even simpler parts like just how to get the AI to let you turn it off) then maybe the first version can help us prevent rogue general intelligences that do things we don't want. It is a hard problem though which is why there's all the talk of existential risk. It's also a little nuanced to explain why there's concern or why people should work on the problem. I thought this post from yesterday was a good response to someone stating that an intelligence explosion was impossible (the original article it's responding to is linked within it): https://intelligence.org/2017/12/06/chollet/ https://intelligence.org/2017/12/06/chollet/
    - AgentME 9y ago
      If people make friendly AIs first, then we might be able to use those friendly AIs to figure out the actions that unfriendly AIs would take to get power, and close them off first.
      - skybrian 9y ago
        Okay, but do you need general AI for that? Why not narrow AI? On the defensive side I think there's a lot that can be done with narrow AI. Consider using really good penetration testers and bug finders to fix Internet security. For any given task, build a tool that just does that. It's like the least privilege principle in computer security.
        AgentME 9y ago
        I think there's a ton of routes to power for a general AI other than network security holes, and someone making a narrow AI just to fix all the bugs in our computer networks will miss nearly all of these. Consider an AI that makes a ton of money on mechanical turk -like jobs online, uses the money to run thousands of shell companies, over time hires the whole population to work on things related to paperclip factories, and crowds out other job opportunities so that anyone that goes against it goes jobless. Or an AI that figures out how to create cultural memes stupidly easily, gets people to vote in totalitarian governments, and then blackmails every single member of the central planning committees to optimize their countries for paperclip production. These ideas are mainly extreme proof-of-concept ideas, maybe there's ideas along these same lines that are much easier to pull off, but maybe these ideas as-is would be do-able for a possible superhuman intellect. I'm kind of thinking about what it looks like when individuals meticulously plan and then make tool-assisted speedrun videos through videogames, and then scaling it up to what it would look like if thousands of people worked together optimally for thousands of years brainstorming and calculating on how to speedrun through conquering a society. Physical limits permit this kind of computation as far as we know. Maybe this imagery is off-base, but I'm purposefully shooting high and worrying I'm not shooting high enough. Chimps would never even consider the possibility of "super-chimp" intelligences creating nuclear weapons. You can't make narrow AIs to defend from these possibilities if half the danger is our inability to even enumerate them.
joshmarlow 9y ago
I kind of think that we are already experiencing the fall-out from the value-alignment problem with corporations. Corporations (and the market in general) seem rather good at optimizing revenue/profit - the problem is that corporate profit doesn't always align with other human values - which ties into some other comments here about disrupting human attention-span being profitable. Perhaps work on AI goal alignment can help us build a more humane market place, or vice versa, research into macro-economics can inspire some ideas in AI goal alignment.
- Karrot_Kream 9y ago
  Unfortunately, I think this is because we have stopped questioning whether corporate values really do end up matching human values. I think when capitalism was seen as a competitor to communism in the Cold War era, people took a slightly more critical eye at economic systems, but now there's no competition so there's no pressure for corporate values to align.
  - Danihan 9y ago
    Corporate values do align with human values in a free market It's just that the values and goals of many humans are unaligned with the values and goals of many other humans. Which results in a cacophonous marketplace.

skybrian 9y ago

Maybe it would be better to stick to narrow AI. There's no reason a self-driving car or stock-picking algorithm or even a cancer research algorithm needs to be general AI.

fossuser 9y ago
In some sense we got lucky with nuclear weapons because they require enriched uranium to make which is hard to get. A self improving general AI may just be be a matter of an intuitive leap, but nothing that’s hard to turn on once you understand it. This property would make it hard to suppress. The upside is that if we can figure out the goal alignment problem we may be able to fix the coordination issues humans have and have a much better end state than all of us burning out and going extinct on our planet. The downside is some variant of the world turning into paper clips. http://slatestarcodex.com/2014/07/30/meditations-on-moloch/ http://slatestarcodex.com/2014/07/30/meditations-on-moloch/ https://wiki.lesswrong.com/wiki/Paperclip_maximizer https://wiki.lesswrong.com/wiki/Paperclip_maximizer
- skybrian 9y ago
  If it's that easy, we're doomed. Even if goal alignment works, you can't make people use it. And even if they did, whose goals would they have it align with? The goals of evil humans are plenty bad enough.
  - fossuser 9y ago
    In the short term it would still probably take a lot of computing power. If you can figure out the goal alignment problem (or even simpler parts like just how to get the AI to let you turn it off) then maybe the first version can help us prevent rogue general intelligences that do things we don't want. It is a hard problem though which is why there's all the talk of existential risk. It's also a little nuanced to explain why there's concern or why people should work on the problem. I thought this post from yesterday was a good response to someone stating that an intelligence explosion was impossible (the original article it's responding to is linked within it): https://intelligence.org/2017/12/06/chollet/ https://intelligence.org/2017/12/06/chollet/
  - AgentME 9y ago
    If people make friendly AIs first, then we might be able to use those friendly AIs to figure out the actions that unfriendly AIs would take to get power, and close them off first.
    - skybrian 9y ago
      Okay, but do you need general AI for that? Why not narrow AI? On the defensive side I think there's a lot that can be done with narrow AI. Consider using really good penetration testers and bug finders to fix Internet security. For any given task, build a tool that just does that. It's like the least privilege principle in computer security.
      - AgentME 9y ago
        I think there's a ton of routes to power for a general AI other than network security holes, and someone making a narrow AI just to fix all the bugs in our computer networks will miss nearly all of these. Consider an AI that makes a ton of money on mechanical turk -like jobs online, uses the money to run thousands of shell companies, over time hires the whole population to work on things related to paperclip factories, and crowds out other job opportunities so that anyone that goes against it goes jobless. Or an AI that figures out how to create cultural memes stupidly easily, gets people to vote in totalitarian governments, and then blackmails every single member of the central planning committees to optimize their countries for paperclip production. These ideas are mainly extreme proof-of-concept ideas, maybe there's ideas along these same lines that are much easier to pull off, but maybe these ideas as-is would be do-able for a possible superhuman intellect. I'm kind of thinking about what it looks like when individuals meticulously plan and then make tool-assisted speedrun videos through videogames, and then scaling it up to what it would look like if thousands of people worked together optimally for thousands of years brainstorming and calculating on how to speedrun through conquering a society. Physical limits permit this kind of computation as far as we know. Maybe this imagery is off-base, but I'm purposefully shooting high and worrying I'm not shooting high enough. Chimps would never even consider the possibility of "super-chimp" intelligences creating nuclear weapons. You can't make narrow AIs to defend from these possibilities if half the danger is our inability to even enumerate them.

joshmarlow 9y ago

I kind of think that we are already experiencing the fall-out from the value-alignment problem with corporations. Corporations (and the market in general) seem rather good at optimizing revenue/profit - the problem is that corporate profit doesn't always align with other human values - which ties into some other comments here about disrupting human attention-span being profitable. Perhaps work on AI goal alignment can help us build a more humane market place, or vice versa, research into macro-economics can inspire some ideas in AI goal alignment.

Karrot_Kream 9y ago
Unfortunately, I think this is because we have stopped questioning whether corporate values really do end up matching human values. I think when capitalism was seen as a competitor to communism in the Cold War era, people took a slightly more critical eye at economic systems, but now there's no competition so there's no pressure for corporate values to align.
- Danihan 9y ago
  Corporate values do align with human values in a free market It's just that the values and goals of many humans are unaligned with the values and goals of many other humans. Which results in a cacophonous marketplace.