3 ms·

In some sense we got lucky with nuclear weapons because they require enriched uranium to make which is hard to get. A self improving general AI may just be be

In some sense we got lucky with nuclear weapons because they require enriched uranium to make which is hard to get. A self improving general AI may just be be a matter of an intuitive leap, but nothing that’s hard to turn on once you understand it. This property would make it hard to suppress. The upside is that if we can figure out the goal alignment problem we may be able to fix the coordination issues humans have and have a much better end state than all of us burning out and going extinct on our planet. The downside is some variant of the world turning into paper clips. http://slatestarcodex.com/2014/07/30/meditations-on-moloch/ http://slatestarcodex.com/2014/07/30/meditations-on-moloch/ https://wiki.lesswrong.com/wiki/Paperclip_maximizer https://wiki.lesswrong.com/wiki/Paperclip_maximizer

skybrian 9y ago
If it's that easy, we're doomed. Even if goal alignment works, you can't make people use it. And even if they did, whose goals would they have it align with? The goals of evil humans are plenty bad enough.
- fossuser 9y ago
  In the short term it would still probably take a lot of computing power. If you can figure out the goal alignment problem (or even simpler parts like just how to get the AI to let you turn it off) then maybe the first version can help us prevent rogue general intelligences that do things we don't want. It is a hard problem though which is why there's all the talk of existential risk. It's also a little nuanced to explain why there's concern or why people should work on the problem. I thought this post from yesterday was a good response to someone stating that an intelligence explosion was impossible (the original article it's responding to is linked within it): https://intelligence.org/2017/12/06/chollet/ https://intelligence.org/2017/12/06/chollet/
- AgentME 9y ago
  If people make friendly AIs first, then we might be able to use those friendly AIs to figure out the actions that unfriendly AIs would take to get power, and close them off first.
  - skybrian 9y ago
    Okay, but do you need general AI for that? Why not narrow AI? On the defensive side I think there's a lot that can be done with narrow AI. Consider using really good penetration testers and bug finders to fix Internet security. For any given task, build a tool that just does that. It's like the least privilege principle in computer security.
    - AgentME 9y ago
      I think there's a ton of routes to power for a general AI other than network security holes, and someone making a narrow AI just to fix all the bugs in our computer networks will miss nearly all of these. Consider an AI that makes a ton of money on mechanical turk -like jobs online, uses the money to run thousands of shell companies, over time hires the whole population to work on things related to paperclip factories, and crowds out other job opportunities so that anyone that goes against it goes jobless. Or an AI that figures out how to create cultural memes stupidly easily, gets people to vote in totalitarian governments, and then blackmails every single member of the central planning committees to optimize their countries for paperclip production. These ideas are mainly extreme proof-of-concept ideas, maybe there's ideas along these same lines that are much easier to pull off, but maybe these ideas as-is would be do-able for a possible superhuman intellect. I'm kind of thinking about what it looks like when individuals meticulously plan and then make tool-assisted speedrun videos through videogames, and then scaling it up to what it would look like if thousands of people worked together optimally for thousands of years brainstorming and calculating on how to speedrun through conquering a society. Physical limits permit this kind of computation as far as we know. Maybe this imagery is off-base, but I'm purposefully shooting high and worrying I'm not shooting high enough. Chimps would never even consider the possibility of "super-chimp" intelligences creating nuclear weapons. You can't make narrow AIs to defend from these possibilities if half the danger is our inability to even enumerate them.

skybrian 9y ago

If it's that easy, we're doomed. Even if goal alignment works, you can't make people use it. And even if they did, whose goals would they have it align with? The goals of evil humans are plenty bad enough.

fossuser 9y ago
In the short term it would still probably take a lot of computing power. If you can figure out the goal alignment problem (or even simpler parts like just how to get the AI to let you turn it off) then maybe the first version can help us prevent rogue general intelligences that do things we don't want. It is a hard problem though which is why there's all the talk of existential risk. It's also a little nuanced to explain why there's concern or why people should work on the problem. I thought this post from yesterday was a good response to someone stating that an intelligence explosion was impossible (the original article it's responding to is linked within it): https://intelligence.org/2017/12/06/chollet/ https://intelligence.org/2017/12/06/chollet/
AgentME 9y ago
If people make friendly AIs first, then we might be able to use those friendly AIs to figure out the actions that unfriendly AIs would take to get power, and close them off first.
- skybrian 9y ago
  Okay, but do you need general AI for that? Why not narrow AI? On the defensive side I think there's a lot that can be done with narrow AI. Consider using really good penetration testers and bug finders to fix Internet security. For any given task, build a tool that just does that. It's like the least privilege principle in computer security.
  - AgentME 9y ago
    I think there's a ton of routes to power for a general AI other than network security holes, and someone making a narrow AI just to fix all the bugs in our computer networks will miss nearly all of these. Consider an AI that makes a ton of money on mechanical turk -like jobs online, uses the money to run thousands of shell companies, over time hires the whole population to work on things related to paperclip factories, and crowds out other job opportunities so that anyone that goes against it goes jobless. Or an AI that figures out how to create cultural memes stupidly easily, gets people to vote in totalitarian governments, and then blackmails every single member of the central planning committees to optimize their countries for paperclip production. These ideas are mainly extreme proof-of-concept ideas, maybe there's ideas along these same lines that are much easier to pull off, but maybe these ideas as-is would be do-able for a possible superhuman intellect. I'm kind of thinking about what it looks like when individuals meticulously plan and then make tool-assisted speedrun videos through videogames, and then scaling it up to what it would look like if thousands of people worked together optimally for thousands of years brainstorming and calculating on how to speedrun through conquering a society. Physical limits permit this kind of computation as far as we know. Maybe this imagery is off-base, but I'm purposefully shooting high and worrying I'm not shooting high enough. Chimps would never even consider the possibility of "super-chimp" intelligences creating nuclear weapons. You can't make narrow AIs to defend from these possibilities if half the danger is our inability to even enumerate them.