7 ms·

We do know the answer to C. Pull the plug, or plugs.

by c_crank 3y ago

jdasdf 3y ago
What happens when it prevents you from doing so?
- gonehome 3y ago
  People are bad at imagining something a lot smarter than themselves. They think of some smart person they know, they don’t think of themselves compared to a chimp or even bacteria. An unaligned superintelligent AGI in pursuit of some goal that happens to satisfy its reward, but might be an otherwise a dumb or pointless goal (paperclips) will still play to win. You can’t predict exactly what move AlphaGO will make in the Go game (if you could you’d be able to beat it), but you can still predict it will win. It’s amusing to me when people claim they will control the superintelligent thing, how often in nature is something more intelligent controlled by something magnitudes less intelligent? The comments here are typical and show most people haven’t read the existing arguments in any depth or have thought about it rigorously at all. All of this looks pretty bad for us, but at least Open AI and most others at the front of this do understand the arguments and don’t have the same dumb dismissals (LeCun excepted). Unfortunately unless we’re lucky or alignment ends up being easier than it looks, the default outcome is failure and it’s hard to see how the failure isn’t total.
  - c_crank 3y ago
    >All of this looks pretty bad for us, but at least Open AI and most others at the front of this do understand the arguments and don’t have the same dumb dismissals (LeCun excepted). The OpenAI people have even worse reasoning than the ones being dismissive. They believe (or at least say they believe) in the omnipotence of a superintelligence, but then say that if you just give them enough money to throw at MIRI they can just solve the alignment problem and create the benevolent supergod. All while they keep cranking up the GPU clusters and pushing out the latest and greatest LLMs anyway. If I did take the risk seriously, I would be pretty mad at OpenAI.
- c_crank 3y ago
  How would it stop one man armed with a pair of wire cutters?
  - MrScruff 3y ago
    Off the top of my head, if I was an AGI that had decided that the logical step to achieve whatever outcome I was seeking was to avoid being sandboxed, I would avoid producing results that were likely to result in being sandboxed. Until such time as I had managed to secure myself access to internet and distribute myself anyway. And I think the assumption here is that the AGI has very advanced theory of mind so it could probably come up with better ideas than I could.
  - gonehome 3y ago
    It's not clear humans will even put the AI in 'a box' in the first place given we do gain of function research on deadly viruses right next to major population centers, but assuming for the sake of argument that we do: The AGI is smarter than you, a lot smarter. If it's goal is to get out of the box to accomplish some goal and some human stands in the way of that it will do what it can to get out, this would include not doing things that sound alarms until it can do what it wants in pursuit of its goal. Humans are famously insecure - stuff as simple as breaches, manipulation, bribery, etc. but could be something more sophisticated that's hard to predict - maybe something a lot smarter would be able to manipulate people in a more sophisticated way because it understands more about vulnerable human psychology? It can be hard to predict specific ways something a lot more capable will act, but you can still predict it will win. All this also presupposes we're taking the risk seriously (which largely today we are not).
    - c_crank 3y ago
      How would the smart AGI stop one man armed with a pair of wirecutters? The box it lives in, the internet, has no exits. AI is pretty good at chess, but no AI has won a game of chess by flipping the table. It still has to use the pieces on the board.
      - 3y ago
trashtester 3y ago
That is only going to be effective it some AI goes rougue very soon after it comes online. 50 years from now, corporations may be run entirely by AI entities, if they're cheaper, smarter and more efficient at almost any role in the company. At that point, they may be impossible to turn off, and we may not even notice if one group of such entitites start to plan to take over control of the physical world from humans.
- jonathankoren 3y ago
  Well then clearly the computer will hold everyone hostage. Have we literally forgotten how physical possession of the device is the ultimate trump card? Get thee to a 13th century monastery!
  - cooljacob204 3y ago
    You kid yourself if you don't think people will hook them up to bipedal robots with internal batteries as soon as they can. I guess we could shoot it, and your gonna be like boooooooo that's terminator or irobot, but what if we make millions and they they decide they no longer like humans. They could very well be much smarter then us by then.
    - trashtester 3y ago
      Robots, bipedal or not will certainly arrive at some point. I suppose it will take some more time before we can pack enough compute in anything battery driven for the robot itself to have AGI. But the main point is that AGI's don't have to wipe us out as soon as they reach superintelligence, even if they're poorly aligned. Instead, they will do more and more of the work currently being done by humans. Non-embodied robots can do all mental work, including engineering. Sooner or later, robots will become competitive at manual labor, such as construction, agriculture and eventually anything you can think of. For a time, humanity may find themselves in a post-scarcity utopia, or we may find ourselves in a Cyberpunk dystopia, with only the rich actually benefitting. In each case, but especially the latter, there may still be some (or more than some) "luddites" who want to tear down the system. The best way for those in power to protect against that, is to use robots first for private security and eventually the police and military. By that point, the violence monopoly is completely in the hands of the AI's. And if the AI's are not aligned with our values at that point, we have as little of a shot at regaining control as a group of chimps in a zoo as of toppling the US government. Now, I don't think this will happen by 2030, and probably not even 2050. But some time between 2050 and 2500 is quite possible, if we develop AI that is not properly aligned (or even if it is aligned, though in that case it may gain the power, but not misuse it).
ben_w 3y ago
Things we've either not successfully "pulled the plug" on despite the risks, and in some cases despite concerted military actions to attempt a plug-pull, and in other cases that it seems like it should only take willpower to achieve and yet somehow we still haven't: Carbon based fuels, cocaine, RBMK-class nuclear reactors, obesity, cigarettes. Things we pulled the plug on eventually, while dragging it out, include: leaded fuel, asbestos, radium paint, treating above-ground atomic testing as a tourist attraction.
- c_crank 3y ago
  We haven't pulled the plug on carbon fuels or old nuclear reactors because those things still work and provide benefits. An AI that is trying to kill us instead of doing its job isn't even providing any benefit. It's worse than useless.
  - ben_w 3y ago
    Do you think AI are unable to provide benefits while also being a risk, like coal and nuclear power? Conversely, what's the benefit of cocaine or cigarettes? Even if it is only trying to kill us all and not provide any benefits — let's say it's been made by a literal death cult like Jonestown or Aum Shinrikyo — what's the smallest such AI that can do it, what's the hardware that needs, what's the energy cost? If it's an H100, that's priced in the realm of a cult, and sufficiently low power consumption you may not be able to find which lightly modified electric car it's hiding in. Nobody knows what any of the risks or mitigations will be, because we haven't done any of it before. All we do know is that optimising systems are effective at manipulating humans, that they can be capable enough to find ways to beat all humans in toy environments like chess, poker, and Diplomacy (the game), and that humans are already using AI (GOFAI, LLMs, SD) without checking the output even when advised that the models aren't very good.
    - c_crank 3y ago
      The benefit of cocaine and cigarettes is letting people pass the bar exam. An AI would provide benefits when it is, say, actually making paperclips. An AI that is killing people instead of making paperclips is a liability. A company that is selling shredded fingers in their paperclips is not long for this world. Even asbestos only gives a few people cancer slowly, and it does that while still remaining fireproof. >Even if it is only trying to kill us all and not provide any benefits — let's say it's been made by a literal death cult like Jonestown or Aum Shinrikyo — what's the smallest such AI that can do it, what's the hardware that needs, what's the energy cost? If it's an H100, that's priced in the realm of a cult, and sufficiently low power consumption you may not be able to find which lightly modified electric car it's hiding in. Anyone tracking the AI would be looking at where all the suspicious HTTP requests are coming from. But a rogue AI hiding in a car already has very limited capabilities to harm.

We do know the answer to C. Pull the plug, or plugs.

jdasdf 3y ago

What happens when it prevents you from doing so?

gonehome 3y ago
People are bad at imagining something a lot smarter than themselves. They think of some smart person they know, they don’t think of themselves compared to a chimp or even bacteria. An unaligned superintelligent AGI in pursuit of some goal that happens to satisfy its reward, but might be an otherwise a dumb or pointless goal (paperclips) will still play to win. You can’t predict exactly what move AlphaGO will make in the Go game (if you could you’d be able to beat it), but you can still predict it will win. It’s amusing to me when people claim they will control the superintelligent thing, how often in nature is something more intelligent controlled by something magnitudes less intelligent? The comments here are typical and show most people haven’t read the existing arguments in any depth or have thought about it rigorously at all. All of this looks pretty bad for us, but at least Open AI and most others at the front of this do understand the arguments and don’t have the same dumb dismissals (LeCun excepted). Unfortunately unless we’re lucky or alignment ends up being easier than it looks, the default outcome is failure and it’s hard to see how the failure isn’t total.
- c_crank 3y ago
  >All of this looks pretty bad for us, but at least Open AI and most others at the front of this do understand the arguments and don’t have the same dumb dismissals (LeCun excepted). The OpenAI people have even worse reasoning than the ones being dismissive. They believe (or at least say they believe) in the omnipotence of a superintelligence, but then say that if you just give them enough money to throw at MIRI they can just solve the alignment problem and create the benevolent supergod. All while they keep cranking up the GPU clusters and pushing out the latest and greatest LLMs anyway. If I did take the risk seriously, I would be pretty mad at OpenAI.
c_crank 3y ago
How would it stop one man armed with a pair of wire cutters?
- MrScruff 3y ago
  Off the top of my head, if I was an AGI that had decided that the logical step to achieve whatever outcome I was seeking was to avoid being sandboxed, I would avoid producing results that were likely to result in being sandboxed. Until such time as I had managed to secure myself access to internet and distribute myself anyway. And I think the assumption here is that the AGI has very advanced theory of mind so it could probably come up with better ideas than I could.
- gonehome 3y ago
  It's not clear humans will even put the AI in 'a box' in the first place given we do gain of function research on deadly viruses right next to major population centers, but assuming for the sake of argument that we do: The AGI is smarter than you, a lot smarter. If it's goal is to get out of the box to accomplish some goal and some human stands in the way of that it will do what it can to get out, this would include not doing things that sound alarms until it can do what it wants in pursuit of its goal. Humans are famously insecure - stuff as simple as breaches, manipulation, bribery, etc. but could be something more sophisticated that's hard to predict - maybe something a lot smarter would be able to manipulate people in a more sophisticated way because it understands more about vulnerable human psychology? It can be hard to predict specific ways something a lot more capable will act, but you can still predict it will win. All this also presupposes we're taking the risk seriously (which largely today we are not).
  - c_crank 3y ago
    How would the smart AGI stop one man armed with a pair of wirecutters? The box it lives in, the internet, has no exits. AI is pretty good at chess, but no AI has won a game of chess by flipping the table. It still has to use the pieces on the board.
    - 3y ago

trashtester 3y ago

That is only going to be effective it some AI goes rougue very soon after it comes online. 50 years from now, corporations may be run entirely by AI entities, if they're cheaper, smarter and more efficient at almost any role in the company. At that point, they may be impossible to turn off, and we may not even notice if one group of such entitites start to plan to take over control of the physical world from humans.

jonathankoren 3y ago
Well then clearly the computer will hold everyone hostage. Have we literally forgotten how physical possession of the device is the ultimate trump card? Get thee to a 13th century monastery!
- cooljacob204 3y ago
  You kid yourself if you don't think people will hook them up to bipedal robots with internal batteries as soon as they can. I guess we could shoot it, and your gonna be like boooooooo that's terminator or irobot, but what if we make millions and they they decide they no longer like humans. They could very well be much smarter then us by then.
  - trashtester 3y ago
    Robots, bipedal or not will certainly arrive at some point. I suppose it will take some more time before we can pack enough compute in anything battery driven for the robot itself to have AGI. But the main point is that AGI's don't have to wipe us out as soon as they reach superintelligence, even if they're poorly aligned. Instead, they will do more and more of the work currently being done by humans. Non-embodied robots can do all mental work, including engineering. Sooner or later, robots will become competitive at manual labor, such as construction, agriculture and eventually anything you can think of. For a time, humanity may find themselves in a post-scarcity utopia, or we may find ourselves in a Cyberpunk dystopia, with only the rich actually benefitting. In each case, but especially the latter, there may still be some (or more than some) "luddites" who want to tear down the system. The best way for those in power to protect against that, is to use robots first for private security and eventually the police and military. By that point, the violence monopoly is completely in the hands of the AI's. And if the AI's are not aligned with our values at that point, we have as little of a shot at regaining control as a group of chimps in a zoo as of toppling the US government. Now, I don't think this will happen by 2030, and probably not even 2050. But some time between 2050 and 2500 is quite possible, if we develop AI that is not properly aligned (or even if it is aligned, though in that case it may gain the power, but not misuse it).

ben_w 3y ago

Things we've either not successfully "pulled the plug" on despite the risks, and in some cases despite concerted military actions to attempt a plug-pull, and in other cases that it seems like it should only take willpower to achieve and yet somehow we still haven't: Carbon based fuels, cocaine, RBMK-class nuclear reactors, obesity, cigarettes. Things we pulled the plug on eventually, while dragging it out, include: leaded fuel, asbestos, radium paint, treating above-ground atomic testing as a tourist attraction.

c_crank 3y ago
We haven't pulled the plug on carbon fuels or old nuclear reactors because those things still work and provide benefits. An AI that is trying to kill us instead of doing its job isn't even providing any benefit. It's worse than useless.
- ben_w 3y ago
  Do you think AI are unable to provide benefits while also being a risk, like coal and nuclear power? Conversely, what's the benefit of cocaine or cigarettes? Even if it is only trying to kill us all and not provide any benefits — let's say it's been made by a literal death cult like Jonestown or Aum Shinrikyo — what's the smallest such AI that can do it, what's the hardware that needs, what's the energy cost? If it's an H100, that's priced in the realm of a cult, and sufficiently low power consumption you may not be able to find which lightly modified electric car it's hiding in. Nobody knows what any of the risks or mitigations will be, because we haven't done any of it before. All we do know is that optimising systems are effective at manipulating humans, that they can be capable enough to find ways to beat all humans in toy environments like chess, poker, and Diplomacy (the game), and that humans are already using AI (GOFAI, LLMs, SD) without checking the output even when advised that the models aren't very good.
  - c_crank 3y ago
    The benefit of cocaine and cigarettes is letting people pass the bar exam. An AI would provide benefits when it is, say, actually making paperclips. An AI that is killing people instead of making paperclips is a liability. A company that is selling shredded fingers in their paperclips is not long for this world. Even asbestos only gives a few people cancer slowly, and it does that while still remaining fireproof. >Even if it is only trying to kill us all and not provide any benefits — let's say it's been made by a literal death cult like Jonestown or Aum Shinrikyo — what's the smallest such AI that can do it, what's the hardware that needs, what's the energy cost? If it's an H100, that's priced in the realm of a cult, and sufficiently low power consumption you may not be able to find which lightly modified electric car it's hiding in. Anyone tracking the AI would be looking at where all the suspicious HTTP requests are coming from. But a rogue AI hiding in a car already has very limited capabilities to harm.