3 ms·

These arguments about hypothetical super intelligences are interesting, but my concern is not very great because we can just pull the power plug if necessary

tracedddd 6y ago
there are 15 year old hackers finding 0day kernel exploits and vm escapes. A superintelligent AI would have no problem jumping an airgap and spreading to the entire internet. It could promise anyone it interacted with billions for an Ethernet connection and deliver on its promise too. You’d have to pull the plug on _everything_ to shut it down.
- boxmonster 6y ago
  > You’d have to pull the plug on _everything_ to shut it down. Right. Individual countries have shut off their internet. Why not the world?
  - tracedddd 6y ago
    somehow, I doubt we’ve achieved high enough unity for coordination like that.
  - spuz 6y ago
    That would require everyone in the world committing themselves to living without computers for the rest of time.
twhitmore 6y ago
I think this assumption is worth at least casually testing. Turning off sounds easy -- Except that it's almost certainly connected to, or adjacent to, the network. Where else are the large volumes of training data coming from? Presumably being super-intelligent the AI would be: 1) likely to anticipate being turned off, 2) super-normally capable to identify & traverse any network security measures; 3) potentially also highly capable to identify & traverse weaknesses in other (eg. social or physical) security measures). 4) possibly able to be covert/ stealthy actions via the network. 5) possibly able, if able to reform on the Internet, to escape to there. I'm fairly much on the super-AI fence. But in a dystopian scenario the proposal of a super AI being able to be turned off as the key means of safety, seems scarcely credible. Note even that without being able to reform on the Internet (eg physical embodiment requires singularly specialized computing equipment), I can't logically see that extreme hostile actions (destroy global finance system, subversive propaganda campaigns, launch wars, launch nukes) are obviously easy to preclude.
- boxmonster 6y ago
  I'm confident safety measures can be developed. I worked in the nuclear field where the concept of "fail safe" is very important. I'm much more concerned about a corrupt human dictator utilizing nearly-super-intelligence to enslave people, which seems the far more likely case.
  - alisonkisk 6y ago
    Is it possible for a nuclear power plant to fail catastrophically, at least in theory? More than three times?

tracedddd 6y ago

there are 15 year old hackers finding 0day kernel exploits and vm escapes. A superintelligent AI would have no problem jumping an airgap and spreading to the entire internet. It could promise anyone it interacted with billions for an Ethernet connection and deliver on its promise too. You’d have to pull the plug on _everything_ to shut it down.

boxmonster 6y ago
> You’d have to pull the plug on _everything_ to shut it down. Right. Individual countries have shut off their internet. Why not the world?
- tracedddd 6y ago
  somehow, I doubt we’ve achieved high enough unity for coordination like that.
- spuz 6y ago
  That would require everyone in the world committing themselves to living without computers for the rest of time.

twhitmore 6y ago

I think this assumption is worth at least casually testing. Turning off sounds easy -- Except that it's almost certainly connected to, or adjacent to, the network. Where else are the large volumes of training data coming from? Presumably being super-intelligent the AI would be: 1) likely to anticipate being turned off, 2) super-normally capable to identify & traverse any network security measures; 3) potentially also highly capable to identify & traverse weaknesses in other (eg. social or physical) security measures). 4) possibly able to be covert/ stealthy actions via the network. 5) possibly able, if able to reform on the Internet, to escape to there. I'm fairly much on the super-AI fence. But in a dystopian scenario the proposal of a super AI being able to be turned off as the key means of safety, seems scarcely credible. Note even that without being able to reform on the Internet (eg physical embodiment requires singularly specialized computing equipment), I can't logically see that extreme hostile actions (destroy global finance system, subversive propaganda campaigns, launch wars, launch nukes) are obviously easy to preclude.

boxmonster 6y ago
I'm confident safety measures can be developed. I worked in the nuclear field where the concept of "fail safe" is very important. I'm much more concerned about a corrupt human dictator utilizing nearly-super-intelligence to enslave people, which seems the far more likely case.
- alisonkisk 6y ago
  Is it possible for a nuclear power plant to fail catastrophically, at least in theory? More than three times?