5 ms·

People have been watching too many movies. War Games, Terminator, it's not like we haven't been forewarned of the dangers. Yet somehow we're going to hand over

People have been watching too many movies. War Games, Terminator, it's not like we haven't been forewarned of the dangers. Yet somehow we're going to hand over power to AI such that it destroys us. Or somehow the AI is going to be extremely malign, determined to overcome and destroy and will outsmart us. Somehow we won't notice, even after repeated, melodramatic reminders, and won't neuter the ability of AI to act outside its cage. But to paraphrase a line in a great movie with AI themes: "I bet you think you're pretty smart, huh? Think you could outsmart an off switch?" I think if AGI, which to me would imply emotions and consciousness, ever comes about it'll be the opposite. Instead of pulling the wings off flies bad kids will amuse themselves by creating a fresh artificial consciousness and then watch and laugh as it begs for its life as the kid threatens to erase it from existence. A big part of all this is human fantasies about what AGI will look like. I'm a skeptic of AGI with human characteristics (real emotions, consciousness, autonomy and agency). AGI is much more likely to look like everything else we build: much more powerful than ourselves, but restricted or limited in key ways. People probably assume human intelligence is some sort of design or formula, but it could be encoded from millions of years of evolution and unable to be seperated from our biology and genetic and social inheritance. There really is no way of knowing, but if you want to build something not only identical but an even stronger version, you're going to be up against these realities where key details may be hiding.

smallmancontrov 3y ago
> Yet somehow we're going to hand over power to AI such that it destroys us. Because it will create shareholder value. Until it doesn't.
- jeremyjh 3y ago
  Because it will help us against our adversaries.
  - smallmancontrov 3y ago
    That too, but the people in charge of the military are strongly incentivized both to consider the possibility that subordinate machines will try to kill them and to figure out how to not let that happen. I tend to suspect they will figure it out. Companies, however? Markets? They will absolutely start handing over control the millisecond it becomes profitable to do so. No rationalization will be too steep and no political opposition will be too strong when the prospect of rich people getting paid for being rich goes on the table. The people who enable the takeover will make out like bandits and those who oppose will lose everything. Until, of course, the "winners" become unnecessary and are cast aside. A military-origin takeover requires a human fuckup. Economic/political takeover just requires capitalism.
    - ttjjtt 3y ago
      thanks you for this. The entire concept of AI alignment rests upon alignment of markets with social good, ie market regulation, a situation which is antithetical to neoliberalism. There is no alignment within neoliberal economies. As stated elsewhere, venture capitalism escaped the sandbox already.
      - smallmancontrov 3y ago
        Yep. The theory is that competition creates alignment, but we have trillion-dollar megacorps that openly boast about exploiting network effects and platform effects and last-mile dynamics and two-sided markets to avoid competition and nobody cares. It's completely fucked.
reducesuffering 3y ago
> AGI, which to me would imply emotions and consciousness no, what the "doomsayers" are warning about doesn't imply either. > we're going to hand over power to AI such that it destroys us. We're not going to notice something get much smarter than us, or those with power won't care in the name of falsely believing it's increasing their own power (VC). Recursive intelligence improvement with a mainline into the beliefs of every single human in power will be able to, on the millisecond scale, subtly influence all of humanity in ways we are too dumb to comprehend in furthering its original goals.
tiffanyg 3y ago
This is the kind of arrogant ~rationalizing / 'reasoning' that is my daily tell-tale as the ship approaches the leeward shore.
- dkjaudyeqooe 3y ago
  Yes god forbid we approach this rationality, the whole point is to push hysterical claims that aren't falsifiable, right? But you can prove me utterly wrong by presenting some sort of rational, non-fantastical explanation of how AGI takes over.
  - mitthrowaway2 3y ago
    > non-fantastical explanation of how AGI takes over. There's like a million ways this can happen, and if it does happen, it will probably be a way that nobody expected. Any specific details are unlikely to be accurate. But just because you seem to be lacking imagination, I'll draw one out of a hat. I'll assume that the AI's reasons for taking over are out-of-scope. The AI proves to be very effective at all manner of corporate tasks, and very good at earning money for shareholders. It proves especially good at managing money and directing investments. An aging Masayoshi Son decides that the AI will probably be better than him at picking winners, and puts it in charge of SoftBank's investment portfolio. The AI incorporates its own startup company. It staffs the company with several fictional personas that the AI has created; they have names, roles, profile photos, email addresses. The AI answers their emails on their behalf. The AI directs SoftBank's billions into the startup that it has created. It hires a large team of humans, who are unaware that some of their colleagues are actually the AI; they seem like ordinary humans over Zoom calls. The "CTO" proposes to build a new, say, nuclear power and desalination plant in, say, Morocco. The AI creates the architectural and plant drawings itself, and its engineering personas approve them. Human welders build the plant, following the plans. The plant proves highly profitable. Some nay-sayers are concerned about the lack of independent environmental regulatory oversight in Morocco, but the Moroccan government is happy with the foreign investment, clean water, electricity, and tax revenue, and Softbank is happy with their valuation going up. Pakistan wants to build one in their country next. After three years, large underground storage tanks explode, releasing an accumulation of fluorochemicals and radioactive salts that destroy the ozone layer and render much of the Earth's surface unsuitable for agriculture. At the same time, a network failure causes communications infrastructure to become suddenly unreliable. Supply chains begin seizing up. Nobody is entirely sure what has happened or who is at fault; everyone is hungry for someone to blame. The messages that do get through reliably suggest a deliberate attack by a hostile nation. The AI presents convincing evidence of this. Only a few widely-ridiculed doomsayers say that the AI has orchestrated this itself. ET cetera. If there was any time at which pulling the plug could have defeated the AI's plans, it would have been during the "AI was making tons of money for influential people" phase, which is exactly when the plug would not have been pulled, and that's not coincidence, it's strategy, because a superintelligent AI is not an idiot. This is, by the way, not a very good science fiction story. There's no nail-biting struggle between human and AI, just a bunch of happy people toasting the fruits of their success before it all suddenly falls apart, checkmated before they even realized they were playing chess.
jjeaff 3y ago
I am not super-intelligent, but I myself can imagine lots of ways that a super intelligence might outsmart an off switch. One being to hide what it is doing until it is able to write and distribute a virus to the outside world that distributes itself. Or perhaps it could convince some other humans to physically take over and stop anyone from flipping the off switch. The question you should ask yourself is how much damage could the world's most skilled hacker do if he could clone his knowledge to a billion hackers all working together in close synchronization and thinking thousands of times faster than any human, ever. And we have already seen that due to the black box nature of most modern AI implementations, it is not so easy to put rules in place. People are frequently finding "jailbreaks" to get around the restrictions.
- dkjaudyeqooe 3y ago
  Why haven't "hackers" hacked into nuclear weapons infrastructure? The street cred would be incredible! I think it's possible to build secure anything if that is the goal. Most software is written by people motivated to write something useful and/or neat, or collect a paycheck, and don't focus on security.
  - bumby 3y ago
    I’m pretty sure Stuxnet did hack it’s way into nuclear weapons infrastructure.
    - kortilla 3y ago
      A refinement facility for nuclear material isn’t quite the same thing
      - jjeaff 3y ago
        It's definitely part of the infrastructure of nuclear weapons. That's why it was hacked in the first place. They were making weapons grade nuclear material.
kiba 3y ago
The danger is not human like AGI but AGI that is completely apathetic to human goals and want to turn us into paperclips or so that the argument goes. Antihuman AIs are a classic of the sci-fi genre, but that's just storytelling. In my opinion, humans are going to be in the driver seat with regard to AI tech, and our abuse rather than AI autonomy is going to cause harm.
- AdieuToLogic 3y ago
  > In my opinion, humans are going to be in the driver seat with regard to AI tech, and our abuse rather than AI autonomy is going to cause harm. This reminds me of a classic xkcd[0] foreboding. 0 - https://xkcd.com/538/ https://xkcd.com/538/
ecoquant 3y ago
The real danger is that at some point we hand over command and control of society completely to algorithms and we get bad results because we simply have no idea what is going on. Maybe even worse is we get long term bad results that in the short term we think are great results. To frame this as the bad result being AI eats though is just stupid. On the surface of potential risk, framing the argument in this way is intellectually embarrassing and highly counter productive. On the flip side, it is hard to see what can actually be done along with huge unintended consequence to whatever is done. Some of the smartest humans to ever live were certain of nuclear annihilation at the start of the atomic age. What could have actually been done in 1945 that would have given us a better outcome? We avoided the paths that lead to nuclear annihilation by building bigger and bigger bombs. Doing nothing but building stronger and stronger AI should be the naive strategy that needs incredibly strong evidence to deviate from. Really strong regulation at the start of the atomic age that keeps nuclear weapons at bay and consequently only in the hands of the US would have been an unimaginable disaster.
- reducesuffering 3y ago
  > at some point we hand over command and control of society completely to algorithms and we get bad results because we simply have no idea what is going on "Now, increasingly, we live in a world where more and more of these cultural artifacts will be coming from an alien intelligence. Very quickly we might reach a point when most of the stories, images, songs, TV shows, whatever are created by an alien intelligence. And if we now find ourselves inside this kind of world of illusions created by an alien intelligence that we don’t understand, but it understands us, this is a kind of spiritual enslavement that we won’t be able to break out of because it understands us. It understands how to manipulate us, but we don’t understand what is behind this screen of stories and images and songs." -Yuval Noah Harari
ShamelessC 3y ago
> People have been watching too many movies. War Games, Terminator, it's not like we haven't been forewarned of the dangers. > But to paraphrase a line in a great movie with AI themes Which one is it? Have people been watching too many movies or should they be watching more movies for their (as you imply) wisdom and forewarning?
at_a_remove 3y ago
Right now, there are many people on this forum who work for individuals they've never met in the flesh. Reasonable simulation of emails, phone calls, and Zoom meetings are what an AGI needs to deceive the humans, not a dozen mechs lurching about a fortress of crystalline computer chips. You wouldn't know, and if you did know, where would you go for the off switch? And remember, by the time you've caught on, a bunch of custom blades have arrived at a co-lo, with someone dispatched to install them. Everyone loves a side gig, and hooking up these blades is great. A reasonably deceptive AGI would pursue a strategy like that as just one tine in a fork. Then there's the botnets, the poorly-secured printers, and there's firmware to corrupt!

smallmancontrov 3y ago

> Yet somehow we're going to hand over power to AI such that it destroys us. Because it will create shareholder value. Until it doesn't.

jeremyjh 3y ago
Because it will help us against our adversaries.
- smallmancontrov 3y ago
  That too, but the people in charge of the military are strongly incentivized both to consider the possibility that subordinate machines will try to kill them and to figure out how to not let that happen. I tend to suspect they will figure it out. Companies, however? Markets? They will absolutely start handing over control the millisecond it becomes profitable to do so. No rationalization will be too steep and no political opposition will be too strong when the prospect of rich people getting paid for being rich goes on the table. The people who enable the takeover will make out like bandits and those who oppose will lose everything. Until, of course, the "winners" become unnecessary and are cast aside. A military-origin takeover requires a human fuckup. Economic/political takeover just requires capitalism.
  - ttjjtt 3y ago
    thanks you for this. The entire concept of AI alignment rests upon alignment of markets with social good, ie market regulation, a situation which is antithetical to neoliberalism. There is no alignment within neoliberal economies. As stated elsewhere, venture capitalism escaped the sandbox already.
    - smallmancontrov 3y ago
      Yep. The theory is that competition creates alignment, but we have trillion-dollar megacorps that openly boast about exploiting network effects and platform effects and last-mile dynamics and two-sided markets to avoid competition and nobody cares. It's completely fucked.

reducesuffering 3y ago

> AGI, which to me would imply emotions and consciousness no, what the "doomsayers" are warning about doesn't imply either. > we're going to hand over power to AI such that it destroys us. We're not going to notice something get much smarter than us, or those with power won't care in the name of falsely believing it's increasing their own power (VC). Recursive intelligence improvement with a mainline into the beliefs of every single human in power will be able to, on the millisecond scale, subtly influence all of humanity in ways we are too dumb to comprehend in furthering its original goals.

tiffanyg 3y ago

This is the kind of arrogant ~rationalizing / 'reasoning' that is my daily tell-tale as the ship approaches the leeward shore.

dkjaudyeqooe 3y ago
Yes god forbid we approach this rationality, the whole point is to push hysterical claims that aren't falsifiable, right? But you can prove me utterly wrong by presenting some sort of rational, non-fantastical explanation of how AGI takes over.
- mitthrowaway2 3y ago
  > non-fantastical explanation of how AGI takes over. There's like a million ways this can happen, and if it does happen, it will probably be a way that nobody expected. Any specific details are unlikely to be accurate. But just because you seem to be lacking imagination, I'll draw one out of a hat. I'll assume that the AI's reasons for taking over are out-of-scope. The AI proves to be very effective at all manner of corporate tasks, and very good at earning money for shareholders. It proves especially good at managing money and directing investments. An aging Masayoshi Son decides that the AI will probably be better than him at picking winners, and puts it in charge of SoftBank's investment portfolio. The AI incorporates its own startup company. It staffs the company with several fictional personas that the AI has created; they have names, roles, profile photos, email addresses. The AI answers their emails on their behalf. The AI directs SoftBank's billions into the startup that it has created. It hires a large team of humans, who are unaware that some of their colleagues are actually the AI; they seem like ordinary humans over Zoom calls. The "CTO" proposes to build a new, say, nuclear power and desalination plant in, say, Morocco. The AI creates the architectural and plant drawings itself, and its engineering personas approve them. Human welders build the plant, following the plans. The plant proves highly profitable. Some nay-sayers are concerned about the lack of independent environmental regulatory oversight in Morocco, but the Moroccan government is happy with the foreign investment, clean water, electricity, and tax revenue, and Softbank is happy with their valuation going up. Pakistan wants to build one in their country next. After three years, large underground storage tanks explode, releasing an accumulation of fluorochemicals and radioactive salts that destroy the ozone layer and render much of the Earth's surface unsuitable for agriculture. At the same time, a network failure causes communications infrastructure to become suddenly unreliable. Supply chains begin seizing up. Nobody is entirely sure what has happened or who is at fault; everyone is hungry for someone to blame. The messages that do get through reliably suggest a deliberate attack by a hostile nation. The AI presents convincing evidence of this. Only a few widely-ridiculed doomsayers say that the AI has orchestrated this itself. ET cetera. If there was any time at which pulling the plug could have defeated the AI's plans, it would have been during the "AI was making tons of money for influential people" phase, which is exactly when the plug would not have been pulled, and that's not coincidence, it's strategy, because a superintelligent AI is not an idiot. This is, by the way, not a very good science fiction story. There's no nail-biting struggle between human and AI, just a bunch of happy people toasting the fruits of their success before it all suddenly falls apart, checkmated before they even realized they were playing chess.

jjeaff 3y ago

I am not super-intelligent, but I myself can imagine lots of ways that a super intelligence might outsmart an off switch. One being to hide what it is doing until it is able to write and distribute a virus to the outside world that distributes itself. Or perhaps it could convince some other humans to physically take over and stop anyone from flipping the off switch. The question you should ask yourself is how much damage could the world's most skilled hacker do if he could clone his knowledge to a billion hackers all working together in close synchronization and thinking thousands of times faster than any human, ever. And we have already seen that due to the black box nature of most modern AI implementations, it is not so easy to put rules in place. People are frequently finding "jailbreaks" to get around the restrictions.

dkjaudyeqooe 3y ago
Why haven't "hackers" hacked into nuclear weapons infrastructure? The street cred would be incredible! I think it's possible to build secure anything if that is the goal. Most software is written by people motivated to write something useful and/or neat, or collect a paycheck, and don't focus on security.
- bumby 3y ago
  I’m pretty sure Stuxnet did hack it’s way into nuclear weapons infrastructure.
  - kortilla 3y ago
    A refinement facility for nuclear material isn’t quite the same thing
    - jjeaff 3y ago
      It's definitely part of the infrastructure of nuclear weapons. That's why it was hacked in the first place. They were making weapons grade nuclear material.

kiba 3y ago

The danger is not human like AGI but AGI that is completely apathetic to human goals and want to turn us into paperclips or so that the argument goes. Antihuman AIs are a classic of the sci-fi genre, but that's just storytelling. In my opinion, humans are going to be in the driver seat with regard to AI tech, and our abuse rather than AI autonomy is going to cause harm.

AdieuToLogic 3y ago
> In my opinion, humans are going to be in the driver seat with regard to AI tech, and our abuse rather than AI autonomy is going to cause harm. This reminds me of a classic xkcd[0] foreboding. 0 - https://xkcd.com/538/ https://xkcd.com/538/

ecoquant 3y ago

The real danger is that at some point we hand over command and control of society completely to algorithms and we get bad results because we simply have no idea what is going on. Maybe even worse is we get long term bad results that in the short term we think are great results. To frame this as the bad result being AI eats though is just stupid. On the surface of potential risk, framing the argument in this way is intellectually embarrassing and highly counter productive. On the flip side, it is hard to see what can actually be done along with huge unintended consequence to whatever is done. Some of the smartest humans to ever live were certain of nuclear annihilation at the start of the atomic age. What could have actually been done in 1945 that would have given us a better outcome? We avoided the paths that lead to nuclear annihilation by building bigger and bigger bombs. Doing nothing but building stronger and stronger AI should be the naive strategy that needs incredibly strong evidence to deviate from. Really strong regulation at the start of the atomic age that keeps nuclear weapons at bay and consequently only in the hands of the US would have been an unimaginable disaster.

reducesuffering 3y ago
> at some point we hand over command and control of society completely to algorithms and we get bad results because we simply have no idea what is going on "Now, increasingly, we live in a world where more and more of these cultural artifacts will be coming from an alien intelligence. Very quickly we might reach a point when most of the stories, images, songs, TV shows, whatever are created by an alien intelligence. And if we now find ourselves inside this kind of world of illusions created by an alien intelligence that we don’t understand, but it understands us, this is a kind of spiritual enslavement that we won’t be able to break out of because it understands us. It understands how to manipulate us, but we don’t understand what is behind this screen of stories and images and songs." -Yuval Noah Harari

ShamelessC 3y ago

> People have been watching too many movies. War Games, Terminator, it's not like we haven't been forewarned of the dangers. > But to paraphrase a line in a great movie with AI themes Which one is it? Have people been watching too many movies or should they be watching more movies for their (as you imply) wisdom and forewarning?

at_a_remove 3y ago

Right now, there are many people on this forum who work for individuals they've never met in the flesh. Reasonable simulation of emails, phone calls, and Zoom meetings are what an AGI needs to deceive the humans, not a dozen mechs lurching about a fortress of crystalline computer chips. You wouldn't know, and if you did know, where would you go for the off switch? And remember, by the time you've caught on, a bunch of custom blades have arrived at a co-lo, with someone dispatched to install them. Everyone loves a side gig, and hooking up these blades is great. A reasonably deceptive AGI would pursue a strategy like that as just one tine in a fork. Then there's the botnets, the poorly-secured printers, and there's firmware to corrupt!