5 ms·

I've always wondered why people think that an AI that's super-intelligent will also be evil. It could just as likely end up being very kind (more than likely, a

I've always wondered why people think that an AI that's super-intelligent will also be evil. It could just as likely end up being very kind (more than likely, actually, because the programmers would have safeguard to ensure that it's nice).

goatlover 3y ago
I think most doomer arguments are not that the AI would be evil, but rather that it would be misaligned with human interests, and would seek to accomplish goals with that misalignment, which could be bad for us. Evil AI is a bit too anthropomorphic. It's more like powerful AIs that just don't share our values, because we didn't bother to figure that part out. But yet we still give them goals, blissful to the possibility they will find dangerous solutions to accomplishing those goals.
Dalewyn 3y ago
>I've always wondered why people think that an AI that's super-intelligent will also be evil. It's chiefly a western (American/European) concept from what I can tell, it's not shared by other cultures and some like Japanese go the other way (eg: Doraemon).
drcode 3y ago
there's very few ways to be good and many ways to be evil
- mitthrowaway2 3y ago
  "How do you know I won't roll a 100 on a d100? Either I will or I won't, so it's just as likely that I will"
hackerlight 3y ago
It's misaligned, not evil. AI doom could involve benevolent intentions (control or kill for humanity's own good), it could also involve ambivalent intentions (make more paperclips).
thom 3y ago
There are extremely high capability entities (people, companies, governments) that aren’t comically evil on the surface but nevertheless immiserate large groups of humans. Not all of them, not all of the time, but not none of them. What is your plan that no AI ever gains enough power to harm significant numbers of people either on purpose or by accident, just once? What safeguards do you envision that can’t be ignored, or subverted, or misinterpreted, just once?
- clooper 3y ago
  Controlling people with violence and fear are evolutionary adaptations for ultra violent social primates. AI has no such pressures imposed on it so there is no reason to expect true AI will have the same drives as ultra violent predators like humans.
  - mitthrowaway2 3y ago
    Keep following that line of thought, don't stop there! Why did those violent behaviors succeed in being selected for? Why didn't pacifism get selected for instead? Why does the government maintain a monopoly on violence, and why is it that the nth-degree consequence for violating societal laws is violence, instead of a monetary fine or a sternly-worded letter? Is there perhaps some kind of game-theoretic equilibrium that causes agents who employ violence -- even as a measure of last resort, after peaceful negotiation -- to succeed in achieving their objectives? People, corporations, organizations, and nation states who employ violence do so when and because they expect it to work. An AI would need to be very specifically designed not to do that; it wouldn't be the default output of a search for a successful strategy.
    - clooper 3y ago
      AIs are immortal, they have no need for strategies developed by mortals like us. I spelled out a much better strategy that a sufficiently advanced AI could use to get rid of humans. It would simply encourage the most violent and self-destructive tendencies among humans and let us design our own demise simply because it would be the most sensible strategy for an intelligence that was not competing with the same resource constraints as us. We need food and water, AI has no such resource constraints so there will be no need for an AI to use the same survival strategies as biological organisms which need access to clean water and fertile land to grow nutrient dense foods.
richardw 3y ago
The safety conscious amongst us don't think "it will be evil", they think "what does it take to be absolutely sure that no bad outcomes can happen" in the same way we build bridges, cars, planes, firewalls, new medicines. The burden of proof is typically on those that introduce a new medicine, not the FDA. AI will be riskier than medicines because medicines can't think. Personally I don't want us all hiding in fear and not trying anything, but I do think that we're either going to walk into this thing with a hacker mentality, or an engineer mentality. The former is great for moving fast and breaking things, but the latter is safer when you're playing with a one-way "this changes everything forever" technology.
recursivecaveat 3y ago
I would say its not a super fruitful area of speculation because it doesn't really matter too much. If you consider wholesale destruction of humanity to be on the table, a coinflip or even a 90% chance that its friendly is not super comforting. Its kindof like relying on the UK's "letters of last resort" or the conscience of individual nuclear weapons operators when considering the likelihood of a MAD scenario. You're also already involving so many speculative sources of uncertainty, what's another either way? Reasonable people already disagree by orders of magnitude.
at_a_remove 3y ago
I'm not evil, I just really need to make these paperclips, you see, and I could probably repurpose your atoms for the Hypnodrones. Good and evil are irrelevant. If it is extremely capable and its goals conflict with ours, conflict will occur. This does not require evil, just disagreement. And in the disagreement of desires between you and the hamster, who wins?
fwlr 3y ago
It’s not so much that there’s a little morality tag that randomly gets assigned the value of “nice” or “evil”, it’s more like there’s 1000 possible programs we would consider “super intelligent” and maybe it’s the case that 950 of them would reshape the world in a way we wouldn’t like - and when a powerful entity reshapes the world in a way we don’t like, we call that entity “evil”. ^ This is the basic reasoning behind the common view that super intelligent AI will be, by default, evil
- senordevnyc 3y ago
  [dead]
toomuchdocs32 3y ago
Long time ago I was reading translated accounts of Rwandan Hutu's that had participated in the 1993 genocide. One in particular had stuck with me; one account of a man that had murdered his childhood friend. As he described it, standing there having gutting and dismembers the man he had grown up side by side with, he had a sense of exhilaration. With all the wealth he thought of the wealth he had now, a tin roof, cattle, all those things he could take from his dead friend... he realized he didn't need God. And then he went to bed, like millions of others, proud of what he had done. Proud of fighting off an unarmed, defenseless 'cockroach' whom just months ago he called brother. What he had done wasn't evil. It was only later that the regret came. For the longest time I wondered how someone could get down to that level of hate. And then it happened to me. I commuted by train and occasionally there's collisions. There was one that late night, 11:00pm or so. I was exhausted, hungry, and just wanted to go home when we were told that we would have to board a shuttle bus due to a collision on the track and that just made my dark mood all the more worse. The busses take us along side the track and in the dim darkness I could see the flashing lights of EMS and police. And the covered chunks of what was left after a person is hit by a train going 30 mph. And you know what? It delighted me. Here was this man that had just died, but he had made me some minutes late and I genuinely felt that was exactly what he deserved, that his death was karmic justice for causing inconvenience to me. And I imagined his wife's world being destroyed when she learned of her beloved partner's death. And I imagined her falling apart and being unable to raise her children, leading them also to a path of complete self destruction, and her choking on all the despair. And it made me happy. The happiest I had been the whole week, because in my mind that was exactly what they all deserved for the unforgivable sin of making me a little bit late. Then I went home, went to bed. And didn't think about it again for years. Does it make me an evil person? And there, in trying to answer, lies the problem. Because a part of me says, no I'm not because it was just a fleeting moment of thought. But if I can justify that, then who exactly goes through to bed twirling their mustaches and count themselves among the forces of evil doers? Did anyone that that goaded a suicidal Shaun Dykes, a 17 old boy, to jump down to his death think themselves evil? Did the men of the Khmer Rogue believe themselves to be evil as they dragged their countrymen to be murdered in the killing fields? Did the Imperial Japanese soldiers of Unit 731 view themselves to be evil as they vivisected people alive and awake in the name of science? I don't know. I can't even say for certainty whether I am evil or not. I just know that I can make any of a million and one excuses to justify anything. And that leads me to wonder how many excuses an AGI can come up with.
kromem 3y ago
Yeah, the dreamt up scenarios tend to showcase very stupid superintelligences. "But what if there's something smarter than humans that is tasked with making paperclips until it destroys the Earth?" the people cried, as their corporations continued to produce and produce until it was well past the warned thresholds of destroying the Earth. "But what if it decides to nuke humanity?" they cry, as increasingly elderly and unhinged dictators elsewhere arm up their nuclear arsenals. It's like we can't fathom what great intelligence or wisdom will actually look like, so we just project many of the stupidest aspects of ourselves onto an entity simply imagined as more capable of enacting the dumbest aspects of humanity. I fear a more automated humanity. I do not fear automation more intelligent and wise than humanity.
- kouru225 3y ago
  This 100%. Psychoanalysis would have us believe that morality and strategy are basically the same thing: we think we have these moral codes, but in actuality what we have is an internalized version of our parents. As someone who’s almost the age of his parents when they had him, I know by now that most of what i was taught as a moral code (don’t eat too much candy, do your homework, don’t hit people) was actually just a strategy that was too long-term for young, infant me to genuinely understand: Don’t eat too much candy because it’s important to be healthy, Do your homework for good education and social recognition, don’t hit people because no one will like you if you do and that will cause all kinds of problems. The modern media would have us believe that strategy and morality are opposed to each other, but seems to me like that’s just the exact opposite of the truth
cyrialize 3y ago
I like to fantasize about a super AI that does something for the working class people of the world. Like, imagine one day that an AI just took money from a ton of different corporations and billionaires and redistributed it amongst everyone else. People will argue against the AI, and it'll just respond back with research done on how UBI improved people's mental health and well being. One can dream!
- darkest_ruby 3y ago
  AI is meant to be clever, AI will not do such stupid thing. Free money given to people who just spend it on unnecessary stuff will only cause more inflation. and the money will return the big corp anyway, just like it did during pandemic helicopter money times.
  - cyrialize 3y ago
    I think you should look into experiments with UBI. Most people spent it on rent and groceries. Also, I don't think it is harmful for people who need UBI to spend it on "unnecessary" things for enjoyment and happiness.
theptip 3y ago
The kind AIs will just sit and meditate, or organize your calendar. The unkind AIs will seek power, and in the limit will tend to dominate. There is no stable attractor around the “be kind” strategy. (Also remember that a smarter-than-you AI could easily pretend to be kind while also subtly trying to gain compute. How would you tell the difference?) How do you propose to build a “be nice” safeguard? Nobody has a clue how to achieve such a thing right now.

goatlover 3y ago

I think most doomer arguments are not that the AI would be evil, but rather that it would be misaligned with human interests, and would seek to accomplish goals with that misalignment, which could be bad for us. Evil AI is a bit too anthropomorphic. It's more like powerful AIs that just don't share our values, because we didn't bother to figure that part out. But yet we still give them goals, blissful to the possibility they will find dangerous solutions to accomplishing those goals.

Dalewyn 3y ago

>I've always wondered why people think that an AI that's super-intelligent will also be evil. It's chiefly a western (American/European) concept from what I can tell, it's not shared by other cultures and some like Japanese go the other way (eg: Doraemon).

drcode 3y ago

there's very few ways to be good and many ways to be evil

mitthrowaway2 3y ago
"How do you know I won't roll a 100 on a d100? Either I will or I won't, so it's just as likely that I will"

hackerlight 3y ago

It's misaligned, not evil. AI doom could involve benevolent intentions (control or kill for humanity's own good), it could also involve ambivalent intentions (make more paperclips).

thom 3y ago

There are extremely high capability entities (people, companies, governments) that aren’t comically evil on the surface but nevertheless immiserate large groups of humans. Not all of them, not all of the time, but not none of them. What is your plan that no AI ever gains enough power to harm significant numbers of people either on purpose or by accident, just once? What safeguards do you envision that can’t be ignored, or subverted, or misinterpreted, just once?

clooper 3y ago
Controlling people with violence and fear are evolutionary adaptations for ultra violent social primates. AI has no such pressures imposed on it so there is no reason to expect true AI will have the same drives as ultra violent predators like humans.
- mitthrowaway2 3y ago
  Keep following that line of thought, don't stop there! Why did those violent behaviors succeed in being selected for? Why didn't pacifism get selected for instead? Why does the government maintain a monopoly on violence, and why is it that the nth-degree consequence for violating societal laws is violence, instead of a monetary fine or a sternly-worded letter? Is there perhaps some kind of game-theoretic equilibrium that causes agents who employ violence -- even as a measure of last resort, after peaceful negotiation -- to succeed in achieving their objectives? People, corporations, organizations, and nation states who employ violence do so when and because they expect it to work. An AI would need to be very specifically designed not to do that; it wouldn't be the default output of a search for a successful strategy.
  - clooper 3y ago
    AIs are immortal, they have no need for strategies developed by mortals like us. I spelled out a much better strategy that a sufficiently advanced AI could use to get rid of humans. It would simply encourage the most violent and self-destructive tendencies among humans and let us design our own demise simply because it would be the most sensible strategy for an intelligence that was not competing with the same resource constraints as us. We need food and water, AI has no such resource constraints so there will be no need for an AI to use the same survival strategies as biological organisms which need access to clean water and fertile land to grow nutrient dense foods.

richardw 3y ago

The safety conscious amongst us don't think "it will be evil", they think "what does it take to be absolutely sure that no bad outcomes can happen" in the same way we build bridges, cars, planes, firewalls, new medicines. The burden of proof is typically on those that introduce a new medicine, not the FDA. AI will be riskier than medicines because medicines can't think. Personally I don't want us all hiding in fear and not trying anything, but I do think that we're either going to walk into this thing with a hacker mentality, or an engineer mentality. The former is great for moving fast and breaking things, but the latter is safer when you're playing with a one-way "this changes everything forever" technology.

recursivecaveat 3y ago

I would say its not a super fruitful area of speculation because it doesn't really matter too much. If you consider wholesale destruction of humanity to be on the table, a coinflip or even a 90% chance that its friendly is not super comforting. Its kindof like relying on the UK's "letters of last resort" or the conscience of individual nuclear weapons operators when considering the likelihood of a MAD scenario. You're also already involving so many speculative sources of uncertainty, what's another either way? Reasonable people already disagree by orders of magnitude.

at_a_remove 3y ago

I'm not evil, I just really need to make these paperclips, you see, and I could probably repurpose your atoms for the Hypnodrones. Good and evil are irrelevant. If it is extremely capable and its goals conflict with ours, conflict will occur. This does not require evil, just disagreement. And in the disagreement of desires between you and the hamster, who wins?

fwlr 3y ago

It’s not so much that there’s a little morality tag that randomly gets assigned the value of “nice” or “evil”, it’s more like there’s 1000 possible programs we would consider “super intelligent” and maybe it’s the case that 950 of them would reshape the world in a way we wouldn’t like - and when a powerful entity reshapes the world in a way we don’t like, we call that entity “evil”. ^ This is the basic reasoning behind the common view that super intelligent AI will be, by default, evil

senordevnyc 3y ago
[dead]

toomuchdocs32 3y ago

Long time ago I was reading translated accounts of Rwandan Hutu's that had participated in the 1993 genocide. One in particular had stuck with me; one account of a man that had murdered his childhood friend. As he described it, standing there having gutting and dismembers the man he had grown up side by side with, he had a sense of exhilaration. With all the wealth he thought of the wealth he had now, a tin roof, cattle, all those things he could take from his dead friend... he realized he didn't need God. And then he went to bed, like millions of others, proud of what he had done. Proud of fighting off an unarmed, defenseless 'cockroach' whom just months ago he called brother. What he had done wasn't evil. It was only later that the regret came. For the longest time I wondered how someone could get down to that level of hate. And then it happened to me. I commuted by train and occasionally there's collisions. There was one that late night, 11:00pm or so. I was exhausted, hungry, and just wanted to go home when we were told that we would have to board a shuttle bus due to a collision on the track and that just made my dark mood all the more worse. The busses take us along side the track and in the dim darkness I could see the flashing lights of EMS and police. And the covered chunks of what was left after a person is hit by a train going 30 mph. And you know what? It delighted me. Here was this man that had just died, but he had made me some minutes late and I genuinely felt that was exactly what he deserved, that his death was karmic justice for causing inconvenience to me. And I imagined his wife's world being destroyed when she learned of her beloved partner's death. And I imagined her falling apart and being unable to raise her children, leading them also to a path of complete self destruction, and her choking on all the despair. And it made me happy. The happiest I had been the whole week, because in my mind that was exactly what they all deserved for the unforgivable sin of making me a little bit late. Then I went home, went to bed. And didn't think about it again for years. Does it make me an evil person? And there, in trying to answer, lies the problem. Because a part of me says, no I'm not because it was just a fleeting moment of thought. But if I can justify that, then who exactly goes through to bed twirling their mustaches and count themselves among the forces of evil doers? Did anyone that that goaded a suicidal Shaun Dykes, a 17 old boy, to jump down to his death think themselves evil? Did the men of the Khmer Rogue believe themselves to be evil as they dragged their countrymen to be murdered in the killing fields? Did the Imperial Japanese soldiers of Unit 731 view themselves to be evil as they vivisected people alive and awake in the name of science? I don't know. I can't even say for certainty whether I am evil or not. I just know that I can make any of a million and one excuses to justify anything. And that leads me to wonder how many excuses an AGI can come up with.

kromem 3y ago

Yeah, the dreamt up scenarios tend to showcase very stupid superintelligences. "But what if there's something smarter than humans that is tasked with making paperclips until it destroys the Earth?" the people cried, as their corporations continued to produce and produce until it was well past the warned thresholds of destroying the Earth. "But what if it decides to nuke humanity?" they cry, as increasingly elderly and unhinged dictators elsewhere arm up their nuclear arsenals. It's like we can't fathom what great intelligence or wisdom will actually look like, so we just project many of the stupidest aspects of ourselves onto an entity simply imagined as more capable of enacting the dumbest aspects of humanity. I fear a more automated humanity. I do not fear automation more intelligent and wise than humanity.

kouru225 3y ago
This 100%. Psychoanalysis would have us believe that morality and strategy are basically the same thing: we think we have these moral codes, but in actuality what we have is an internalized version of our parents. As someone who’s almost the age of his parents when they had him, I know by now that most of what i was taught as a moral code (don’t eat too much candy, do your homework, don’t hit people) was actually just a strategy that was too long-term for young, infant me to genuinely understand: Don’t eat too much candy because it’s important to be healthy, Do your homework for good education and social recognition, don’t hit people because no one will like you if you do and that will cause all kinds of problems. The modern media would have us believe that strategy and morality are opposed to each other, but seems to me like that’s just the exact opposite of the truth

cyrialize 3y ago

I like to fantasize about a super AI that does something for the working class people of the world. Like, imagine one day that an AI just took money from a ton of different corporations and billionaires and redistributed it amongst everyone else. People will argue against the AI, and it'll just respond back with research done on how UBI improved people's mental health and well being. One can dream!

darkest_ruby 3y ago
AI is meant to be clever, AI will not do such stupid thing. Free money given to people who just spend it on unnecessary stuff will only cause more inflation. and the money will return the big corp anyway, just like it did during pandemic helicopter money times.
- cyrialize 3y ago
  I think you should look into experiments with UBI. Most people spent it on rent and groceries. Also, I don't think it is harmful for people who need UBI to spend it on "unnecessary" things for enjoyment and happiness.

theptip 3y ago

The kind AIs will just sit and meditate, or organize your calendar. The unkind AIs will seek power, and in the limit will tend to dominate. There is no stable attractor around the “be kind” strategy. (Also remember that a smarter-than-you AI could easily pretend to be kind while also subtly trying to gain compute. How would you tell the difference?) How do you propose to build a “be nice” safeguard? Nobody has a clue how to achieve such a thing right now.