3 ms·

The issue isn't "evil AI". Very few people in the transhumanist/AI movement are concerned about evil AIs. The issue is that unless we are very careful to engine

by gizmo 15y ago

The issue isn't "evil AI". Very few people in the transhumanist/AI movement are concerned about evil AIs. The issue is that unless we are very careful to engineer an AI that cares about human values it's going to be completely indifferent to human values. Indifference is the enemy, evil is not. To illustrate, consider the African Elephant. After it reaches 40 to 60 its teeth fall out and it slowly dies in agony from starvation. This is not because nature is evil or because natural selection is evil. This sort of thing just happens because nature is completely indifferent to human values. An AI, just like a process like natural selection, is just not going to care about humans unless we get it right. And since we have only one attempt to get it right, the stakes are so absurdly high. Anyway, the guys at LessWrong have put a great deal of thought into these issues. If you're interested in this sort of thing, check out the sequences[1]. It's a few million words, but it has a terrific ratio of text to insight. [1] http://commonsenseatheism.com/?p=12774 http://commonsenseatheism.com/?p=12774

kristofferR 15y ago
Sure, "Evil AI" was just a quick and sloppy way to describe an AI acting detrimental to humans, I didn't actually mean evil. Sorry, my mistake. The thing is - in order for a true AI to act independently it has to have a purpose. Humans act the way we do because we want to have some sort of positive experience and want to avoid negative experiences. If we couldn't experience anything either positive or negative we wouldn't have a reason or motivation to do anything, we would just be. At that point we might as well be static dead objects. For AIs to act intelligently & independently and not just as algorithms solving a single task they need to have some sort of purpose/goal to reach. An independent AI can't be indifferent - it needs a basis for taking decisions. I don't see how the purpose we give the AI could be detrimental for humans without severe negligence. In addition - for an AI to be useful for humans it needs to understand humans. We obviously create AIs to serve us and in order to serve us independently it without needing manual input of tasks (which would just make it an advanced computer) it needs to understand us.
- atucker 15y ago
  Let's say that we tell the AI to eliminate malaria. So it incinerates the biosphere. Now we don't have any malaria, but we also don't have any humans. The AI would have done exactly what we asked it to do, but not what we wanted to do. For any reasonable request, you need to specify a ridiculous amount of background information as to what is and isn't acceptable. Probably any simple list you create will be missing something, and we'll be miserable/unhappy as a result of it's exclusion.
- randallsquared 15y ago
  I didn't actually mean evil. I think "evil" is exactly the right word. An entity with goals which are incompatible with your goals (in the sense that both of them cannot be reached) can be described as literally evil. That's exactly what "evil" means.
endtime 15y ago
I agree with what you're saying, but since the nature of your objection is semantic...it depends on what one means by evil. I'd say killing me to make paperclips from my atoms is evil.

by gizmo 15y ago

kristofferR 15y ago

Sure, "Evil AI" was just a quick and sloppy way to describe an AI acting detrimental to humans, I didn't actually mean evil. Sorry, my mistake. The thing is - in order for a true AI to act independently it has to have a purpose. Humans act the way we do because we want to have some sort of positive experience and want to avoid negative experiences. If we couldn't experience anything either positive or negative we wouldn't have a reason or motivation to do anything, we would just be. At that point we might as well be static dead objects. For AIs to act intelligently & independently and not just as algorithms solving a single task they need to have some sort of purpose/goal to reach. An independent AI can't be indifferent - it needs a basis for taking decisions. I don't see how the purpose we give the AI could be detrimental for humans without severe negligence. In addition - for an AI to be useful for humans it needs to understand humans. We obviously create AIs to serve us and in order to serve us independently it without needing manual input of tasks (which would just make it an advanced computer) it needs to understand us.

atucker 15y ago
Let's say that we tell the AI to eliminate malaria. So it incinerates the biosphere. Now we don't have any malaria, but we also don't have any humans. The AI would have done exactly what we asked it to do, but not what we wanted to do. For any reasonable request, you need to specify a ridiculous amount of background information as to what is and isn't acceptable. Probably any simple list you create will be missing something, and we'll be miserable/unhappy as a result of it's exclusion.
randallsquared 15y ago
I didn't actually mean evil. I think "evil" is exactly the right word. An entity with goals which are incompatible with your goals (in the sense that both of them cannot be reached) can be described as literally evil. That's exactly what "evil" means.

endtime 15y ago

I agree with what you're saying, but since the nature of your objection is semantic...it depends on what one means by evil. I'd say killing me to make paperclips from my atoms is evil.