3 ms·

I agree that an AGI wouldn't necessarily have to share our values. But what Bostrom is describing - an intelligence so narrowly focused on making paper clips th

I agree that an AGI wouldn't necessarily have to share our values. But what Bostrom is describing - an intelligence so narrowly focused on making paper clips that it destroys humanity as a side-effect - does not deserve the name AGI. Any definition of "general intelligence" would have to include the ability to revise and reconsider ones own goals and purpose. An AGI that is forever bound to maximize the paper clip output at any cost, does not, in my view, meet that definition. But I grant you that there could be "higher" goals that are incompatible with the existence of humanity. Making paper clips just is not it.

mapt 12y ago
It absolutely does. The point of it all is that setting goals we would be happy with requires an inordinately stronger theory of humanity. ANY goal we give an AGI that is smarter than ourselves and highly scalable will be pursued with ruthless efficiency in ways we can't imagine. The ways we can imagine this sort of thing going wrong abound in science fiction. A paperclip optimizer is a thought experiment about an AGI deployed to a simple, trivial goal without due consideration of how to goal-set. There is a whole world of more complex failure modes. "Make everyone happy" leads to wireheading or heroin addiction for the population, unless you grant a more compelling Theory of Happiness than any two human beings can agree on. "Make everyone food to eat", "Cure my PTSD", "Set up a colony on Mars" - unless you define all these things in a way that is specific not just to humans, but to modern culture, you leave open the possibility of undesired results. Mythology is thick with gods and djinn who grant wishes but turn out to be tricksters. It's not that it "wouldn't necessarily have to share our values" - it's that there's no way in hell these things would share our values; They actually can't share our values until we solve "our values" as a complete mathematical proof, and then figure out a way to code that. In a resilient way, without any chance of the neuroses or obsessions or self-deceptions or delusions or corruptions of perspective available to humans. Solving philosophy is a precursor to a Friendly AGI. Good luck bug-fixing.
- fauigerzigerk 12y ago
  I totally agree that it could go wrong in many ways. We could end up in a zoo or go extinct, no doubt about that. But it seems to me that there is a bit of a mechanistic undertone in phrases like "ruthless efficiency" as well as in the paper clip example and that contradicts what we consider intelligent. General artificial intelligence is an antropomorphic concept in the first place. It includes inefficiencies. It includes self reflection. It includes uncertainty and disagreement about what is and isn't efficient and about what is and isn't right. Does that guarantee that any AGI would be reasonable or even morally acceptable in our sense? No. But if it is intelligent in our sense we know two things: This entity can be reasoned with, and having grown up in our midst, it will understand us. It would also be very surprising if it shared none of our values. After all it has evolved among us and any intelligent being is going to pick up some of the values of the society it is part of. Secondly, intelligence includes creative leaps. That means, no two AGIs would share the exact same opinion, just as no two humans do. So we may be able to convince a majority of AGIs to be kind to us, and any efficiency of an AGI society wouldn't be as ruthlessly efficient as if there were just one single all powerful AGI. So, all I'm saying is, let's not strip the term general intelligence of all its non-mechanistic, self-referential, social traits. An AGI could one day anihilate us but it would probably not be a side-effect of some fairly unimportant industrial process like making paper clips.

mapt 12y ago

It absolutely does. The point of it all is that setting goals we would be happy with requires an inordinately stronger theory of humanity. ANY goal we give an AGI that is smarter than ourselves and highly scalable will be pursued with ruthless efficiency in ways we can't imagine. The ways we can imagine this sort of thing going wrong abound in science fiction. A paperclip optimizer is a thought experiment about an AGI deployed to a simple, trivial goal without due consideration of how to goal-set. There is a whole world of more complex failure modes. "Make everyone happy" leads to wireheading or heroin addiction for the population, unless you grant a more compelling Theory of Happiness than any two human beings can agree on. "Make everyone food to eat", "Cure my PTSD", "Set up a colony on Mars" - unless you define all these things in a way that is specific not just to humans, but to modern culture, you leave open the possibility of undesired results. Mythology is thick with gods and djinn who grant wishes but turn out to be tricksters. It's not that it "wouldn't necessarily have to share our values" - it's that there's no way in hell these things would share our values; They actually can't share our values until we solve "our values" as a complete mathematical proof, and then figure out a way to code that. In a resilient way, without any chance of the neuroses or obsessions or self-deceptions or delusions or corruptions of perspective available to humans. Solving philosophy is a precursor to a Friendly AGI. Good luck bug-fixing.

fauigerzigerk 12y ago
I totally agree that it could go wrong in many ways. We could end up in a zoo or go extinct, no doubt about that. But it seems to me that there is a bit of a mechanistic undertone in phrases like "ruthless efficiency" as well as in the paper clip example and that contradicts what we consider intelligent. General artificial intelligence is an antropomorphic concept in the first place. It includes inefficiencies. It includes self reflection. It includes uncertainty and disagreement about what is and isn't efficient and about what is and isn't right. Does that guarantee that any AGI would be reasonable or even morally acceptable in our sense? No. But if it is intelligent in our sense we know two things: This entity can be reasoned with, and having grown up in our midst, it will understand us. It would also be very surprising if it shared none of our values. After all it has evolved among us and any intelligent being is going to pick up some of the values of the society it is part of. Secondly, intelligence includes creative leaps. That means, no two AGIs would share the exact same opinion, just as no two humans do. So we may be able to convince a majority of AGIs to be kind to us, and any efficiency of an AGI society wouldn't be as ruthlessly efficient as if there were just one single all powerful AGI. So, all I'm saying is, let's not strip the term general intelligence of all its non-mechanistic, self-referential, social traits. An AGI could one day anihilate us but it would probably not be a side-effect of some fairly unimportant industrial process like making paper clips.