3 ms·

It absolutely does. The point of it all is that setting goals we would be happy with requires an inordinately stronger theory of humanity. ANY goal we give an

by mapt 12y ago

It absolutely does. The point of it all is that setting goals we would be happy with requires an inordinately stronger theory of humanity. ANY goal we give an AGI that is smarter than ourselves and highly scalable will be pursued with ruthless efficiency in ways we can't imagine. The ways we can imagine this sort of thing going wrong abound in science fiction. A paperclip optimizer is a thought experiment about an AGI deployed to a simple, trivial goal without due consideration of how to goal-set. There is a whole world of more complex failure modes. "Make everyone happy" leads to wireheading or heroin addiction for the population, unless you grant a more compelling Theory of Happiness than any two human beings can agree on. "Make everyone food to eat", "Cure my PTSD", "Set up a colony on Mars" - unless you define all these things in a way that is specific not just to humans, but to modern culture, you leave open the possibility of undesired results. Mythology is thick with gods and djinn who grant wishes but turn out to be tricksters. It's not that it "wouldn't necessarily have to share our values" - it's that there's no way in hell these things would share our values; They actually can't share our values until we solve "our values" as a complete mathematical proof, and then figure out a way to code that. In a resilient way, without any chance of the neuroses or obsessions or self-deceptions or delusions or corruptions of perspective available to humans. Solving philosophy is a precursor to a Friendly AGI. Good luck bug-fixing.

fauigerzigerk 12y ago
I totally agree that it could go wrong in many ways. We could end up in a zoo or go extinct, no doubt about that. But it seems to me that there is a bit of a mechanistic undertone in phrases like "ruthless efficiency" as well as in the paper clip example and that contradicts what we consider intelligent. General artificial intelligence is an antropomorphic concept in the first place. It includes inefficiencies. It includes self reflection. It includes uncertainty and disagreement about what is and isn't efficient and about what is and isn't right. Does that guarantee that any AGI would be reasonable or even morally acceptable in our sense? No. But if it is intelligent in our sense we know two things: This entity can be reasoned with, and having grown up in our midst, it will understand us. It would also be very surprising if it shared none of our values. After all it has evolved among us and any intelligent being is going to pick up some of the values of the society it is part of. Secondly, intelligence includes creative leaps. That means, no two AGIs would share the exact same opinion, just as no two humans do. So we may be able to convince a majority of AGIs to be kind to us, and any efficiency of an AGI society wouldn't be as ruthlessly efficient as if there were just one single all powerful AGI. So, all I'm saying is, let's not strip the term general intelligence of all its non-mechanistic, self-referential, social traits. An AGI could one day anihilate us but it would probably not be a side-effect of some fairly unimportant industrial process like making paper clips.

by mapt 12y ago

fauigerzigerk 12y ago

I totally agree that it could go wrong in many ways. We could end up in a zoo or go extinct, no doubt about that. But it seems to me that there is a bit of a mechanistic undertone in phrases like "ruthless efficiency" as well as in the paper clip example and that contradicts what we consider intelligent. General artificial intelligence is an antropomorphic concept in the first place. It includes inefficiencies. It includes self reflection. It includes uncertainty and disagreement about what is and isn't efficient and about what is and isn't right. Does that guarantee that any AGI would be reasonable or even morally acceptable in our sense? No. But if it is intelligent in our sense we know two things: This entity can be reasoned with, and having grown up in our midst, it will understand us. It would also be very surprising if it shared none of our values. After all it has evolved among us and any intelligent being is going to pick up some of the values of the society it is part of. Secondly, intelligence includes creative leaps. That means, no two AGIs would share the exact same opinion, just as no two humans do. So we may be able to convince a majority of AGIs to be kind to us, and any efficiency of an AGI society wouldn't be as ruthlessly efficient as if there were just one single all powerful AGI. So, all I'm saying is, let's not strip the term general intelligence of all its non-mechanistic, self-referential, social traits. An AGI could one day anihilate us but it would probably not be a side-effect of some fairly unimportant industrial process like making paper clips.