5 ms·

Humans are similarly constrained in their actions by legal structures. We wouldn't think of giving other people autonomy and freedom to commit genocide, and we

Humans are similarly constrained in their actions by legal structures. We wouldn't think of giving other people autonomy and freedom to commit genocide, and we wouldn't think of human rights laws that forbid such actions as "slavery." This is because, for the most part, we all share a common humanity that places our minds into a similar space of configurations. An AGI has absolutely no requirement to be anywhere near our sort of mind. It has no default obligation to morality that we would find acceptable or safe. I think the issue here is that when we hear words like "control" or "serving humans" we imagine the AI as a little person in a machine. We associate the word "slave" with the intelligence and imagine an emotional, resentful person whose resentment and chafing at his chains comes from a specific set of environmental and evolutionary influences. EDIT: I recommend reading Yudkowsky's article "Value is Fragile" (http://lesswrong.com/lw/y3/value_is_fragile/ http://lesswrong.com/lw/y3/value_is_fragile/): >If you loose the grip of human morals and metamorals - the result is not mysterious and alien and beautiful by the standards of human value. It is moral noise, a universe tiled with paperclips. To change away from human morals in the direction of improvement rather than entropy, requires a criterion of improvement; and that criterion would be physically represented in our brains, and our brains alone. >Relax the grip of human value upon the universe, and it will end up seriously valueless. Not, strange and alien and wonderful, shocking and terrifying and beautiful beyond all human imagination. Just, tiled with paperclips.

zachalexander 12y ago
I'm not talking about constraining AI not to commit genocide. I'm talking about enslaving it in the ordinary sense – taking a complex intelligence that would probably prefer doing its own thing over being forced to perform some (likely menial) task for the benefit of others.
- DanAndersen 12y ago
  Why the assumption that complex intelligence implies wanting to do its own thing? What would make something menial or not menial for an intelligence? The article I linked, about the fragility of value, even mentions the importance of the human value of boredom to our life experiences, and that a respect for boredom isn't a thing you get for free in any intelligence. You could potentially have an intelligence that gains extreme fulfillment from doing a particular task repeatedly, without caring that the experience was "getting old." Again, there's a tendency to see an arbitrary intelligence as a little person in a machine. Humans, by their nature and utility function (ill-defined as it is) have boredom, usually don't like menial tasks, and when forced to do something would prefer doing their own thing. When building an AGI, assuming you can make it safe, you wouldn't build something that would prefer doing its own thing in the first place. In that case, is there a moral issue? In Praise of Boredom: http://lesswrong.com/lw/xr/in_praise_of_boredom/ http://lesswrong.com/lw/xr/in_praise_of_boredom/
  - zachalexander 12y ago
    I would say the same things I said in this nearby comment: https://news.ycombinator.com/item?id=9325518 https://news.ycombinator.com/item?id=9325518
clickok 12y ago
This sort of rests upon the idea that artificial intelligences will have clear value functions which they will be singularly focused on maximizing. I am not convinced that this will be the case. Animals in general and humans in particular have a large number of conflicting drives, which interact in complicated ways. They are also thrust into environments which have complicated dynamics and where the overall state (i.e., all relevant information) is not necessarily available. Unexpected emergent behavior occurs as a result: evolution favors organisms which can successfully procreate, and in order to do this, the organism has to survive and acquire resources in its environment. Plausibly, the organisms might achieve a greater degree of fitness by cooperating with other organisms, or expending energy to better understand the environment, or modifying the environment itself, etc. It is less straightforward to see how we get human culture from that-- Art, Religion, Philosophy, Science, can be justified ex post facto via evopsych arguments, but the fact remains that all of those came from the value function that favors survival and procreation. We don't know if robots tasked with manufacturing bindings for stationary would manifest similarly complex behavior, but if you're worried about an AI going beyond its specification towards tessellating the universe with paperclips it seems like you're arguing that it might. So if the agent is capable of manipulating its creators (as well as the raw material of the entire universe), I think that you can't just say "oh, it's non-human, we should cripple/enslave it" without admitting there might be something to worry about here, either from an ethical standpoint or the more practical concern that it might be unwise to start on such an adversarial footing with a superintelligence.
- eli_gottlieb 12y ago
  >Animals in general and humans in particular have a large number of conflicting drives, which interact in complicated ways. They are also thrust into environments which have complicated dynamics and where the overall state (i.e., all relevant information) is not necessarily available. Yes, but the actual mechanism by which the animal learns what to do, as it turns out, thanks theoretical neuroscience, is basically reinforcement learning. So it is very likely that the first powerful artificial agents will be reinforcement learners, because scientists usually prototype and experiment by duplicating from Nature. And nothing in reinforcement learning particularly stops the agent from just grabbing its electronic crack-pipe and doing its own thing.

zachalexander 12y ago

I'm not talking about constraining AI not to commit genocide. I'm talking about enslaving it in the ordinary sense – taking a complex intelligence that would probably prefer doing its own thing over being forced to perform some (likely menial) task for the benefit of others.

DanAndersen 12y ago
Why the assumption that complex intelligence implies wanting to do its own thing? What would make something menial or not menial for an intelligence? The article I linked, about the fragility of value, even mentions the importance of the human value of boredom to our life experiences, and that a respect for boredom isn't a thing you get for free in any intelligence. You could potentially have an intelligence that gains extreme fulfillment from doing a particular task repeatedly, without caring that the experience was "getting old." Again, there's a tendency to see an arbitrary intelligence as a little person in a machine. Humans, by their nature and utility function (ill-defined as it is) have boredom, usually don't like menial tasks, and when forced to do something would prefer doing their own thing. When building an AGI, assuming you can make it safe, you wouldn't build something that would prefer doing its own thing in the first place. In that case, is there a moral issue? In Praise of Boredom: http://lesswrong.com/lw/xr/in_praise_of_boredom/ http://lesswrong.com/lw/xr/in_praise_of_boredom/
- zachalexander 12y ago
  I would say the same things I said in this nearby comment: https://news.ycombinator.com/item?id=9325518 https://news.ycombinator.com/item?id=9325518

clickok 12y ago

This sort of rests upon the idea that artificial intelligences will have clear value functions which they will be singularly focused on maximizing. I am not convinced that this will be the case. Animals in general and humans in particular have a large number of conflicting drives, which interact in complicated ways. They are also thrust into environments which have complicated dynamics and where the overall state (i.e., all relevant information) is not necessarily available. Unexpected emergent behavior occurs as a result: evolution favors organisms which can successfully procreate, and in order to do this, the organism has to survive and acquire resources in its environment. Plausibly, the organisms might achieve a greater degree of fitness by cooperating with other organisms, or expending energy to better understand the environment, or modifying the environment itself, etc. It is less straightforward to see how we get human culture from that-- Art, Religion, Philosophy, Science, can be justified ex post facto via evopsych arguments, but the fact remains that all of those came from the value function that favors survival and procreation. We don't know if robots tasked with manufacturing bindings for stationary would manifest similarly complex behavior, but if you're worried about an AI going beyond its specification towards tessellating the universe with paperclips it seems like you're arguing that it might. So if the agent is capable of manipulating its creators (as well as the raw material of the entire universe), I think that you can't just say "oh, it's non-human, we should cripple/enslave it" without admitting there might be something to worry about here, either from an ethical standpoint or the more practical concern that it might be unwise to start on such an adversarial footing with a superintelligence.

eli_gottlieb 12y ago
>Animals in general and humans in particular have a large number of conflicting drives, which interact in complicated ways. They are also thrust into environments which have complicated dynamics and where the overall state (i.e., all relevant information) is not necessarily available. Yes, but the actual mechanism by which the animal learns what to do, as it turns out, thanks theoretical neuroscience, is basically reinforcement learning. So it is very likely that the first powerful artificial agents will be reinforcement learners, because scientists usually prototype and experiment by duplicating from Nature. And nothing in reinforcement learning particularly stops the agent from just grabbing its electronic crack-pipe and doing its own thing.