4 ms·

The current most-widely-accepted answer to this is "indirect normativity". This is basically saying that human values are complex enough that we should instruct

The current most-widely-accepted answer to this is "indirect normativity". This is basically saying that human values are complex enough that we should instruct an AI to study humans to figure out what they are, rather than try to program values in directly and risk getting them wrong. I'd be a lot more comfortable if metaethics and related philosophy advanced to the point where we didn't have to rely on indirect normativity. But I don't think this is something you can just throw money and people at, because this is a field which is sufficiently difficult and hard to judge, that work which isn't first-rate tends to detract from progress by adding noise rather than help.

yanazendo 12y ago
Indirect normativity is a losing strategy because you don't know what the resultant ethical system will be. https://ordinaryideas.wordpress.com/2012/04/21/indirect-normativity-write-up/ https://ordinaryideas.wordpress.com/2012/04/21/indirect-norm... Humanity has already solved ethics, so there's no reason for us to give up and ask AI to define ethics for us.
- Houshalter 12y ago
  >Indirect normativity is a losing strategy because you don't know what the resultant ethical system will be. That's the point... >Humanity has already solved ethics Where on earth do you get this idea?
  - yanazendo 12y ago
    Awareness of ethical systems. What makes you think ethics isn't solved? Here's a relevant essay http://medium.com/@yanazendo/how-to-be-lazy-and-successful-at-the-same-time-1c283c6880b3 http://medium.com/@yanazendo/how-to-be-lazy-and-successful-a...
    - Houshalter 12y ago
      That's not even close to a formalization of human morality. Such a thing probably isn't even possible. See this: http://lesswrong.com/lw/ld/the_hidden_complexity_of_wishes/ http://lesswrong.com/lw/ld/the_hidden_complexity_of_wishes/
vegedor 12y ago
Isn't the addition of a little noise one way to train neural networks?
TheOtherHobbes 12y ago
>This is basically saying that human values are complex enough that we should instruct an AI to study humans to figure out what they are, rather than try to program values in directly and risk getting them wrong. Oh god no. A thousand times no. Humans are masters at hiding sociopathy behind cultural and political obfuscation. A machine that looks at human leaders and generalises human values from them is absolutely guaranteed to be a monster. >I'd be a lot more comfortable if metaethics and related philosophy advanced to the point where we didn't have to rely on indirect normativity. We already have any number of perfectly sustainable ethical systems. What we don't have is the ability to follow them in practice - partly because all political systems tend to reward power and influence with more power and influence, and partly because even when there are token checks and balances (e.g. in the form of representative democracy) it's so easy to game them with appeals to irrationality and emotion. This rarely works out well for most of the population. But I don't much like the content of the proposal. I think you can have extremely useful intelligence amplification without any need for agency or personality. Much better modelling along the lines of 'If you do this, this will be the outcome' would be a game changer on its own.

yanazendo 12y ago

Indirect normativity is a losing strategy because you don't know what the resultant ethical system will be. https://ordinaryideas.wordpress.com/2012/04/21/indirect-normativity-write-up/ https://ordinaryideas.wordpress.com/2012/04/21/indirect-norm... Humanity has already solved ethics, so there's no reason for us to give up and ask AI to define ethics for us.

Houshalter 12y ago
>Indirect normativity is a losing strategy because you don't know what the resultant ethical system will be. That's the point... >Humanity has already solved ethics Where on earth do you get this idea?
- yanazendo 12y ago
  Awareness of ethical systems. What makes you think ethics isn't solved? Here's a relevant essay http://medium.com/@yanazendo/how-to-be-lazy-and-successful-at-the-same-time-1c283c6880b3 http://medium.com/@yanazendo/how-to-be-lazy-and-successful-a...
  - Houshalter 12y ago
    That's not even close to a formalization of human morality. Such a thing probably isn't even possible. See this: http://lesswrong.com/lw/ld/the_hidden_complexity_of_wishes/ http://lesswrong.com/lw/ld/the_hidden_complexity_of_wishes/

vegedor 12y ago

Isn't the addition of a little noise one way to train neural networks?

TheOtherHobbes 12y ago

>This is basically saying that human values are complex enough that we should instruct an AI to study humans to figure out what they are, rather than try to program values in directly and risk getting them wrong. Oh god no. A thousand times no. Humans are masters at hiding sociopathy behind cultural and political obfuscation. A machine that looks at human leaders and generalises human values from them is absolutely guaranteed to be a monster. >I'd be a lot more comfortable if metaethics and related philosophy advanced to the point where we didn't have to rely on indirect normativity. We already have any number of perfectly sustainable ethical systems. What we don't have is the ability to follow them in practice - partly because all political systems tend to reward power and influence with more power and influence, and partly because even when there are token checks and balances (e.g. in the form of representative democracy) it's so easy to game them with appeals to irrationality and emotion. This rarely works out well for most of the population. But I don't much like the content of the proposal. I think you can have extremely useful intelligence amplification without any need for agency or personality. Much better modelling along the lines of 'If you do this, this will be the outcome' would be a game changer on its own.