6 ms·

It's disingenuous to characterize moral disagreement as ignorance. Eliezer is aware that people think the torture is worse and of the arguments they make to jus

It's disingenuous to characterize moral disagreement as ignorance. Eliezer is aware that people think the torture is worse and of the arguments they make to justify it, he simply thinks they're wrong. Why does he think they're wrong? Probably because the resulting decision theories are exploitable or inconsistent in well understood ways when operating under uncertainty. If you prefer 10000000000000000000000000000000000 dust-specks over 1 tortures, then you should prefer to continuously blind someone with dust specks over taking a 0.0000000000000001% chance that they will torture someone. The odds that any given person is about go crazy is more than 0.0000000000000001%, so... better blind everyone. In an uncertain world, setting the badness of dust specks to ε is equivalent to setting it to 0... and setting it to 0 means the "optimal" solution to the system will probably involve throwing a whole lot of dust around. If you want a decision theory that doesn't go around blinding everyone with dust specks, or worse, when given your preferences then you must compromise on torture. > A system that is optimizing a function of n variables, where the objective depends on a subset of size k<n, will often set the remaining unconstrained variables to extreme values; if one of those unconstrained variables is actually something we care about, the solution found may be highly undesirable. http://edge.org/conversation/jaron_lanier-the-myth-of-ai http://edge.org/conversation/jaron_lanier-the-myth-of-ai

zyxley 12y ago
> If you prefer 10000000000000000000000000000000000 dust-specks over 1 tortures, then you should prefer to continuously blind someone with dust specks over taking a 0.0000000000000001% chance that they will torture someone. I fail to see how the first part of this sentence implies the second.
- Strilanc 12y ago
  That is indeed a common place where people disagree. The basic reason it should follow is that not doing so makes you inconsistent and exploitable (e.g. pumping money out of people with the Allais paradox[1]). The fact that humans have preference reversals as you move uncertainties around while holding the expected outcome constant is definitely interesting. Intuitively, it feels really truly right to value things that way. Decision theoretically, it is not optimal. Thus disagreement. Eventually we want to actually mechanize our value systems, for AI. That means nailing everything down in code, without appeal to common sense, and figuring out the consequences of that code. If an AI can change decisions when you vary uncertainties as expected outcomes are held constant, that can introduce failure cases like framing issues or infinite loops of switching back and forth between two outcomes. When writing the code you have to decide if those failures, and the resulting exploitability and inconsistency, are worse than not matching human intuitions in all cases. 1: http://en.wikipedia.org/wiki/Allais_paradox http://en.wikipedia.org/wiki/Allais_paradox
  - zyxley 12y ago
    > The basic reason it should follow is that not doing so makes you inconsistent and exploitable I still don't understand whatever framework you're trying to put forward here. How does preferring specks over torture in some absurd hypothetical somehow lead to the conclusion that I should start blinding people?
    - Strilanc 12y ago
      I'm not trying to put forth a framework, I'm trying to explain the side you're not on without straw-manning them. - If you prefer 100000000000000000000000000000000000000 specks to 1 torture - and you follow the axiom that A > B implies x% A > x% B, and value people equally - then you prefer 1 speck to a 0.000000000000000000000000000000000000001% chance of torture - and you will want to throw a speck in someone's eyes whenever it decreases the odds of torture by 0.000000000000000000000000000000000000001% - any given person has a more than 0.000000000000000000000000000000000001% of going nuts and trying to torture someone in the next day - and you can decrease those odds non-negligibly by continuously throwing specks into their eyes... so do that I will grant you that no person would take that last step. It's too stupid, too not-common-sense. But code... code does whatever you tell it to. And if you tell it to prefer 100000000000000000000000000000000000000 specks to 1 torture, and to follow decision theory axioms, don't be surprised when the solutions it spits out uses specks of dust to prevent people from seeing for absurdly small chances of preventing torture.
      - zyxley 12y ago
        What I see here is proof that this is a bad model that produces insane results, rather than it proving that I've made a bad decision.
        jtheory 12y ago
        The whole point is that the model is bad. The context is that it seems intuitively safe to teach a powerful AI that "any amount of torture is always worse than giving someone a speck of dust in the eye". Actually following that rule could be disastrous (if the AI is powerful enough), because it will realize: * some people spontaneously torture others * the chances of spontaneous torture is a bit lower if all people are experiencing a barrage of dust specks sufficient to blind them * so: eyeball dust storms commencing in 3, 2, 1... One lesson to pick up here is that if you go by intuition, rather than doing the hard work of figuring out a logic way to decide this, if the hidden assumptions behind the intuition stop being true, we'll be in a bad place.

zyxley 12y ago

> If you prefer 10000000000000000000000000000000000 dust-specks over 1 tortures, then you should prefer to continuously blind someone with dust specks over taking a 0.0000000000000001% chance that they will torture someone. I fail to see how the first part of this sentence implies the second.

Strilanc 12y ago
That is indeed a common place where people disagree. The basic reason it should follow is that not doing so makes you inconsistent and exploitable (e.g. pumping money out of people with the Allais paradox[1]). The fact that humans have preference reversals as you move uncertainties around while holding the expected outcome constant is definitely interesting. Intuitively, it feels really truly right to value things that way. Decision theoretically, it is not optimal. Thus disagreement. Eventually we want to actually mechanize our value systems, for AI. That means nailing everything down in code, without appeal to common sense, and figuring out the consequences of that code. If an AI can change decisions when you vary uncertainties as expected outcomes are held constant, that can introduce failure cases like framing issues or infinite loops of switching back and forth between two outcomes. When writing the code you have to decide if those failures, and the resulting exploitability and inconsistency, are worse than not matching human intuitions in all cases. 1: http://en.wikipedia.org/wiki/Allais_paradox http://en.wikipedia.org/wiki/Allais_paradox
- zyxley 12y ago
  > The basic reason it should follow is that not doing so makes you inconsistent and exploitable I still don't understand whatever framework you're trying to put forward here. How does preferring specks over torture in some absurd hypothetical somehow lead to the conclusion that I should start blinding people?
  - Strilanc 12y ago
    I'm not trying to put forth a framework, I'm trying to explain the side you're not on without straw-manning them. - If you prefer 100000000000000000000000000000000000000 specks to 1 torture - and you follow the axiom that A > B implies x% A > x% B, and value people equally - then you prefer 1 speck to a 0.000000000000000000000000000000000000001% chance of torture - and you will want to throw a speck in someone's eyes whenever it decreases the odds of torture by 0.000000000000000000000000000000000000001% - any given person has a more than 0.000000000000000000000000000000000001% of going nuts and trying to torture someone in the next day - and you can decrease those odds non-negligibly by continuously throwing specks into their eyes... so do that I will grant you that no person would take that last step. It's too stupid, too not-common-sense. But code... code does whatever you tell it to. And if you tell it to prefer 100000000000000000000000000000000000000 specks to 1 torture, and to follow decision theory axioms, don't be surprised when the solutions it spits out uses specks of dust to prevent people from seeing for absurdly small chances of preventing torture.
    - zyxley 12y ago
      What I see here is proof that this is a bad model that produces insane results, rather than it proving that I've made a bad decision.
      - jtheory 12y ago
        The whole point is that the model is bad. The context is that it seems intuitively safe to teach a powerful AI that "any amount of torture is always worse than giving someone a speck of dust in the eye". Actually following that rule could be disastrous (if the AI is powerful enough), because it will realize: * some people spontaneously torture others * the chances of spontaneous torture is a bit lower if all people are experiencing a barrage of dust specks sufficient to blind them * so: eyeball dust storms commencing in 3, 2, 1... One lesson to pick up here is that if you go by intuition, rather than doing the hard work of figuring out a logic way to decide this, if the hidden assumptions behind the intuition stop being true, we'll be in a bad place.