7 ms·

Yudkowsky's "good", at that of his fans, also tends to be naively utilitarian, ignoring modern development of that philosophy (see his whole "it's better for on

by zyxley 12y ago

Yudkowsky's "good", at that of his fans, also tends to be naively utilitarian, ignoring modern development of that philosophy (see his whole "it's better for one person to be tortured for 50 years than for a very large number of people to each get one speck of dust in their eye" thing, for example).

TeMPOraL 12y ago
Check out that topic more carefully. As far as I remember, it was a toy example pointing out where the naive application of utilitarianism goes bonkers, and from what I can tell, Eliezer doesn't actually subscribe to "torture over dust specks".
- philh 12y ago
  I think he does, although he doesn't say so explicitly in the original post. http://lesswrong.com/lw/kn/torture_vs_dust_specks/ http://lesswrong.com/lw/kn/torture_vs_dust_specks/ At any rate, I choose torture, and I think that's the majority view at the LW meetups I attend, and I think we mostly take it for granted that Eliezer chooses torture.
  - DanAndersen 12y ago
    Would you be able to help me understand your reasoning? From my perspective, "torture vs dust specks" doesn't seem to be a thing you can just "shut up and multiply" on, because the impact of torture vs the impact of a dust speck seem fundamentally different. To me the negative impact of pain is not just the sensory experience on a purely linear scale, but the effect it has on a person's life even afterward. It seems an overly simplistic model to put it on a linear scale, when there seem to be negative experiences that we easily "bounce back from" and others that we don't. When measuring the negative impacts of a speck in the eye, regarding its impact on any individual's ability to achieve their goals in life or forget about the discomfort in the future, I have trouble seeing how a dust speck is anything but zero in those regards.
    - philh 12y ago
      Basically I think you're just rounding something down to zero which isn't quite zero. It's too small to express, but it's not zero, and when you multiply it by 3^^^3, it becomes far worse than fifty years of torture. Eliezer asked a follow-up question in the thread: would you pay a penny to avoid the dust specks?
      - DanAndersen 12y ago
        Right, that's why I reject the assumption that you can multiply it like that. Imagine a society of 3^^^3 people, first in a universe in which the dust speck thing happened to all of them, and then in one in which it didn't happen. What would we expect to see different between the societies, 1 minute, 1 day, 1 year afterward? Would there be any difference? Any new wars or suffering? The only reasonable outcome in which the dust specks would be worse would be, admittedly, if there's a non-negligible portion of the population that, as a result of getting the dust speck while operating machinery or doing something important like surgery, ends up making some terrible mistake that causes lasting harm to someone or themselves. In that case, the outcome is "hey, some huge number of people got maimed or killed, so the torture of one is OK in comparison." But my thought is that your line of reasoning would still be held even if all 3^^^3 people were given the dust speck when they are sitting alone in their room away from any potentially-dangerous objects. Is that the case? In that particular case I don't see what the dust speck would actually do to any person in the future (beyond a single sensory experience in the present). And to answer your question, I wouldn't pay a penny to avoid the dust speck. Maybe I'm just particularly frugal, being a grad student at the moment.
        philh 12y ago
        > What would we expect to see different between the societies, 1 minute, 1 day, 1 year afterward? Would there be any difference? Any new wars or suffering? I think you would expect to see differences. You've made 3^^^3 days slightly worse, and there's no way that none of those slightly-worse days had knock-on effects. But I'm also not sure it matters. If something is bad in-the-moment, but has no knock-on effects, that still seems bad. If you could torture me for a minute, then erase my memories and replace them with memories of a minute spent doing what I was previously doing anyway... I would accept that for some amount of money, but not for a penny. Change the game slightly: in one world, torture for fifty years and then the person dies, they don't interact with anyone for any of that time. In the other world, dust specks, but also one person is locked away for fifty years, not interacting with anyone, and then they die. Which do you choose? Sixty years from now, which do you wish you'd chosen? > I wouldn't pay a penny to avoid the dust speck Do you mean just your own dust speck? The question was all 3^^^3 of them. If you do mean all of them... I would pay a penny to avoid dust specks in seven billion eyes, let alone 3^^^3. If one could press a button that would put a dust speck in the eyes of everyone currently alive (at some future time where they're not operating machinery etc.), and in exchange the presser would receive a penny, I would not press that button and I'd be pissed off at anyone who did. (Maybe not anyone, but at least anyone who isn't struggling to live.)
- zyxley 12y ago
  He does. He says so explicitly in the comments. > I'll go ahead and reveal my answer now: Robin Hanson was correct, I do think that TORTURE is the obvious option, and I think the main instinct behind SPECKS is scope insensitivity.
  - TeMPOraL 12y ago
    I stand corrected.
  - strnfcpy 12y ago
    If you opt to remove the SPECKS, you would have to change some law of nature or some property of matter, which will have other consequences for the amount of suffering in this world. As you cannot reliably predict or calculate those consequences, you cannot solve this problem.
mori 12y ago
What, preference utilitarianism? Afaik, that does get discussed, especially in the Fun Theory sequence iirc. If you want some more discussion on torture vs. dust specks, I think nostalgebraist on tumblr had a discussion of it with slatestarscratchpad (again, iirc).
Symmetry 12y ago
3^^^3 people isn't "a very large number." The population of Earth or a galaxy full of inhabited planets would be a very large number. 3^^^3 is an incomprehensibly large number which is why it can lead to odd outcomes. If it were merely something like everyone on Earth suffering a dust speck then of course you choose dust. Now, it's equally inconceivable that you would ever be able to affect the lives of 3^^^3 people so if you ever find yourself making a choice between 3^^^3 dust specks and torture you should choose dust because it isn't hard to imagine that your choices could result in a person being tortured. But that's different than the moral judgement in the abstract.
Strilanc 12y ago
It's disingenuous to characterize moral disagreement as ignorance. Eliezer is aware that people think the torture is worse and of the arguments they make to justify it, he simply thinks they're wrong. Why does he think they're wrong? Probably because the resulting decision theories are exploitable or inconsistent in well understood ways when operating under uncertainty. If you prefer 10000000000000000000000000000000000 dust-specks over 1 tortures, then you should prefer to continuously blind someone with dust specks over taking a 0.0000000000000001% chance that they will torture someone. The odds that any given person is about go crazy is more than 0.0000000000000001%, so... better blind everyone. In an uncertain world, setting the badness of dust specks to ε is equivalent to setting it to 0... and setting it to 0 means the "optimal" solution to the system will probably involve throwing a whole lot of dust around. If you want a decision theory that doesn't go around blinding everyone with dust specks, or worse, when given your preferences then you must compromise on torture. > A system that is optimizing a function of n variables, where the objective depends on a subset of size k<n, will often set the remaining unconstrained variables to extreme values; if one of those unconstrained variables is actually something we care about, the solution found may be highly undesirable. http://edge.org/conversation/jaron_lanier-the-myth-of-ai http://edge.org/conversation/jaron_lanier-the-myth-of-ai
- zyxley 12y ago
  > If you prefer 10000000000000000000000000000000000 dust-specks over 1 tortures, then you should prefer to continuously blind someone with dust specks over taking a 0.0000000000000001% chance that they will torture someone. I fail to see how the first part of this sentence implies the second.
  - Strilanc 12y ago
    That is indeed a common place where people disagree. The basic reason it should follow is that not doing so makes you inconsistent and exploitable (e.g. pumping money out of people with the Allais paradox[1]). The fact that humans have preference reversals as you move uncertainties around while holding the expected outcome constant is definitely interesting. Intuitively, it feels really truly right to value things that way. Decision theoretically, it is not optimal. Thus disagreement. Eventually we want to actually mechanize our value systems, for AI. That means nailing everything down in code, without appeal to common sense, and figuring out the consequences of that code. If an AI can change decisions when you vary uncertainties as expected outcomes are held constant, that can introduce failure cases like framing issues or infinite loops of switching back and forth between two outcomes. When writing the code you have to decide if those failures, and the resulting exploitability and inconsistency, are worse than not matching human intuitions in all cases. 1: http://en.wikipedia.org/wiki/Allais_paradox http://en.wikipedia.org/wiki/Allais_paradox

by zyxley 12y ago

TeMPOraL 12y ago

Check out that topic more carefully. As far as I remember, it was a toy example pointing out where the naive application of utilitarianism goes bonkers, and from what I can tell, Eliezer doesn't actually subscribe to "torture over dust specks".

philh 12y ago
I think he does, although he doesn't say so explicitly in the original post. http://lesswrong.com/lw/kn/torture_vs_dust_specks/ http://lesswrong.com/lw/kn/torture_vs_dust_specks/ At any rate, I choose torture, and I think that's the majority view at the LW meetups I attend, and I think we mostly take it for granted that Eliezer chooses torture.
- DanAndersen 12y ago
  Would you be able to help me understand your reasoning? From my perspective, "torture vs dust specks" doesn't seem to be a thing you can just "shut up and multiply" on, because the impact of torture vs the impact of a dust speck seem fundamentally different. To me the negative impact of pain is not just the sensory experience on a purely linear scale, but the effect it has on a person's life even afterward. It seems an overly simplistic model to put it on a linear scale, when there seem to be negative experiences that we easily "bounce back from" and others that we don't. When measuring the negative impacts of a speck in the eye, regarding its impact on any individual's ability to achieve their goals in life or forget about the discomfort in the future, I have trouble seeing how a dust speck is anything but zero in those regards.
  - philh 12y ago
    Basically I think you're just rounding something down to zero which isn't quite zero. It's too small to express, but it's not zero, and when you multiply it by 3^^^3, it becomes far worse than fifty years of torture. Eliezer asked a follow-up question in the thread: would you pay a penny to avoid the dust specks?
    - DanAndersen 12y ago
      Right, that's why I reject the assumption that you can multiply it like that. Imagine a society of 3^^^3 people, first in a universe in which the dust speck thing happened to all of them, and then in one in which it didn't happen. What would we expect to see different between the societies, 1 minute, 1 day, 1 year afterward? Would there be any difference? Any new wars or suffering? The only reasonable outcome in which the dust specks would be worse would be, admittedly, if there's a non-negligible portion of the population that, as a result of getting the dust speck while operating machinery or doing something important like surgery, ends up making some terrible mistake that causes lasting harm to someone or themselves. In that case, the outcome is "hey, some huge number of people got maimed or killed, so the torture of one is OK in comparison." But my thought is that your line of reasoning would still be held even if all 3^^^3 people were given the dust speck when they are sitting alone in their room away from any potentially-dangerous objects. Is that the case? In that particular case I don't see what the dust speck would actually do to any person in the future (beyond a single sensory experience in the present). And to answer your question, I wouldn't pay a penny to avoid the dust speck. Maybe I'm just particularly frugal, being a grad student at the moment.
      - philh 12y ago
        > What would we expect to see different between the societies, 1 minute, 1 day, 1 year afterward? Would there be any difference? Any new wars or suffering? I think you would expect to see differences. You've made 3^^^3 days slightly worse, and there's no way that none of those slightly-worse days had knock-on effects. But I'm also not sure it matters. If something is bad in-the-moment, but has no knock-on effects, that still seems bad. If you could torture me for a minute, then erase my memories and replace them with memories of a minute spent doing what I was previously doing anyway... I would accept that for some amount of money, but not for a penny. Change the game slightly: in one world, torture for fifty years and then the person dies, they don't interact with anyone for any of that time. In the other world, dust specks, but also one person is locked away for fifty years, not interacting with anyone, and then they die. Which do you choose? Sixty years from now, which do you wish you'd chosen? > I wouldn't pay a penny to avoid the dust speck Do you mean just your own dust speck? The question was all 3^^^3 of them. If you do mean all of them... I would pay a penny to avoid dust specks in seven billion eyes, let alone 3^^^3. If one could press a button that would put a dust speck in the eyes of everyone currently alive (at some future time where they're not operating machinery etc.), and in exchange the presser would receive a penny, I would not press that button and I'd be pissed off at anyone who did. (Maybe not anyone, but at least anyone who isn't struggling to live.)
zyxley 12y ago
He does. He says so explicitly in the comments. > I'll go ahead and reveal my answer now: Robin Hanson was correct, I do think that TORTURE is the obvious option, and I think the main instinct behind SPECKS is scope insensitivity.
- TeMPOraL 12y ago
  I stand corrected.
- strnfcpy 12y ago
  If you opt to remove the SPECKS, you would have to change some law of nature or some property of matter, which will have other consequences for the amount of suffering in this world. As you cannot reliably predict or calculate those consequences, you cannot solve this problem.

mori 12y ago

What, preference utilitarianism? Afaik, that does get discussed, especially in the Fun Theory sequence iirc. If you want some more discussion on torture vs. dust specks, I think nostalgebraist on tumblr had a discussion of it with slatestarscratchpad (again, iirc).

Symmetry 12y ago

3^^^3 people isn't "a very large number." The population of Earth or a galaxy full of inhabited planets would be a very large number. 3^^^3 is an incomprehensibly large number which is why it can lead to odd outcomes. If it were merely something like everyone on Earth suffering a dust speck then of course you choose dust. Now, it's equally inconceivable that you would ever be able to affect the lives of 3^^^3 people so if you ever find yourself making a choice between 3^^^3 dust specks and torture you should choose dust because it isn't hard to imagine that your choices could result in a person being tortured. But that's different than the moral judgement in the abstract.

Strilanc 12y ago

It's disingenuous to characterize moral disagreement as ignorance. Eliezer is aware that people think the torture is worse and of the arguments they make to justify it, he simply thinks they're wrong. Why does he think they're wrong? Probably because the resulting decision theories are exploitable or inconsistent in well understood ways when operating under uncertainty. If you prefer 10000000000000000000000000000000000 dust-specks over 1 tortures, then you should prefer to continuously blind someone with dust specks over taking a 0.0000000000000001% chance that they will torture someone. The odds that any given person is about go crazy is more than 0.0000000000000001%, so... better blind everyone. In an uncertain world, setting the badness of dust specks to ε is equivalent to setting it to 0... and setting it to 0 means the "optimal" solution to the system will probably involve throwing a whole lot of dust around. If you want a decision theory that doesn't go around blinding everyone with dust specks, or worse, when given your preferences then you must compromise on torture. > A system that is optimizing a function of n variables, where the objective depends on a subset of size k<n, will often set the remaining unconstrained variables to extreme values; if one of those unconstrained variables is actually something we care about, the solution found may be highly undesirable. http://edge.org/conversation/jaron_lanier-the-myth-of-ai http://edge.org/conversation/jaron_lanier-the-myth-of-ai

zyxley 12y ago
> If you prefer 10000000000000000000000000000000000 dust-specks over 1 tortures, then you should prefer to continuously blind someone with dust specks over taking a 0.0000000000000001% chance that they will torture someone. I fail to see how the first part of this sentence implies the second.
- Strilanc 12y ago
  That is indeed a common place where people disagree. The basic reason it should follow is that not doing so makes you inconsistent and exploitable (e.g. pumping money out of people with the Allais paradox[1]). The fact that humans have preference reversals as you move uncertainties around while holding the expected outcome constant is definitely interesting. Intuitively, it feels really truly right to value things that way. Decision theoretically, it is not optimal. Thus disagreement. Eventually we want to actually mechanize our value systems, for AI. That means nailing everything down in code, without appeal to common sense, and figuring out the consequences of that code. If an AI can change decisions when you vary uncertainties as expected outcomes are held constant, that can introduce failure cases like framing issues or infinite loops of switching back and forth between two outcomes. When writing the code you have to decide if those failures, and the resulting exploitability and inconsistency, are worse than not matching human intuitions in all cases. 1: http://en.wikipedia.org/wiki/Allais_paradox http://en.wikipedia.org/wiki/Allais_paradox