11 ms·

Careful. Yudkowski and LW have a dark side. (google Roko's Basilisk) Rationality applied to ethics is a sticky wicket. You have to subscribe to some metric of

• Moshe_Silnorin 12y ago
  Are you attempting reverse psychology?
  • biesnecker 12y ago
    I'm pretty sure it's a typo, but it's fun thinking about what reverse physiology would even be.
    • zyxley 12y ago
      Well, if you reverse everything, you've got predominantly left-handed nega-humans made out of antimatter.
• Houshalter 12y ago
  The Roko's Basilisk thing is way overblown. The author addressed the issue here: https://www.reddit.com/r/xkcd/comments/2myg86/xkcd_1450_aibox_experiment/cm8vn6e https://www.reddit.com/r/xkcd/comments/2myg86/xkcd_1450_aibo... And of course some of the topics he writes about are always going to be controversial. But utilitarianism is hardly radical.
  • zyxley 12y ago
    > Which - keeping in mind that at the time I had absolutely no idea this would all blow up the way it did - caused me to yell quite loudly at Roko for violating ethics given his own premises, I mean really, WTF? You're going to get everyone who reads your article tortured so that you can argue against an AI proposal? In the twisted alternate reality of RationalWiki, this became proof that I believed in Roko's Basilisk, since I yelled at the person who invented it without including twenty lines of disclaimers about what I didn't necessarily believe. I don't understand what Yudkwosky's trying to argue there. "Even though I don't actually believe in it, I yelled at him for invoking the wrath of the nonexistent basilisk that I don't believe in"?
    • unbudgingprawn 12y ago
      It seems like he deeply believes in it. I don't think he realizes it's just a modern version of Pascal's wager
      • kenbellows 12y ago
        In attempting to debunk this sort of statement, he has literally written the following: "In the hands of RationalWiki generally, and RationalWiki leader David Gerard particularly ..., [the fictional version of the Roko's Basilisk event] somehow metamorphosed into a Singularity cult that tried to get people to believe a Pascal's Wager argument to donate to their AI god on pain of torture. This cult that has literally never existed anywhere except in the imagination of David Gerard."[1] [1] https://www.reddit.com/r/xkcd/comments/2myg86/xkcd_1450_aibox_experiment/cm8vn6e https://www.reddit.com/r/xkcd/comments/2myg86/xkcd_1450_aibo...
        • unbudgingprawn 12y ago
          "This sentence is a lie." He doesn't seem to actually debunk the idea, just yells about David Gerard for a bit. Can you explain how it's different from Pascal's Wager?
          • Neslit 12y ago
            At least the part where it's implied that "Someone from organization X used the basilisk to convince innocent bystanders to donate to X" is false, for any value of X that I know of. Pascal's Wager is/was (I imagine?) used to get people to subscribe to various churches and give them money.
    • TeMPOraL 12y ago
      > caused me to yell quite loudly at Roko for violating ethics given his own premises As far as I understand this part of the sentence, Eliezer got angry at Roko because he did something that in the same thread he stated is unethical.
    • 12y ago
  • Retric 12y ago
    People tend to ignore that for a logical trail to work every stated and unstated assumption needs to be 100% true or it's pointless. Roko's Basilisk is great example where even the slightest weakness in any part of the argument makes the whole idea pointless. (And there are a lot of unstated assumptions.) EX: What's the maximum computational power of the universe?
    • convexfunction 12y ago
      Eliezer agrees with you! Roko's Basilisk in particular is wrong. As a matter of policy though, you do not just publicly publish information hazards that might be real; that was why it was a big deal.
      • nsxwolf 12y ago
        You don't what? What policy? Information hazards? What?! I feel like I just stumbled into a Scientology auditing session or something reading your comment.
        • TeMPOraL 12y ago
          It's a concept coined by Nick Bostrom. Basically, some information can be harmful. As a simple, non-controvertial information hazard, imagine you learned an identity of a covert assasin of a foreign government, and you start telling about it to people around you. Don't be surprised if not only you drop dead, but the people you interact with as well. http://www.nickbostrom.com/information-hazards.pdf http://www.nickbostrom.com/information-hazards.pdf http://wiki.lesswrong.com/wiki/Information_hazard http://wiki.lesswrong.com/wiki/Information_hazard
        • convexfunction 12y ago
          You should familiarize yourself with that feeling so you can recognize it later; it's the feeling of falling into an inferential gap. (The thing might be worthless, true, like Scientology. I usually find that resolving confusion is worth my time, though.)
        • 12y ago
• TeMPOraL 12y ago
  The lesson I learned from Roko's Basilisk event that I think Eliezer and most of the LW community also endorses is that if you're not careful, applying some models too far can have those "extreme, nay dire consequences for certain thought experiments" - so it's best not to commit too far to those methods, because they may have some caveats, or you might just have applied them wrong.
  • maxerickson 12y ago
    Yet in this same thread here there is a discussion of torture and dust specks. As if the mystical power to prevent dust specks is close enough to reality to be interesting. I wonder how many people that fed that question into their mentation have considered sabotaging a coal plant. Obviously, that's ethically more complicated than the thought experiment, but it follows the outline. (If you believe that the substitutes for coal power are available and superior, then taking it offline should still be a net good, one that is distributed over a great many people, with the downside (jail) concentrated on the saboteur.)
    • TeMPOraL 12y ago
      It's actually a fair question. But 1) I don't believe substitutes for coal power are immediately available, and taking out a plant would likely lead to many deaths (think: accidents caused by sudden blackout, rural hospitals without enough fuel for backup power generators) and inconvenience (think: areas with interrupted electrical service for months as people try to fix the grid and bring in additional power sources); 2) downsides of a coal plant aren't that bad in comparison. Note that dustspeck vs. torture example involved 3^^^3 dustspecks, which is more than number of atoms in the universe raised to the power of number of atoms in the universe. It was purposefully chosen as such an extreme number, as to clearly outweight the utilitarian calculation in favour of torture - so that the example could be used to study our moral intuitions.
      • maxerickson 12y ago
        My point is largely that I don't think you learn much until you actually go ahead and twist the knife yourself (or not!). Short of that, it's just a mind puzzle, "how does what I think my beliefs are match up with what I think my beliefs are" or so.
• zyxley 12y ago
  Yudkowsky's "good", at that of his fans, also tends to be naively utilitarian, ignoring modern development of that philosophy (see his whole "it's better for one person to be tortured for 50 years than for a very large number of people to each get one speck of dust in their eye" thing, for example).
  • TeMPOraL 12y ago
    Check out that topic more carefully. As far as I remember, it was a toy example pointing out where the naive application of utilitarianism goes bonkers, and from what I can tell, Eliezer doesn't actually subscribe to "torture over dust specks".
    • philh 12y ago
      I think he does, although he doesn't say so explicitly in the original post. http://lesswrong.com/lw/kn/torture_vs_dust_specks/ http://lesswrong.com/lw/kn/torture_vs_dust_specks/ At any rate, I choose torture, and I think that's the majority view at the LW meetups I attend, and I think we mostly take it for granted that Eliezer chooses torture.
      • DanAndersen 12y ago
        Would you be able to help me understand your reasoning? From my perspective, "torture vs dust specks" doesn't seem to be a thing you can just "shut up and multiply" on, because the impact of torture vs the impact of a dust speck seem fundamentally different. To me the negative impact of pain is not just the sensory experience on a purely linear scale, but the effect it has on a person's life even afterward. It seems an overly simplistic model to put it on a linear scale, when there seem to be negative experiences that we easily "bounce back from" and others that we don't. When measuring the negative impacts of a speck in the eye, regarding its impact on any individual's ability to achieve their goals in life or forget about the discomfort in the future, I have trouble seeing how a dust speck is anything but zero in those regards.
        • 12y ago
• philh 12y ago
  > EY's definition of good is ruthlessly utilitarian, Note that people frequently misunderstand Eliezer, especially wrt the basilisk. (Though it's not clear whether you're thinking about the basilisk in this paragraph, so that "especially" may not be relevant.) With this in mind, can you give an example of a specific moral claim where you think you disagree with Eliezer?
  • Strilanc 12y ago
    (Different person here.) The most common example I see is the dust-specks vs torture thing[1], where Eliezer thinks increasing dust specks eventually outweighs the torture and various other people think torture outweighs any number of dust specks. Eliezer is basing his position on taking a few decision theory axioms as obviously true, and others are basing their position on taking their moral intuitions as obviously true (i.e. a "One's modus ponens is another's modus tollens" situation). 1: http://lesswrong.com/lw/kn/torture_vs_dust_specks/ http://lesswrong.com/lw/kn/torture_vs_dust_specks/
    • spacehome 12y ago
      Yea, I agree with your interpretation on the root cause of the disagreement. I don't mean to weigh in one way or the other, but I would like to point out that people's mathematical intuition when it comes to infinity or obscenely large numbers can be arbitrarily wrong. It takes a lot of training to even recognize the mistakes you're making, let alone correct for them.
  • pyrois 12y ago
    I'll give it a shot, but first, a list of disclaimers: Yudkowsky gets a ton of unjustified internet hate and scorn, and I disagree with a lot of it. I read a number of the sequences, and quite enjoyed them. I also think his reaction to Roko's Basilisk was pretty reasonable: someone on your form comes up with a way to basically guarantee eternal torture for anyone who reads it, and then posts it, thus guaranteeing eternal torture for your forum readers? Who cares that the idea won't actually guarantee any such thing, Roko _thought_ that it might; I would be pissed as hell. Anyway, my point is, I'm only offering a gentle and hopefully reasoned disagreement with someone I regard highly. I'm not jumping on the "fuck that guy" train. Moving on. There was a post that boiled down to the question "Would you rather 3 ||| 3 people (where | is an ascii stand-in for Knuth's up-arrow notation) get a mote of dust in their eye, or one person be horrifically tortured for 50 years?" and his conclusion was basically "you can use math to assign some incredibly small epsilon of suffering to getting a mote of dust in your eye, but eventually, if you sum enough people, it's more suffering overall than one poor person getting horrifically tortured". My problem with his post was that I think that any morality scheme that results in the person getting tortured is fundamentally flawed, and I don't care how much math you throw at me to try and "prove" that it's better. I don't think I've ever heard of an attempt to rigorously derive morality that I agree with. Morality is too contextual, too messy, for us to perfectly capture in those sorts of models. It's especially bad when we try and model morality mathematically, and then take as gospel the result of that model, rather than say "oh, um, that's not a great result, the model must be wrong".
• maxerickson 12y ago
  If you take seriously the idea of an infohazard, you have a problem, because the idea that information can be hazardous is itself hazardous information (at a minimum, it's a prompt for us sloppy thinkers to start imagining what sort of information might be hazardous). Something as innocuous as "Think safe thoughts!" enjoys a similar property.
  • ikeboy 12y ago
    And yet you posted this to the internet. Some people never learn, I guess.
    • maxerickson 12y ago
      Anything that somehow includes my saying "Think safe thoughts!" in a causal chain is already essentially inevitable. The future is not easily shaped.
      • ikeboy 12y ago
        What you posted, according to your own premises, is worse than just saying "Think safe thoughts!" If I modded this, I would delete your post.
        • maxerickson 12y ago
          What are my premises? If I don't take info hazards as a serious threat, there isn't anything to sort those statements with.
  • jtheory 12y ago
    That doesn't make sense, though -- info hazards are not normally things you might just think of, or we'd be screwed either way; "whatever you do, don't think of a pink elephant", right? Toss out silly ideas like the basilisk thing, and think up real info hazard possibilities. E.g., let's say you knew what Snowden was up to, a few weeks before he figured out how to publish what he'd found and leave the US. You'd have done a lot of harm to post that info publicly. Suppose you find a laptop left in a taxi, with police records including personal details of rape victims in your city for the past 20 years. Post it to 4chan? Uh, no. What about ongoing undercover investigations? Post it? You accidentally shoulder-surf sensitive credentials -- bank manager's login, nuclear plant access code, whatever. Post them online? Simple ways to make highly dangerous weapons, drugs, poisons, etc. can also be info hazards. It's honestly not that hard to imagine real-life examples, without having the dangerous info in question or descending into government assassin-type scenarios. It's just any data that, if you make it generally available, will predictably result in bad things happening. In some cases -- like revealing important security flaws -- a public disclosure may be a last resort if less harmful methods have failed, but it's likely not an ethical first resort in those cases.
• danbmil99 12y ago
  I think it's interesting, and indicative of the subject, that a severely downvoted comment of mine has a huge thread of discussion under it.