5 ms·

The Roko's Basilisk thing is way overblown. The author addressed the issue here: https://www.reddit.com/r/xkcd/comments/2myg86/xkcd_1450_aibox_experiment/cm8vn6

The Roko's Basilisk thing is way overblown. The author addressed the issue here: https://www.reddit.com/r/xkcd/comments/2myg86/xkcd_1450_aibox_experiment/cm8vn6e https://www.reddit.com/r/xkcd/comments/2myg86/xkcd_1450_aibo... And of course some of the topics he writes about are always going to be controversial. But utilitarianism is hardly radical.

zyxley 12y ago
> Which - keeping in mind that at the time I had absolutely no idea this would all blow up the way it did - caused me to yell quite loudly at Roko for violating ethics given his own premises, I mean really, WTF? You're going to get everyone who reads your article tortured so that you can argue against an AI proposal? In the twisted alternate reality of RationalWiki, this became proof that I believed in Roko's Basilisk, since I yelled at the person who invented it without including twenty lines of disclaimers about what I didn't necessarily believe. I don't understand what Yudkwosky's trying to argue there. "Even though I don't actually believe in it, I yelled at him for invoking the wrath of the nonexistent basilisk that I don't believe in"?
- unbudgingprawn 12y ago
  It seems like he deeply believes in it. I don't think he realizes it's just a modern version of Pascal's wager
  - kenbellows 12y ago
    In attempting to debunk this sort of statement, he has literally written the following: "In the hands of RationalWiki generally, and RationalWiki leader David Gerard particularly ..., [the fictional version of the Roko's Basilisk event] somehow metamorphosed into a Singularity cult that tried to get people to believe a Pascal's Wager argument to donate to their AI god on pain of torture. This cult that has literally never existed anywhere except in the imagination of David Gerard."[1] [1] https://www.reddit.com/r/xkcd/comments/2myg86/xkcd_1450_aibox_experiment/cm8vn6e https://www.reddit.com/r/xkcd/comments/2myg86/xkcd_1450_aibo...
    - unbudgingprawn 12y ago
      "This sentence is a lie." He doesn't seem to actually debunk the idea, just yells about David Gerard for a bit. Can you explain how it's different from Pascal's Wager?
      - Neslit 12y ago
        At least the part where it's implied that "Someone from organization X used the basilisk to convince innocent bystanders to donate to X" is false, for any value of X that I know of. Pascal's Wager is/was (I imagine?) used to get people to subscribe to various churches and give them money.
- TeMPOraL 12y ago
  > caused me to yell quite loudly at Roko for violating ethics given his own premises As far as I understand this part of the sentence, Eliezer got angry at Roko because he did something that in the same thread he stated is unethical.
- mori 12y ago
  Well, yes? If Roko's basilisk was true, it'd be unethical to spread. It turned out to not be true, but if Roko was reasoning from the premise of it being true, or even possibly-true, then the action was still unethical.
- philh 12y ago
  More like, "he believed that he'd accidentally invented an eternal torture machine and then he tried to turn it on, and I yelled at him even though the eternal torture machine didn't work."
  - jtheory 12y ago
    Or maybe -- he posted something that the fact of his posting it revealed he actually intuitively knew it was bunk. See "Belief in belief", which is right near the beginning of the first sequence.... http://lesswrong.com/lw/i4/belief_in_belief/ http://lesswrong.com/lw/i4/belief_in_belief/ It should have been obvious to Roko that either it was a worthless idea (in which case he was safe to post it... but shouldn't, because it's worthless); the other case was that it was a serious, valid idea, in which case he obviously should not post it.
- MBlume 12y ago
  If you think you've discovered a serious info-hazard and your first action is to post it publicly, you are not someone I want frequenting the same online spaces I frequent. Whether that particular info-hazard is, upon examination, remotely hazardous is aside from the point. (If you discover something that scares the crap out of you and you need someone to talk you down/point out why you're wrong/help you sleep at night, you find one very smart, very stable friend and you ask permission and then you talk through it, you do not just publish)
- kbenson 12y ago
  Does changing some of the variables help? I believe there is a secret government agency with the purpose of controlling the entire world will come about. I believe that this agency will control everything, and will punish all people it determines learned about it's possibility and did not either work to make it a reality, or immediately turn themselves in to the authorities. In light of this, it's important to keep knowledge of this future event from people so they aren't targeted. Whoops, I just posted this to a public forum. Regardless of whether the belief is true, if the poster believes it, then they are acting in a reprehensible manner. If you incorrectly believe you have a button that sets off a nuclear warhead in a heavily populated area and choose to press it, you still shoulder some responsibility for your intended actions.
Retric 12y ago
People tend to ignore that for a logical trail to work every stated and unstated assumption needs to be 100% true or it's pointless. Roko's Basilisk is great example where even the slightest weakness in any part of the argument makes the whole idea pointless. (And there are a lot of unstated assumptions.) EX: What's the maximum computational power of the universe?
- convexfunction 12y ago
  Eliezer agrees with you! Roko's Basilisk in particular is wrong. As a matter of policy though, you do not just publicly publish information hazards that might be real; that was why it was a big deal.
  - nsxwolf 12y ago
    You don't what? What policy? Information hazards? What?! I feel like I just stumbled into a Scientology auditing session or something reading your comment.
    - TeMPOraL 12y ago
      It's a concept coined by Nick Bostrom. Basically, some information can be harmful. As a simple, non-controvertial information hazard, imagine you learned an identity of a covert assasin of a foreign government, and you start telling about it to people around you. Don't be surprised if not only you drop dead, but the people you interact with as well. http://www.nickbostrom.com/information-hazards.pdf http://www.nickbostrom.com/information-hazards.pdf http://wiki.lesswrong.com/wiki/Information_hazard http://wiki.lesswrong.com/wiki/Information_hazard
    - convexfunction 12y ago
      You should familiarize yourself with that feeling so you can recognize it later; it's the feeling of falling into an inferential gap. (The thing might be worthless, true, like Scientology. I usually find that resolving confusion is worth my time, though.)
    - 12y ago

zyxley 12y ago

> Which - keeping in mind that at the time I had absolutely no idea this would all blow up the way it did - caused me to yell quite loudly at Roko for violating ethics given his own premises, I mean really, WTF? You're going to get everyone who reads your article tortured so that you can argue against an AI proposal? In the twisted alternate reality of RationalWiki, this became proof that I believed in Roko's Basilisk, since I yelled at the person who invented it without including twenty lines of disclaimers about what I didn't necessarily believe. I don't understand what Yudkwosky's trying to argue there. "Even though I don't actually believe in it, I yelled at him for invoking the wrath of the nonexistent basilisk that I don't believe in"?

unbudgingprawn 12y ago
It seems like he deeply believes in it. I don't think he realizes it's just a modern version of Pascal's wager
- kenbellows 12y ago
  In attempting to debunk this sort of statement, he has literally written the following: "In the hands of RationalWiki generally, and RationalWiki leader David Gerard particularly ..., [the fictional version of the Roko's Basilisk event] somehow metamorphosed into a Singularity cult that tried to get people to believe a Pascal's Wager argument to donate to their AI god on pain of torture. This cult that has literally never existed anywhere except in the imagination of David Gerard."[1] [1] https://www.reddit.com/r/xkcd/comments/2myg86/xkcd_1450_aibox_experiment/cm8vn6e https://www.reddit.com/r/xkcd/comments/2myg86/xkcd_1450_aibo...
  - unbudgingprawn 12y ago
    "This sentence is a lie." He doesn't seem to actually debunk the idea, just yells about David Gerard for a bit. Can you explain how it's different from Pascal's Wager?
    - Neslit 12y ago
      At least the part where it's implied that "Someone from organization X used the basilisk to convince innocent bystanders to donate to X" is false, for any value of X that I know of. Pascal's Wager is/was (I imagine?) used to get people to subscribe to various churches and give them money.
TeMPOraL 12y ago
> caused me to yell quite loudly at Roko for violating ethics given his own premises As far as I understand this part of the sentence, Eliezer got angry at Roko because he did something that in the same thread he stated is unethical.
mori 12y ago
Well, yes? If Roko's basilisk was true, it'd be unethical to spread. It turned out to not be true, but if Roko was reasoning from the premise of it being true, or even possibly-true, then the action was still unethical.
philh 12y ago
More like, "he believed that he'd accidentally invented an eternal torture machine and then he tried to turn it on, and I yelled at him even though the eternal torture machine didn't work."
- jtheory 12y ago
  Or maybe -- he posted something that the fact of his posting it revealed he actually intuitively knew it was bunk. See "Belief in belief", which is right near the beginning of the first sequence.... http://lesswrong.com/lw/i4/belief_in_belief/ http://lesswrong.com/lw/i4/belief_in_belief/ It should have been obvious to Roko that either it was a worthless idea (in which case he was safe to post it... but shouldn't, because it's worthless); the other case was that it was a serious, valid idea, in which case he obviously should not post it.
MBlume 12y ago
If you think you've discovered a serious info-hazard and your first action is to post it publicly, you are not someone I want frequenting the same online spaces I frequent. Whether that particular info-hazard is, upon examination, remotely hazardous is aside from the point. (If you discover something that scares the crap out of you and you need someone to talk you down/point out why you're wrong/help you sleep at night, you find one very smart, very stable friend and you ask permission and then you talk through it, you do not just publish)
kbenson 12y ago
Does changing some of the variables help? I believe there is a secret government agency with the purpose of controlling the entire world will come about. I believe that this agency will control everything, and will punish all people it determines learned about it's possibility and did not either work to make it a reality, or immediately turn themselves in to the authorities. In light of this, it's important to keep knowledge of this future event from people so they aren't targeted. Whoops, I just posted this to a public forum. Regardless of whether the belief is true, if the poster believes it, then they are acting in a reprehensible manner. If you incorrectly believe you have a button that sets off a nuclear warhead in a heavily populated area and choose to press it, you still shoulder some responsibility for your intended actions.

Retric 12y ago

People tend to ignore that for a logical trail to work every stated and unstated assumption needs to be 100% true or it's pointless. Roko's Basilisk is great example where even the slightest weakness in any part of the argument makes the whole idea pointless. (And there are a lot of unstated assumptions.) EX: What's the maximum computational power of the universe?

convexfunction 12y ago
Eliezer agrees with you! Roko's Basilisk in particular is wrong. As a matter of policy though, you do not just publicly publish information hazards that might be real; that was why it was a big deal.
- nsxwolf 12y ago
  You don't what? What policy? Information hazards? What?! I feel like I just stumbled into a Scientology auditing session or something reading your comment.
  - TeMPOraL 12y ago
    It's a concept coined by Nick Bostrom. Basically, some information can be harmful. As a simple, non-controvertial information hazard, imagine you learned an identity of a covert assasin of a foreign government, and you start telling about it to people around you. Don't be surprised if not only you drop dead, but the people you interact with as well. http://www.nickbostrom.com/information-hazards.pdf http://www.nickbostrom.com/information-hazards.pdf http://wiki.lesswrong.com/wiki/Information_hazard http://wiki.lesswrong.com/wiki/Information_hazard
  - convexfunction 12y ago
    You should familiarize yourself with that feeling so you can recognize it later; it's the feeling of falling into an inferential gap. (The thing might be worthless, true, like Scientology. I usually find that resolving confusion is worth my time, though.)
  - 12y ago