5 ms·

The second half of the article that talks about Yudkowsky, rationalists, and Roko's Basilisk is absolute trash and a gross mischaracterization of the actual pos

The second half of the article that talks about Yudkowsky, rationalists, and Roko's Basilisk is absolute trash and a gross mischaracterization of the actual positions taken. >Yudkowski has for more than a decade pursued the possibility of perfect human reasoning His website is literally called "Less Wrong". It's fundamentally a quest for improvement, not perfection. >His system of coldly logical reason, it turned out, was by many accounts completely undone by a logical paradox known as Roko’s Basilisk. Roko's Basilisk is a thought experiment designed to import your "don't negotiate with terrorists" intuitions into a really weird corner of decision theory. It's possibly why prior decision theories held were rejected as flawed, but it's not a current issue with their logic. Roko's Basilisk has, unfortunately, gotten way more coverage and fame than it deserves in terms of actual importance. This is because of the regretful (but understandable) decision to censor discussion of it on Less Wrong, which backfired spectacularly. >For super-nerd bonus points, it’s also arguably a spin on Godel’s incompleteness theorem, which argues that no purely rational algorithmic system can completely and consistently model reality, or prove its own rationality. That's not at all what Godel's incompleteness theorem argues. That's much more narrowly about formal logic.

darkpuma 8y ago
> "This is because of the regretful (but understandable) decision to censor discussion of it on Less Wrong, which backfired spectacularly." Can you explain the (understandable) rationale behind Less Wrong censoring discussion of Roko's Basilisk? I've been lead to believe it was censored out of fear that discussing it could somehow inspire its actual creation. Honestly from a distance Less Wrong has always struck me as vaguely cult-like, but I'm open to the possibility I've just gotten the wrong impression.
- deleted 8y ago
  [deleted]
- chongli 8y ago
  I think the more charitable interpretation is that the discussion was taking over everything. If people are fixating on a narrow topic for a long time (without much progress) then sometimes it's helpful to say "alright people, let's move along now." You're right about Less Wrong feeling like a cult, though. It gives me the heebie-jeebies.
  - cma 8y ago
    >You're right about Less Wrong feeling like a cult, though. It gives me the heebie-jeebies. They even have a succession plan similar to finding the next Dalai Lama: https://www.youtube.com/watch?v=d9_2qhkWHN8 https://www.youtube.com/watch?v=d9_2qhkWHN8
- ThrustVectoring 8y ago
  Sure thing. The basilisk can easily be misread as an argument to do everything in one's power to support AI research of some sort. It's more accurately an argument that you should ensure your decision theory doesn't coerce you into putting all your effort into supporting AI research. LessWrong has a bunch of really scrupulous people reading it, who take pride in taking ideas seriously, so censoring the discussion seemed worthwhile for avoiding the failure mode of having readers make the mistake mentioned previously.
- tunesmith 8y ago
  The entry on the less wrong wiki has some info on it: https://wiki.lesswrong.com/wiki/Roko%27s_basilisk https://wiki.lesswrong.com/wiki/Roko%27s_basilisk
- kerbalspacepro 8y ago
  Cults don't write about how to avoid being a cult, while the LW crowd has reams of texts and blog posts about how they are purposefully avoiding trying to become a cult.
  - saalweachter 8y ago
    Cults do talk quite a bit about why they're not a cult.
    - TeMPOraL 8y ago
      But I haven't seen cults talking about how to personally ensure you're not falling into trap of cult-like thinking. LW is a rather self-conscious community.
      - diego898 8y ago
        Sorry for the tangent, but from the outside I've also felt very "outside" the LW community. I'm not exactly sure how to "join" it. I know that I'm supposed to read the various required readings first, but then what? How do I see what LW is "talking about" or "fighting about" at a point in time? I mean this in a serious, logistical sense. Thanks!
        ThrustVectoring 8y ago
        The LW community has decentralized and deformalized things to a huge extent. It's not just you - the logistics of participating are much less clear than it used to be, and the whole system is far less legible. Honestly the best strategy is attend local meet-ups, come across as intelligent and interested to the people who hold gatekeeping power over the informal non-public groups, and get in the loop that way. This is significantly easier in Berkeley, where a lot of folks end up moving to. You can also skip some of these steps by attending a CFAR workshop, which will both introduce you to many members of the LW community and put you on an alumni-only mailing list which houses a chunk of the general discussion.
        8y ago
- Obi_Juan_Kenobi 8y ago
  I don't fuck with LW, but SCC definitely has that culty vibe. Overall I think Scott has some interesting articles and the general idea of rationalism is .. fine. But that's not at all what the communities around it are like. You just get tons of 'rational racism' with HBD (human biodiversity for those not in the know, which is basically just race realism with a nerdy veneer), IQ worship, and a group that collectively abandons anything like the rationalist ideal and becomes a post-hoc clusterfuck. The community can almost be thought of like these racist AIs: garbage in, garbage out. They cling to any bit of rational-sounding data that supports their preconceptions, and have to adopt severe anti-academic stances to reject the mountains of data that contradict their views. I suppose the deeper rot is that these rationalists fancy themselves like scientists, able to interact with primary data and draw conclusions, but any actual expert (i.e. someone that has performed research and published it in a community of peers, not whacko journals) is just confused how anyone could be so wrong. You regularly see posts where someone comes in who actually knows what they're talking about, authoritatively shows that all the nonsense they've been talking about makes no sense, and at best they get a 'huh, interesting' before the nonsense picks back up. The idea of a rationalist community is to organize around questioning your own assumptions, confronting diverse views, and engaging in good-faith debate. But the reality is that basically none of that happens for anything controversial to the community, so it's just a veneer of legitimacy for in-group circle-jerking. And it just so happens that the in-group tends to be pretty damn racist.
- adlkjnndnnd 8y ago
  My guess would have been that by discussing it, you leave a trace in the networks allowing an eventual AI to prove that you knew about it. Since you knew about it, it can hold you accountable, and take revenge on all of your descendants if you didn't devote sufficient energy on bringing the AI into existence.
- Jach 8y ago
  Read Eliezer himself, 4 years ago: https://www.reddit.com/r/xkcd/comments/2myg86/xkcd_1450_aibox_experiment/cm8vn6e/ https://www.reddit.com/r/xkcd/comments/2myg86/xkcd_1450_aibo... The "meat" of interest here for those too lazy to click (I encourage people click through, though, really, and maybe even take a glance at https://arxiv.org/abs/1401.5577 https://arxiv.org/abs/1401.5577) is probably this long parenthetical: > (But taking Roko's premises at face value, his idea would zap people as soon as they read it. Which - keeping in mind that at the time I had absolutely no idea this would all blow up the way it did - caused me to yell quite loudly at Roko for violating ethics given his own premises, I mean really, WTF? You're going to get everyone who reads your article tortured so that you can argue against an AI proposal? In the twisted alternate reality of RationalWiki, this became proof that I believed in Roko's Basilisk, since I yelled at the person who invented it without including twenty lines of disclaimers about what I didn't necessarily believe. And since I had no idea this would blow up that way at the time, I suppose you could even read the sentences I wrote that way, which I did not edit for hours first because I had no idea this was going to haunt me for years to come. And then, since Roko's Basilisk was a putatively a pure infohazard of no conceivable use or good to anyone, and since I didn't really want to deal with the argument, I deleted it from LessWrong which seemed to me like a perfectly good general procedure for dealing with putative pure infohazards that jerkwads were waving in people's faces. Which brought out the censorship!! trolls and was certainly, in retrospect, a mistake.) Edit: this plus more is all covered by the LW wiki's page someone else linked, check that out too if you really care. https://wiki.lesswrong.com/wiki/Roko%27s_basilisk https://wiki.lesswrong.com/wiki/Roko%27s_basilisk
TeMPOraL 8y ago
Also the neoreactionary bit. I literally paused on that in total surprise. As a long-time LW lurker - yes, the rationalist community plays with plenty of non-mainstream ideas and try to work them in both directions (up from first principles, and towards extreme conclusions). But characterizing rationalists as equal to neoreactionaries? That's a complete mischaracterization of the community.
- Obi_Juan_Kenobi 8y ago
  If you don't see the NRs in the rationalist community, you're not looking. Ultimately I agree that they're not equal; I think many people involved in LW or SSC are good-faith rationalists. But there's also a strong segment that has adopted - post hoc - a crazy set of priors to support NR views. What strikes me is how antithetical this post-hoc "pick your priors, any priors!" is to the rationalist ideal, but it doesn't get called out. It's given equal treatment, and it means that the community is categorically incapable of discussing culture-war issues.
  - ThrustVectoring 8y ago
    There are neoreactionaries, but they aren't anywhere near as important or central as the article implies. A huge chunk of this is the fault of the discourse norms, which allow for a lot of discussions that would be prohibited or highly discouraged elsewhere.

darkpuma 8y ago

> "This is because of the regretful (but understandable) decision to censor discussion of it on Less Wrong, which backfired spectacularly." Can you explain the (understandable) rationale behind Less Wrong censoring discussion of Roko's Basilisk? I've been lead to believe it was censored out of fear that discussing it could somehow inspire its actual creation. Honestly from a distance Less Wrong has always struck me as vaguely cult-like, but I'm open to the possibility I've just gotten the wrong impression.

deleted 8y ago
[deleted]
chongli 8y ago
I think the more charitable interpretation is that the discussion was taking over everything. If people are fixating on a narrow topic for a long time (without much progress) then sometimes it's helpful to say "alright people, let's move along now." You're right about Less Wrong feeling like a cult, though. It gives me the heebie-jeebies.
- cma 8y ago
  >You're right about Less Wrong feeling like a cult, though. It gives me the heebie-jeebies. They even have a succession plan similar to finding the next Dalai Lama: https://www.youtube.com/watch?v=d9_2qhkWHN8 https://www.youtube.com/watch?v=d9_2qhkWHN8
ThrustVectoring 8y ago
Sure thing. The basilisk can easily be misread as an argument to do everything in one's power to support AI research of some sort. It's more accurately an argument that you should ensure your decision theory doesn't coerce you into putting all your effort into supporting AI research. LessWrong has a bunch of really scrupulous people reading it, who take pride in taking ideas seriously, so censoring the discussion seemed worthwhile for avoiding the failure mode of having readers make the mistake mentioned previously.
tunesmith 8y ago
The entry on the less wrong wiki has some info on it: https://wiki.lesswrong.com/wiki/Roko%27s_basilisk https://wiki.lesswrong.com/wiki/Roko%27s_basilisk
kerbalspacepro 8y ago
Cults don't write about how to avoid being a cult, while the LW crowd has reams of texts and blog posts about how they are purposefully avoiding trying to become a cult.
- saalweachter 8y ago
  Cults do talk quite a bit about why they're not a cult.
  - TeMPOraL 8y ago
    But I haven't seen cults talking about how to personally ensure you're not falling into trap of cult-like thinking. LW is a rather self-conscious community.
    - diego898 8y ago
      Sorry for the tangent, but from the outside I've also felt very "outside" the LW community. I'm not exactly sure how to "join" it. I know that I'm supposed to read the various required readings first, but then what? How do I see what LW is "talking about" or "fighting about" at a point in time? I mean this in a serious, logistical sense. Thanks!
      - ThrustVectoring 8y ago
        The LW community has decentralized and deformalized things to a huge extent. It's not just you - the logistics of participating are much less clear than it used to be, and the whole system is far less legible. Honestly the best strategy is attend local meet-ups, come across as intelligent and interested to the people who hold gatekeeping power over the informal non-public groups, and get in the loop that way. This is significantly easier in Berkeley, where a lot of folks end up moving to. You can also skip some of these steps by attending a CFAR workshop, which will both introduce you to many members of the LW community and put you on an alumni-only mailing list which houses a chunk of the general discussion.
      - 8y ago
Obi_Juan_Kenobi 8y ago
I don't fuck with LW, but SCC definitely has that culty vibe. Overall I think Scott has some interesting articles and the general idea of rationalism is .. fine. But that's not at all what the communities around it are like. You just get tons of 'rational racism' with HBD (human biodiversity for those not in the know, which is basically just race realism with a nerdy veneer), IQ worship, and a group that collectively abandons anything like the rationalist ideal and becomes a post-hoc clusterfuck. The community can almost be thought of like these racist AIs: garbage in, garbage out. They cling to any bit of rational-sounding data that supports their preconceptions, and have to adopt severe anti-academic stances to reject the mountains of data that contradict their views. I suppose the deeper rot is that these rationalists fancy themselves like scientists, able to interact with primary data and draw conclusions, but any actual expert (i.e. someone that has performed research and published it in a community of peers, not whacko journals) is just confused how anyone could be so wrong. You regularly see posts where someone comes in who actually knows what they're talking about, authoritatively shows that all the nonsense they've been talking about makes no sense, and at best they get a 'huh, interesting' before the nonsense picks back up. The idea of a rationalist community is to organize around questioning your own assumptions, confronting diverse views, and engaging in good-faith debate. But the reality is that basically none of that happens for anything controversial to the community, so it's just a veneer of legitimacy for in-group circle-jerking. And it just so happens that the in-group tends to be pretty damn racist.
adlkjnndnnd 8y ago
My guess would have been that by discussing it, you leave a trace in the networks allowing an eventual AI to prove that you knew about it. Since you knew about it, it can hold you accountable, and take revenge on all of your descendants if you didn't devote sufficient energy on bringing the AI into existence.
Jach 8y ago
Read Eliezer himself, 4 years ago: https://www.reddit.com/r/xkcd/comments/2myg86/xkcd_1450_aibox_experiment/cm8vn6e/ https://www.reddit.com/r/xkcd/comments/2myg86/xkcd_1450_aibo... The "meat" of interest here for those too lazy to click (I encourage people click through, though, really, and maybe even take a glance at https://arxiv.org/abs/1401.5577 https://arxiv.org/abs/1401.5577) is probably this long parenthetical: > (But taking Roko's premises at face value, his idea would zap people as soon as they read it. Which - keeping in mind that at the time I had absolutely no idea this would all blow up the way it did - caused me to yell quite loudly at Roko for violating ethics given his own premises, I mean really, WTF? You're going to get everyone who reads your article tortured so that you can argue against an AI proposal? In the twisted alternate reality of RationalWiki, this became proof that I believed in Roko's Basilisk, since I yelled at the person who invented it without including twenty lines of disclaimers about what I didn't necessarily believe. And since I had no idea this would blow up that way at the time, I suppose you could even read the sentences I wrote that way, which I did not edit for hours first because I had no idea this was going to haunt me for years to come. And then, since Roko's Basilisk was a putatively a pure infohazard of no conceivable use or good to anyone, and since I didn't really want to deal with the argument, I deleted it from LessWrong which seemed to me like a perfectly good general procedure for dealing with putative pure infohazards that jerkwads were waving in people's faces. Which brought out the censorship!! trolls and was certainly, in retrospect, a mistake.) Edit: this plus more is all covered by the LW wiki's page someone else linked, check that out too if you really care. https://wiki.lesswrong.com/wiki/Roko%27s_basilisk https://wiki.lesswrong.com/wiki/Roko%27s_basilisk

TeMPOraL 8y ago

Also the neoreactionary bit. I literally paused on that in total surprise. As a long-time LW lurker - yes, the rationalist community plays with plenty of non-mainstream ideas and try to work them in both directions (up from first principles, and towards extreme conclusions). But characterizing rationalists as equal to neoreactionaries? That's a complete mischaracterization of the community.

Obi_Juan_Kenobi 8y ago
If you don't see the NRs in the rationalist community, you're not looking. Ultimately I agree that they're not equal; I think many people involved in LW or SSC are good-faith rationalists. But there's also a strong segment that has adopted - post hoc - a crazy set of priors to support NR views. What strikes me is how antithetical this post-hoc "pick your priors, any priors!" is to the rationalist ideal, but it doesn't get called out. It's given equal treatment, and it means that the community is categorically incapable of discussing culture-war issues.
- ThrustVectoring 8y ago
  There are neoreactionaries, but they aren't anywhere near as important or central as the article implies. A huge chunk of this is the fault of the discourse norms, which allow for a lot of discussions that would be prohibited or highly discouraged elsewhere.