5 ms·

This seems like an awfully uncharitable and cherry-picked treatment of a community of well-intentioned and occasionally eccentric disagreeable online nerds. You

This seems like an awfully uncharitable and cherry-picked treatment of a community of well-intentioned and occasionally eccentric disagreeable online nerds. You're effectively just noticing that a community that actively aims to overcome bias is not free from bias itself, but no one within that community would ever deny that.

cogman10 3mo ago
I think the issue with the community is they take their eccentric beliefs as being gospel truths which can lead to truly unhinged and irrational beliefs and actions (see: the Zizians). That's what can make the rationalist movement feel more like a religious cult and less like a bunch of eccentrics. As an outsider looking in, it seems like the entire movement is filled with just accepting priors with very little evidence for why those priors should be accepted.
- FeepingCreature 3mo ago
  Consider a community filled with 100 people; 95 of them who don't take their eccentric beliefs as gospel truths and 5 who do. Those 5 will inevitably become the public face of the community when they make the news. However there is very little that the 95 could have done to stop them.
  - astrange 3mo ago
    The entire point of rationalism is to take your own beliefs seriously and follow them as if you were a robot. (Except that they don't claim them to be "their own beliefs" but instead call them "priors".)
  - cogman10 3mo ago
    > However there is very little that the 95 could have done to stop them. How about the 95% stop having gatherings where they all proclaim that their beliefs are true even if they don't seriously believe them? How is the 5% supposed to know that the 95% is actually larping belief? And how is an outsider who observes the 95% espousing their whacky beliefs supposed to know who is a member of the 95% and who is a member of the 5% (and if those percentages are accurate). You can observe the same dynamic in any church. Everyone in the congregation will have different levels of belief, but when it comes to sharing a testimony you'll be hard pressed to distinguish between who the true believers are and who's just "faking it until you make it".
    - zahlman 3mo ago
      > gatherings where they all proclaim that their beliefs are true even if they don't seriously believe them? What on Earth are you referring to? What gatherings? What beliefs? What proclamations? What evidence of insincerity?
      - cogman10 3mo ago
        > What gatherings? Rationalist have groups meetup and conferences. For example [1] > What beliefs? It's pretty diverse and not uniformly held. Some of the more strange ones that seem to have pretty wide appeal are around AI safety. For example, Roko's basilisk. > What proclamations? What evidence of insincerity? I was responding to the comment above me that suggested that 95% don't actually hold rationalist beliefs and are instead simply having fun with it. Only a small 5% are actually sincere. [1] https://www.reddit.com/r/slatestarcodex/comments/1car63w/lessonline_rationalist_festival_may_31st_june_2nd/ https://www.reddit.com/r/slatestarcodex/comments/1car63w/les...
        zahlman 3mo ago
        Okay, but those pieces don't establish your claim. You would need a Roko's basilisk-like concept that only 5% of the community agree with, and a "group meetup" or "conference" where "they all" (i.e.: also the 95%) "proclaim that their beliefs (i.e.: including the basilisk thing) are true". If I observed such a gathering knowing that 5% of the people there have sincerely come to a particular conclusion based on the available evidence and the other 95% think the evidence doesn't support that conclusion, I would be extremely surprised to see everyone going around "larping belief". And I don't think you can actually show such a thing occurring.
        cogman10 3mo ago
        Which claim exactly did I make which you are refuting? Please actually quote it. > I would be extremely surprised to see everyone going around "larping belief". And I don't think you can actually show such a thing occurring. I never claimed I could or that I observed such a thing. The 95/5% didn't come from me, it came from the person I responded to. My point was that - It's impossible to distinguish who's a true believer and who isn't. - That the meetups of the rationalists re-enforce beliefs, particularly in true believers. Also that these beliefs of the rationalists can be pretty whacky and based on a jenga tower of priors. - That there isn't "nothing" a rational person could do to dissuade someone from a whacky belief.
  - joegibbs 3mo ago
    Doesn't it say something if 5% of a group are wacko lunatics if, for instance, a random sample of the rest of the population has only 0.5% wackos?
    - FeepingCreature 3mo ago
      I would never disagree that the rationality community contains a higher than normal share of lunatics. Whether it produces or attracts them is an open question.
- s1artibartfast 3mo ago
  >the issue with the community is they take their eccentric beliefs as being gospel truths I am genuinely curious where you get that impression. The central element to the whole thing is that people should be willing to question their own beliefs and change their mind based on the data. If there is anything that turns rationalists into eccentrics, I it is being overly open to new ideas or beliefs.
twic 3mo ago
It's not just that they aren't free of bias. They appear to incubate truly wacky stuff - Leverage Research [1], the Zizians, all the AI/x-risk stuff. I am not the only person to have noticed this [2]. There's something about their approach that appears to actually make people more susceptible to nonsense, not less. [1] https://lydialaurenson.substack.com/p/the-inside-story-of-leverage-research https://lydialaurenson.substack.com/p/the-inside-story-of-le... [2] https://asteriskmag.com/issues/11/why-are-there-so-many-rationalist-cults https://asteriskmag.com/issues/11/why-are-there-so-many-rati...
- FeepingCreature 3mo ago
  Raising existential risk from AI in the same sentence as "rationality demands we murder our landlord" is a bit weird a grouping.
- krapp 3mo ago
  Regarding the Zizians, I recommend the Behind the Bastards series on them and the rationalist community. People involved with the rationalists will want to distance themselves from it of course, but the series makes a decent argument that the Zizians were simply the result of taking the ideology to its rational conclusion, that there is a definite throughline from one to the other in terms of philosophy and ideals. https://www.youtube.com/watch?v=9mJAerUL-7w https://www.youtube.com/watch?v=9mJAerUL-7w
  - lux-lux-lux 3mo ago
    I’ve interacted with a number of rationalists that are just as crazy as Ziz but managed to not implode with the help of money (either lucking into it via family/crypto or a tech job). My personal hot take is that if she’d been cis a number of people would still be alive.
  - s1artibartfast 3mo ago
    taking what ideology to the conclusions? The zizians throught some truly bizzare things, most of which are in no way representative of the rationilst community.
  - 3mo ago
g-b-r 3mo ago
It's not the bias the problem, it's the conviction of being better and "less wrong" than the rest of us. That, along with constantly calling yourself rational and being well-intentioned, easily leads to cultish beliefs to be pursued despite how wacky they seem at their face, because it was your superior brains that led you to them. If we add the often good (or enormous) financial resources, it makes for a seriously dangerous cult.
- zahlman 3mo ago
  > It's not the bias the problem, it's the conviction of being better and "less wrong" than the rest of us. There is not any such conviction. This is just your interpretation of a domain name, which is not intended that way. > That, along with constantly calling yourself rational This is explicitly not what you are supposed to do, and the point is explicitly not to create an identity around it.
  - justonepost2 3mo ago
    https://xcancel.com/tenobrus/status/2023473440364392850#m https://xcancel.com/tenobrus/status/2023473440364392850#m
    - g-b-r 3mo ago
      Oh, they're working to instate a dictatorship, they're not that bad...
      - justonepost2 3mo ago
        For the record, I’m not saying this particular loser on twitter will ever have the power to institute a dictatorship. But it is insane to say that EA and rationalism* are not about believing you are superior and more rational than your peers, or that most EA/rats don’t make an identity out of it (look at any of their online presences). Anthropic literally exists to funnel money and influence to its EA/rat cofounders and early investors, who sincerely believe that they and they alone are smart and rational enough to shepherd our society through its “technological adolescence”. I wish people who agree with this would at least be honest about it.
mbgerring 3mo ago
The people who sincerely believe that humans should be turned into mulch for the Computer God are “well meaning”? Say more
nialv7 3mo ago
> online nerds you are heavily downplaying the power and influence some of these people have. Sam Bankman-Fried is one of these people, for example

cogman10 3mo ago

I think the issue with the community is they take their eccentric beliefs as being gospel truths which can lead to truly unhinged and irrational beliefs and actions (see: the Zizians). That's what can make the rationalist movement feel more like a religious cult and less like a bunch of eccentrics. As an outsider looking in, it seems like the entire movement is filled with just accepting priors with very little evidence for why those priors should be accepted.

FeepingCreature 3mo ago
Consider a community filled with 100 people; 95 of them who don't take their eccentric beliefs as gospel truths and 5 who do. Those 5 will inevitably become the public face of the community when they make the news. However there is very little that the 95 could have done to stop them.
- astrange 3mo ago
  The entire point of rationalism is to take your own beliefs seriously and follow them as if you were a robot. (Except that they don't claim them to be "their own beliefs" but instead call them "priors".)
- cogman10 3mo ago
  > However there is very little that the 95 could have done to stop them. How about the 95% stop having gatherings where they all proclaim that their beliefs are true even if they don't seriously believe them? How is the 5% supposed to know that the 95% is actually larping belief? And how is an outsider who observes the 95% espousing their whacky beliefs supposed to know who is a member of the 95% and who is a member of the 5% (and if those percentages are accurate). You can observe the same dynamic in any church. Everyone in the congregation will have different levels of belief, but when it comes to sharing a testimony you'll be hard pressed to distinguish between who the true believers are and who's just "faking it until you make it".
  - zahlman 3mo ago
    > gatherings where they all proclaim that their beliefs are true even if they don't seriously believe them? What on Earth are you referring to? What gatherings? What beliefs? What proclamations? What evidence of insincerity?
    - cogman10 3mo ago
      > What gatherings? Rationalist have groups meetup and conferences. For example [1] > What beliefs? It's pretty diverse and not uniformly held. Some of the more strange ones that seem to have pretty wide appeal are around AI safety. For example, Roko's basilisk. > What proclamations? What evidence of insincerity? I was responding to the comment above me that suggested that 95% don't actually hold rationalist beliefs and are instead simply having fun with it. Only a small 5% are actually sincere. [1] https://www.reddit.com/r/slatestarcodex/comments/1car63w/lessonline_rationalist_festival_may_31st_june_2nd/ https://www.reddit.com/r/slatestarcodex/comments/1car63w/les...
      - zahlman 3mo ago
        Okay, but those pieces don't establish your claim. You would need a Roko's basilisk-like concept that only 5% of the community agree with, and a "group meetup" or "conference" where "they all" (i.e.: also the 95%) "proclaim that their beliefs (i.e.: including the basilisk thing) are true". If I observed such a gathering knowing that 5% of the people there have sincerely come to a particular conclusion based on the available evidence and the other 95% think the evidence doesn't support that conclusion, I would be extremely surprised to see everyone going around "larping belief". And I don't think you can actually show such a thing occurring.
        cogman10 3mo ago
        Which claim exactly did I make which you are refuting? Please actually quote it. > I would be extremely surprised to see everyone going around "larping belief". And I don't think you can actually show such a thing occurring. I never claimed I could or that I observed such a thing. The 95/5% didn't come from me, it came from the person I responded to. My point was that - It's impossible to distinguish who's a true believer and who isn't. - That the meetups of the rationalists re-enforce beliefs, particularly in true believers. Also that these beliefs of the rationalists can be pretty whacky and based on a jenga tower of priors. - That there isn't "nothing" a rational person could do to dissuade someone from a whacky belief.
- joegibbs 3mo ago
  Doesn't it say something if 5% of a group are wacko lunatics if, for instance, a random sample of the rest of the population has only 0.5% wackos?
  - FeepingCreature 3mo ago
    I would never disagree that the rationality community contains a higher than normal share of lunatics. Whether it produces or attracts them is an open question.
s1artibartfast 3mo ago
>the issue with the community is they take their eccentric beliefs as being gospel truths I am genuinely curious where you get that impression. The central element to the whole thing is that people should be willing to question their own beliefs and change their mind based on the data. If there is anything that turns rationalists into eccentrics, I it is being overly open to new ideas or beliefs.

twic 3mo ago

It's not just that they aren't free of bias. They appear to incubate truly wacky stuff - Leverage Research [1], the Zizians, all the AI/x-risk stuff. I am not the only person to have noticed this [2]. There's something about their approach that appears to actually make people more susceptible to nonsense, not less. [1] https://lydialaurenson.substack.com/p/the-inside-story-of-leverage-research https://lydialaurenson.substack.com/p/the-inside-story-of-le... [2] https://asteriskmag.com/issues/11/why-are-there-so-many-rationalist-cults https://asteriskmag.com/issues/11/why-are-there-so-many-rati...

FeepingCreature 3mo ago
Raising existential risk from AI in the same sentence as "rationality demands we murder our landlord" is a bit weird a grouping.
krapp 3mo ago
Regarding the Zizians, I recommend the Behind the Bastards series on them and the rationalist community. People involved with the rationalists will want to distance themselves from it of course, but the series makes a decent argument that the Zizians were simply the result of taking the ideology to its rational conclusion, that there is a definite throughline from one to the other in terms of philosophy and ideals. https://www.youtube.com/watch?v=9mJAerUL-7w https://www.youtube.com/watch?v=9mJAerUL-7w
- lux-lux-lux 3mo ago
  I’ve interacted with a number of rationalists that are just as crazy as Ziz but managed to not implode with the help of money (either lucking into it via family/crypto or a tech job). My personal hot take is that if she’d been cis a number of people would still be alive.
- s1artibartfast 3mo ago
  taking what ideology to the conclusions? The zizians throught some truly bizzare things, most of which are in no way representative of the rationilst community.
- 3mo ago

g-b-r 3mo ago

It's not the bias the problem, it's the conviction of being better and "less wrong" than the rest of us. That, along with constantly calling yourself rational and being well-intentioned, easily leads to cultish beliefs to be pursued despite how wacky they seem at their face, because it was your superior brains that led you to them. If we add the often good (or enormous) financial resources, it makes for a seriously dangerous cult.

zahlman 3mo ago
> It's not the bias the problem, it's the conviction of being better and "less wrong" than the rest of us. There is not any such conviction. This is just your interpretation of a domain name, which is not intended that way. > That, along with constantly calling yourself rational This is explicitly not what you are supposed to do, and the point is explicitly not to create an identity around it.
- justonepost2 3mo ago
  https://xcancel.com/tenobrus/status/2023473440364392850#m https://xcancel.com/tenobrus/status/2023473440364392850#m
  - g-b-r 3mo ago
    Oh, they're working to instate a dictatorship, they're not that bad...
    - justonepost2 3mo ago
      For the record, I’m not saying this particular loser on twitter will ever have the power to institute a dictatorship. But it is insane to say that EA and rationalism* are not about believing you are superior and more rational than your peers, or that most EA/rats don’t make an identity out of it (look at any of their online presences). Anthropic literally exists to funnel money and influence to its EA/rat cofounders and early investors, who sincerely believe that they and they alone are smart and rational enough to shepherd our society through its “technological adolescence”. I wish people who agree with this would at least be honest about it.

mbgerring 3mo ago

The people who sincerely believe that humans should be turned into mulch for the Computer God are “well meaning”? Say more

nialv7 3mo ago

> online nerds you are heavily downplaying the power and influence some of these people have. Sam Bankman-Fried is one of these people, for example