31 ms·

Why are there so many rationalist cults?

• thedudeabides5 1y ago
  Purity Spirals + Cheap Talk = irrational rationalists
  • thedudeabides5 1y ago
    Eliezer Yudkowsky, shows little interest in running one. He has consistently been distant from and uninvolved in rationalist community-building efforts, from Benton House (the first rationalist group house) to today’s Lightcone Infrastructure (which hosts LessWrong, an online forum, and Lighthaven, a conference center). He surrounds himself with people who disagree with him, discourages social isolation. Ummm, EY literally has a semi-permanent office in Lighthouse (at least until recently) and routinely blocks people on Twitter as a matter of course.
    • doubleunplussed 1y ago
      Blocking people on Twitter doesn't necessarily imply intolerance of people who disagree with you. People often block for different reasons than disagreement.
      • thedudeabides5 1y ago
        He surrounds himself with people who disagree with him, discourages social isolation. lol blocking people is literally encouraging social isolation by limiting your exposure to people who disagree with you. not very rational mind you
        • doubleunplussed 1y ago
          I don't think you really understood my comment
  • throw0101a 1y ago
    > Purity Spirals This is an interesting idea (phenomenon?): > A purity spiral is a theory which argues for the existence of a form of groupthink in which it becomes more beneficial to hold certain views than to not hold them, and more extreme views are rewarded while expressing doubt, nuance, or moderation is punished (a process sometimes called "moral outbidding").[1] It is argued that this feedback loop leads to members competing to demonstrate the zealotry or purity of their views.[2][3] * https://en.wikipedia.org/wiki/Purity_spiral https://en.wikipedia.org/wiki/Purity_spiral
    • doubleunplussed 1y ago
      Certainly something they're aware of - the same concept was discussed as early as in 2007 on Less Wrong under the name "evaporative cooling of group beliefs" https://www.lesswrong.com/posts/ZQG9cwKbct2LtmL3p/evaporative-cooling-of-group-beliefs https://www.lesswrong.com/posts/ZQG9cwKbct2LtmL3p/evaporativ...
      • thedudeabides5 1y ago
        haha, classic LWer reply "someone wrote a weirdly labeled thing about this once so they must not be subject to the phenomena' does the existence of a piece of writing on a topic really immunize the community from the disease @doubleunplussed?
        • doubleunplussed 1y ago
          I didn't say it did, I said they're aware of it. Also thought it was worth pointing out that the LW "weird label" predates the label in the comment I replied to.
• api 1y ago
  Why are there so many cults? People want to feel like they belong to something, and in a world in the midst of a loneliness and isolation epidemic the market conditions are ideal for cults.
  • iwontberude 1y ago
    Your profile says that you want to keep your identity small, but you have like over 30 thousand comments spelling out exactly who you are and how you think. Why not shard accounts? Anyways. Just a random thought.
    • keybored 1y ago
      [deleted]
      • shadowgovt 1y ago
        "SC identity?"
  • mindslight 1y ago
    Also, who would want to join an "irrationalist cult" ?
    • shadowgovt 1y ago
      Hey now, the Discordians have an ancient and respectable tradition. ;)
      • NoGravitas 1y ago
        Five tons of flax!
  • FuriouslyAdrift 1y ago
    Because we are currently living in an age of narcissism and tribalism / Identitarianism is the societal version of narcissism.
• optimalsolver 1y ago
  Pertinent Twitter comment: "Rationalism is such an insane name for a school of thought. Like calling your ideology correctism or winsargumentism" https://x.com/growing_daniel/status/1893554844725616666 https://x.com/growing_daniel/status/1893554844725616666
  • nyeah 1y ago
    Great names! Are you using them, or are they available? /s
  • wiredfool 1y ago
    Objectivisim?
  • hn_throwaway_99 1y ago
    To be honest I don't understand that objection. If you strip it from all its culty sociological effects, one of the original ideas of rationalism was to try to use logical reasoning and statistical techniques to explicitly avoid the pitfalls of known cognitive biases. Given that foundational tenet, "rationalism" seems like an extremely appropriate moniker. I fully accept that the rationalist community may have morphed into something far beyond that original tenet, but I think rationalism just describes the approach, not that it's the "one true philosophy".
    • ameliaquining 1y ago
      That it refers to a different but confusingly related concept in philosophy is a real downside of the name.
      • hn_throwaway_99 1y ago
        That point seems fair enough to me, as I'm not familiar with the specifics and history of the related concept in philosophy. But that seems different than the objection that calling yourself "rationalist" somehow implies you think that you have the "1 true answer" to the world's problems.
    • 1y ago
• mlinhares 1y ago
  > One is Black Lotus, a Burning Man camp led by alleged rapist Brent Dill, which developed a metaphysical system based on the tabletop roleplaying game Mage the Ascension. What the actual f. This is such an insane thing to read and understand what it means that i might need to go and sit in silence for the rest of the day. How did we get to this place with people going completely nuts like this?
  • optimalsolver 1y ago
    astronauts_meme.jpg
  • linohh 1y ago
    Running a cult is a somewhat reliable source of narcissistic supply. The internet tells you how to do it. So an increasing number of people do it.
    • deleted 1y ago
      [deleted]
  • TimorousBestie 1y ago
    Mage: The Ascension is basically a delusions of grandeur simulator, so I can see how an already unstable personality might get attached to it and become more unstable.
    • mlinhares 1y ago
      I don't know, i'd understand something like Wraith (which I did see people developing issues, the shadow mechanic is such a terrible thing) but Mage is so, like, straightforward? Use your mind to control reality, reality fights back with paradox, its cool for a teenager but you read a bit more fantasy and you'll definitely find cooler stuff. But i guess for you to join a cult your mind must stay a teen mind forever.
      • wavefunction 1y ago
• thrance 1y ago
  Reminds me somewhat of the Culte de la Raison (Cult of Reason) birthed by the french revolution. It didn't last long. https://en.wikipedia.org/wiki/Cult_of_Reason https://en.wikipedia.org/wiki/Cult_of_Reason
• amiga386 1y ago
  See also Rational Magic: Why a Silicon Valley culture that was once obsessed with reason is going woo (2023) https://www.thenewatlantis.com/publications/rational-magic https://www.thenewatlantis.com/publications/rational-magic and its discussion on HN: https://news.ycombinator.com/item?id=35961817 https://news.ycombinator.com/item?id=35961817
• gjsman-1000 1y ago
  It’s especially popular in Silicon Valley. Quite possibly, places like Reddit and Hacker News, are training for the required level of intellectual smugness, and certitude that you can dismiss every annoying argument with a logical fallacy. That sounds smug of me, but I’m actually serious. One of their defects, is that once you memorize all the fallacies (“Appeal to authority,” “Ad hominem,”) you can easily reach the point where you more easily recognize the fallacies in everyone else’s arguments than your own. You more easily doubt other people’s cited authorities, than your own. You slap “appeal to authority” against a disliked opinion, while citing an authority next week for your own. It’s a fast path from there to perceived intellectual superiority, and an even faster path from there into delusion. Rational delusion.
  • shadowgovt 1y ago
    It's generally worth remembering that some of the fallacies are actually structural, and some are rhetorical. A contradiction creates a structural fallacy; if you find one, it's a fair belief that at least one of the supporting claims is false. In contrast, appeal to authority is probabilistic: we don't know, given the current context, if the authority is right, so they might be wrong... But we don't have time to read the universe into this situation so an appeal to authority is better than nothing. ... and this observation should be coupled with the observation that the school of rhetoric wasn't teaching a method for finding truth; it was teaching a method for beating an opponent in a legal argument. "Appeal to authority is a logical fallacy" is a great sword to bring to bear if your goal is to turn off the audience's ability to ask whether we should give the word of the environmental scientist and the washed-up TV actor equal weight on the topic of environmental science...
    • gjsman-1000 1y ago
      … however, even that is up for debate. Maybe the TV actor in your own example is Al Gore filming An Inconvenient Truth and the environmental scientist was in the minority which isn’t so afraid of climate change. Fast forward to 2025, the scientist’s minority position was wrong, while Al Gore’s documentary was legally ruled to have 9 major errors; so you were stupid on both sides, with the TV actor being closer.
• bobson381 1y ago
  I keep thinking about the first Avengers movie, when Loki is standing above everyone going "See, is this not your natural state?". There's some perverse security in not getting a choice, and these rationalist frameworks, based in logic, can lead in all kinds of crazy arbitrary directions - powered by nothing more than a refusal to suffer any kind of ambiguity.
  • csours 1y ago
    Humans are not chickens, but we sure do seem to love having a pecking order.
    • lazide 1y ago
      Making good decisions is hard, and being accountable to the results of them is not fun. Easier to outsource if you can.
    • snarf21 1y ago
      I think it is more simple in that we love tribalism. A long time ago being part of a tribe had such huge benefits over going it alone that it was always worth any tradeoffs. We have a much better ability to go it alone now but we still love to belong to a group. Too often we pick a group based on a single shared belief and don't recognize all the baggage that comes along. Life is also too complicated today. It is difficult for someone to be knowledgeable in one topic let alone the 1000s that make up our society.
      • csours 1y ago
        maybe the real innie/outie is the in-group/out-group. no spoilers, i haven't finished that show yet
  • jacquesm 1y ago
    They mostly seem to lean that way because it gives them carte blanche to do as they please. It is just a modern version of 'god has led my hand'.
• nathan_compton 1y ago
  Thinking too hard about anything will drive you insane but I think the real issue here is that rationalists simply over-estimate both the power of rational thought and their ability to do it. If you think of people who tend to make that kind of mistake you can see how you get a lot of crazy groups. I guess I'm a radical skeptic, secular humanist, utilitarianish sort of guy, but I'm not dumb enough to think throwing around the words "bayesian prior" and "posterior distribution" makes actually figuring out how something works or predicting the outcome of an intervention easy or certain. I've had a lot of life at this point and gotten to some level of mastery at a few things and my main conclusion is that most of the time its just hard to know stuff and that the single most common cognitive mistake people make is too much certainty.
  • nyeah 1y ago
    I'm lucky enough work in a pretty rational place (small "r"). We're normally data-limited. Being "more rational" would mean taking/finding more of the right data, talking to the right people, reading the right stuff. Not just thinking harder and harder about what we already know. There's a point where more passive thinking stops adding value and starts subtracting sanity. It's pretty easy to get to that point. We've all done it.
    • naasking 1y ago
      > We're normally data-limited. This is a common sentiment but is probably not entirely true. A great example is cosmology. Yes, more data would make some work easier, but astrophysicists and cosmologists have shown that you can gather and combine existing data and look at it in novel new ways to produce unexpected results, like place bounds that can include/exclude various theories. I think a philosophy that encourages more analysis rather than sitting back on our laurels with an excuse that we need more data is good, as long as it's done transparently and honestly.
      • nyeah 1y ago
        I suspect you didn't read some parts of my comment. I didn't say everyone in the world is always data-limited, I said we normally are where I work. I didn't recommend "sitting back on our laurels." I made very specific recommendations. The qualifier "normally" already covers "not entirely true". Of course it's not entirely true. It's mostly true for us now. (In fact twenty years ago we used more numerical models than we do now, because we were facing more unsolved problems where the solution was pretty well knowable just by doing more complicated calculations, but without taking more data. Back then, when people started taking lots of data, it was often a total waste of time. But right now, most of those problems seem to be solved. We're facing different problems that seem much harder to model, so we rely more on data. This stage won't be permanent either.) It's not a sentiment, it's a reality that we have to deal with.
• righthand 1y ago
  [flagged]
  • potato3732842 1y ago
    I think it speaks volumes that you think "american" is the approximate level of scope that this behavior lives at. Stuff like this crosses all aspects of society. Certain americans of certain backgrounds, demographics and life experiences are fare more likely engage in it than others. I think those people are minority, but they are definitely an overly visible one if not a local majority in a lot of internet spaces so it's easy to mistake them for the majority.
    • righthand 1y ago
      Sure many people across the globe are susceptible to cult-think. It’s just been a century long trend in America to seek a superior way of living to “save all Americans” is all. No offense to other countries peoples, I’m sure they’re just as good cult members championing over application as any American. It probably speaks more volumes that you are taking my comment about this so literally.
      • righthand 1y ago
        Are Apple, Disney, NFL, etc. fandoms with religious dedication a cult? America you can choose your cult identities and even belong to many.
• incomingpain 1y ago
  We live in an irrational time. It's unclear if it was simply under reported in history or social changes in the last ~50-75 years have had breaking consequences. People are trying to make sense of this. For examples. The Canadian government heavily subsidizes junk food, then spends heavily on healthcare because of the resulting illnesses. It restrict and limits healthy food through supply management and promotes a “food pyramid” favoring domestic unhealthy food. Meanwhile, it spends billions marketing healthy living, yet fines people up to $25,000 for hiking in forests and zones cities so driving is nearly mandatory. Government is an easy target for irrational behaviours.
  • watwut 1y ago
    Scientology is here since 1953 and it has similarly bonkers set of believes. And is huge. Your rant about government or not being allowed to hike in some places in Canada is unrelated to the issue.
  • codr7 1y ago
    There's nothing irrational about it, this is how you maximize power and profit at any and all costs.
    • incomingpain 1y ago
      I completely get that point of view; and yes if that's the goal, it's completely rational. But from a societal cohesion or perhaps even an ethical point of view it's just pure irrationality. When typing the post, I was thinking, different levels of government, changing ideologies of politicians leaving inconsistent governance.
      • codr7 1y ago
        I couldn't agree more, but we've long since given up our power collectively in hope of escaping responsibility.
• noqc 1y ago
  Perhaps I will get downvoted to death again for saying so, but the obvious answer is because the name "rationalist" is structurally indistinguishable from the name "scientology" or "the illuminati". You attract people who are desperate for an authority to appeal to, but for whatever reason are no longer affiliated with the church of their youth. Even a rationalist movement which held nothing as dogma would attract people seeking dogma, and dogma would form. The article begins by saying the rationalist community was "drawn together by AI researcher Eliezer Yudkowsky’s blog post series The Sequences". Obviously the article intends to make the case that this is a cult, but it's already done with the argument at this point.
  • johnisgood 1y ago
    I do not see any reasons for you to get down-voted. I agree that the term "rationalist" would appeal to many people, and the obvious need to belong to a group plays a huge role.
    • noqc 1y ago
      There are a lot of rationalists in this community. Pointing out that the entire thing is a cult attracts downvotes from people who wish to, for instance, avoid being identified with the offshoots.
      • 6177c40f 1y ago
        No, the downvotes are because rationalism isn't a cult and people take offense to being blatantly insulted. This article is about cults that are rationalism-adjacent, it's not claiming that rationalism is itself a cult.
        • noqc 1y ago
          That's almost word for word what I said...
          • 6177c40f 1y ago
• throwaway984393 1y ago
  [dead]
• cjs_ac 1y ago
  Rationalism is the belief that reason is the primary path to knowledge, as opposed to, say, the observation that is championed by empiricism. It's a belief system that prioritises imposing its tenets on reality rather than asking reality what reality's tenets are. From the outset, it's inherently cult-like.
  • Ifkaluva 1y ago
    That is the definition of “rationalism” as proposed by philosophers like Descartes and Kant, but I don’t think that is an accurate representation of the type of “rationalism” this article describes. This article describes “rationalism” as described in LessWrong and the sequences by Eliezer Yudkowsky. A good amount of it based on empirical findings from psychology behavior science. It’s called “rationalism” because it seeks to correct common reasoning heuristics that are purported to lead to incorrect reasoning, not in contrast to empiricism.
    • FergusArgyll 1y ago
      I was going to write a similar comment as op, so permit me to defend it: Many of their "beliefs" - Super-duper intelligence, doom - are clearly not believed by the market; Observing the market is a kind of empiricism and it's completely discounted by the lw-ers
    • glenstein 1y ago
      Agreed, I appreciate that there's a conceptual distinction between the philosophical versions of rationalism and empiricism, but what's being talked about here is a conception that (again, at least notionally) is interested in and compatible with both. I am pretty sure many of the LessWrong posts are about how to understand the meaning of different types of data and are very much about examining, developing, criticizing a rich variety of empirical attitudes.
  • gethly 1y ago
    But you cannot have reason without substantial proof of how things behave by observing them in the first place. Reason is simply a logical approach to yes and no questions where you factually know, from observation of past events, how things work. And therefore you can simulate an outcome by the exercise of reasoning applied onto a situation that you have not yet observed and come to a logical outcome, given the set of rules and presumptions.
• AIPedant 1y ago
  I think I found the problem! The rationalist community was drawn together by AI researcher Eliezer Yudkowsky’s blog post series The Sequences, a set of essays about how to think more rationally I actually don't mind Yudkowski as an individual - I think he is almost always wrong and undeservedly arrogant, but mostly sincere. Yet treating him as an AI researcher and serious philosopher (as opposed to a sci-fi essayist and self-help writer) is the kind of slippery foundation that less scrupulous people can build cults from. (See also Maharishi Mahesh Yogi and related trends - often it is just a bit of spiritual goofiness as with David Lynch, sometimes you get a Charles Manson.)
  • polytely 1y ago
    Don't forget the biggest scifi guy turned cult leader of all L. Ron Hubbard
    • AIPedant 1y ago
      I don't think Yudkowski is at all like L. Ron Hubbard. Hubbard was insane and pure evil. Yudkowski seems like a decent and basically reasonable guy, he's just kind of a blowhard and he's wrong about the science. L. Ron Hubbard is more like the Zizians.
      • pingou 1y ago
        I don't have a horse in the battle but could you provide a few examples where he was wrong?
        • bglazer 1y ago
          Here's one: Yudkowsky has been confidently asserting (for years) that AI will extinct humanity because it will learn how to make nanomachines using "strong" covalent bonds rather than the "weak" van der Waals forces used by biological systems like proteins. I'm certain that knowledgeable biologists/physicists have tried to explain to him why this belief is basically nonsense, but he just keeps repeating it. Heck there's even a LessWrong post that lays it out quite well [1]. This points to a general disregard for detailed knowledge of existing things and a preference for "first principles" beliefs, no matter how wrong they are. [1] https://www.lesswrong.com/posts/8viKzSrYhb6EFk6wg/why-yudkowsky-is-wrong-about-covalently-bonded-equivalents https://www.lesswrong.com/posts/8viKzSrYhb6EFk6wg/why-yudkow...
• iwontberude 1y ago
  They watched too much eXistenZ
• lenerdenator 1y ago
  Because humans like people who promise answers.
  • andy99 1y ago
    Boring as it is, this is the answer. It's just more religion. Church, cult, cult, church. So we'll get bored someplace else every Sunday. Does this really change our everyday lives?
    • optimalsolver 1y ago
      Funnily enough, the actress who voiced this line is a Scientologist: https://en.wikipedia.org/wiki/Nancy_Cartwright#Personal_life https://en.wikipedia.org/wiki/Nancy_Cartwright#Personal_life
      • andy99 1y ago
        I think they were making fun of the "Moonies" so she was probably able to rationalize it. Pretty sure Isaac Hayes quit South Park over their making fun of scientologists.
        • ZeroGravitas 1y ago
          I read recently that he suffered a serious medical event atound that time and it was actually cult members speaking on his behalf that withdrew him from the show. I think it was a relative of his claiming this.
          • nobody9999 1y ago
            >I read recently that he suffered a serious medical event atound that time and it was actually cult members speaking on his behalf that withdrew him from the show. I saw him perform after he left the show and several months before he passed. He looked pretty unhealthy and I'm glad I had the chance to see him before that happened, He obviously was having some medical issues, but didn't discuss them during his performance.
• saasapologist 1y ago
  I think we've strayed too far from the Aristotelian dynamics of the self. Outside of sexuality and the proclivities of their leaders, emphasis on physical domination of the self is lacking. The brain runs wild, the spirit remains aimless. In the Bay, the difference between the somewhat well-adjusted "rationalists" and those very much "in the mush" is whether or not someone tells you they're in SF or "on the Berkeley side of things"
• j_m_b 1y ago
  > One way that thinking for yourself goes wrong is that you realize your society is wrong about something, don’t realize that you can’t outperform it, and wind up even wronger. many such cases
  • quantummagic 1y ago
    It's almost the defining characteristic of our time.
    • teiferer 1y ago
      Tell-tale slogan: "Let's derive from first principles"
  • shadowgovt 1y ago
    It is an unfortunate reality of our existence that sometimes Chesterton actually did build that fence for a good reason, a good reason that's still here. (One of my favorite TED talks was about a failed experiment in introducing traditional Western agriculture to a people in Zambia. It turns out when you concentrate too much food in one place, the hippos come and eat it all and people can't actually out-fight hippos in large numbers. In hindsight, the people running the program should have asked how likely it was that folks in a region that had exposure to other people's agriculture for thousands of years, hadn't ever, you know... tried it. https://www.ted.com/talks/ernesto_sirolli_want_to_help_someone_shut_up_and_listen/transcript https://www.ted.com/talks/ernesto_sirolli_want_to_help_someo...)
    • ljlolel 1y ago
      TEDx
    • bobson381 1y ago
      You sound like you'd like the book Seeing like a State.
    • im3w1l 1y ago
      Shoot the hippos to death for even more food. If it doesn't seem to work it's just a matter of having more and bigger guns.
• numbsafari 1y ago
  Why are so many cults founded on fear or hate? Because empathy is hard.
  • raxxorraxor 1y ago
    Empathy is usally a limited resource of those that generously ascribe it to themselves and it is often mixed up with self-serving desires. Perhaps Rationalists have similar difficulties with reasoning. While I believe Rationalism can be some form of occupational disease in tech circles, it sometimes does pose interesting questions. You just have to be aware that the perspective to analyse circumstances is intentionally constrained and in the end you still have to compare your prognosis to a reality that always choses empiricism.
• meroes 1y ago
  It grew out of many different threads: different websites, communities, etc all around the same time. I noticed it contemporaneously in the philosophy world where Nick Bostrom’s Simulation argument was boosted more than it deserved (like everyone was just accepting it at the lay-level). Looking back I see it also developed from less wrong and other sites, but I was wondering what was going on with simulations taking over philosophy talk. Now I see how it all coalesced. All of it has the appearance of sounding so smart, and a few sites were genuine. But it got taken over.
  • potatolicious 1y ago
    Yeah, a lot of the comments here are really just addressing cults writ large and opposed to why this one was particularly successful. A significant part of this is the intersection of the cult with money and status - this stuff really took off once prominent SV personalities became associated with it, and got turbocharged when it started intersecting with the angel/incubator/VC scene, when there was implicit money involved. It's unusually successful because -- for a time at least -- there was status (and maybe money) in carrying water for it.
    • jacquesm 1y ago
      Paypal will be traced as the root cause of many of our future troubles.
      • varjag 1y ago
        Wish I could upvote this twice. It's like intersectionality for evil.
    • wredcoll 1y ago
      https://en.m.wikipedia.org/wiki/Barth%C3%A9lemy-Prosper_Enfantin https://en.m.wikipedia.org/wiki/Barth%C3%A9lemy-Prosper_Enfa... Sometimes history really does rhyme. > Enfantin and Amand Bazard were proclaimed Pères Suprêmes ("Supreme Fathers") – a union which was, however, only nominal, as a divergence was already manifest. Bazard, who concentrated on organizing the group, had devoted himself to political reform, while Enfantin, who favoured teaching and preaching, dedicated his time to social and moral change. The antagonism was widened by Enfantin's announcement of his theory of the relation of man and woman, which would substitute for the "tyranny of marriage" a system of "free love".[1]
• alphazard 1y ago
  The terminology here is worth noting. Is a Rationalist Cult a cult that practices Rationalism according to third parties, or is it a cult that says they are Rationalist? Clearly all of these groups that believe in demons or realities dictated by tabletop games are not what third parties would call Rationalist. They might call themselves that. There are some pretty simple tests that can out these groups as not rational. None of these people have ever seen a demon, so world models including demons have never predicted any of their sense data. I doubt these people would be willing to make any bets about when or if a demon will show up. Many of us would be glad to make a market concerning predictions made by tabletop games about physical phenomenon.
  • ameliaquining 1y ago
    The article is talking about cults that arose out of the rationalist social milieu, which is a separate question from whether the cult's beliefs qualify as "rationalist" in some sense (a question that usually has no objective answer anyway).
  • glenstein 1y ago
    Yeah, I would say the groups in question are notionally, aspirationally rational and I would hate for the takeaway to be disengagement from principles of critical thinking and skeptical thinking writ large. Which, to me, raises the fascinating question of what does a "good" version look like, of groups and group dynamics centered around a shared interest in best practices associated with critical thinking? At a first impression, I think maybe these virtues (which are real!) disappear into the background of other, more applied specializations, whether professions, hobbies, backyard family barbecues.
    • alphazard 1y ago
      It would seem like the quintessential Rationalist institution to congregate around is the prediction market. Status in the community has to be derived from a history of making good bets (PnL as a %, not in absolute terms). And the sense of community would come from (measurably) more rational people teaching (measurably) less rational people how to be more rational.
• wiredfool 1y ago
  It's really worth reading up on the techniques from Large Group Awareness Training so that you can recognize them when they pop up. Once you see them listed (social pressure, sleep deprivation, control of drinking/bathroom, control of language/terminology, long exhausting activities, financial buy in, etc) and see where they've been used in cults and other cult adjacent things it's a little bit of a warning signal when you run across them IRL.
  • derektank 1y ago
    Related, the BITE model of authoritarian control is also a useful framework for identifying malignant group behavior. It's amazing how consistent these are across groups and cultures, from Mao's inner circle to NXIVM and on. https://freedomofmind.com/cult-mind-control/bite-model-pdf-download/ https://freedomofmind.com/cult-mind-control/bite-model-pdf-d...
• keybored 1y ago
  Cue all the surface-level “tribalism/loneliness/hooman nature” comments instead of the simple analysis that Rationalism (this kind) is severely brain-broken and irredeemable and will just foster even worse outcomes in a group setting. It’s a bit too close to home (ideologically) to get a somewhat detached analysis.
• bubblyworld 1y ago
  What is the base rate here? Hard to know the scope of the problem without knowing how many non-rationalists (is that even a coherent group of people?) end up forming weird cults, as a comparison. My impression is that crazy beliefs are common amongst everybody. A much simpler theory is that rationalists are mostly normal people, and normal people tend to form cults.
  • deleted 1y ago
    [deleted]
  • glenstein 1y ago
    I was wondering about this too. You could also say it's a sturgeon's law question. They do note at the beginning of the article that many, if not most such groups have reasonably normal dynamics, for what it's worth. But I think there's a legitimate question of whether we ought to expect groups centered on rational thinking to be better able to escape group dynamics we associate with irrationality.
• zzzeek 1y ago
  because humans are biological creatures iterating through complex chemical processes that are attempting to allow a large organism to survive and reproduce within the specific ecosystem provided by the Earth in the present day. "Rational reasoning" is a quaint side effect that sometimes is emergent from the nervous system of these organisms, but it's nothing more than that. It's normal that the surviving/reproducing organism's emergent side effect of "rational thought", when it is particularly intense, will self-refer to the organism and act as though it has some kind of dominion over the organism itself, but this is, like the rationalism itself, just an emergent effect that is accidental and transient. Same as if you see a cloud that looks like an elephant (it's still just a cloud).
• rkapsoro 1y ago
  Something like 15 years ago I once went to a Less Wrong/Overcoming Bias meetup in my town after being a reader of Yudkowsky's blog for some years. I was like, Bayesian Conspiracy, cool, right? The group was weird and involved quite a lot of creepy oversharing. I didn't return.
• dfabulich 1y ago
  The whole game of Rationalism is that you should ignore gut intuitions and cultural norms that you can't justify with rational arguments. Well, it turns out that intuition and long-lived cultural norms often have rational justifications, but individuals may not know what they are, and norms/intuitions provide useful antibodies against narcissist would-be cult leaders. Can you find the "rational" justification not to isolate yourself from non-Rationalists, not to live with them in a polycule, and not to take a bunch of psychedelic drugs with them? If you can't solve that puzzle, you're in danger of letting the group take advantage of you.
  • StevenWaterman 1y ago
    Yeah, I think this is exactly it. If something sounds extremely stupid, or if everyone around you says it's extremely stupid, it probably is. If you can't justify it, it's probably because you have failed to find the reason it's stupid, not because it's actually genius. And the crazy thing is, none of that is fundamentally opposed to rationalism. You can be a rationalist who ascribes value to gut instinct and societal norms. Those are the product of millions of years of pre-training. I have spent a fair bit of time thinking about the meaning of life. And my conclusions have been pretty crazy. But they sound insane, so until I figure out why they sound insane, I'm not acting on those conclusions. And I'm definitely not surrounding myself with people who take those conclusions seriously.
  • empath75 1y ago
    > The whole game of Rationalism is that you should ignore gut intuitions and cultural norms that you can't justify with rational arguments. The game as it is _actually_ played is that you use rationalist arguments to justify your pre-existing gut intuitions and personal biases.
    • NoGravitas 1y ago
      Or worse - to justify the gut intuitions and personal biases of your cult leader.
    • xbar 1y ago
      Which is to say, Rationalism is easily abused to justify any behavior contrary to its own tenets, just like any other -ism.
• JohnMakin 1y ago
  One of a few issues I have with groups like these, is that they often confidently and aggressively spew a set of beliefs that on their face logically follow from one another, until you realize they are built on a set of axioms that are either entirely untested or outright nonsense. This is common everywhere, but I feel especially pronounced in communities like this. It also involves quite a bit of navel gazing that makes me feel a little sick participating in. The smartest people I have ever known have been profoundly unsure of their beliefs and what they know. I immediately become suspicious of anyone who is very certain of something, especially if they derived it on their own.
  • bobson381 1y ago
    There should be an extremist cult of people who are certain only that uncertainty is the only certain thing
    • mapontosevenths 1y ago
      There already is, they're called "Politicians."
    • ameliaquining 1y ago
      Like Robert Anton Wilson if he were way less chill, perhaps.
    • rpcope1 1y ago
      More people should read Sextus Empiricus as he's basically the O.G. Phyrronist skeptic and goes pretty hard on this very train of thought.
      • bobson381 1y ago
        Cool. Any specific recs or places to start with him?
        • rpcope1 1y ago
          Probably the Hackett book, "Sextus Empiricus: Selections from the Major Writings on Scepticism"
• animal_spirits 1y ago
  > If someone is in a group that is heading towards dysfunctionality, try to maintain your relationship with them; don’t attack them or make them defend the group. Let them have normal conversations with you. This is such an important skill we should all have. I learned this best from watching the documentary Behind the Curve, about flat earthers, and have applied it to my best friend diving into the Tartarian conspiracy theory.
• deleted 1y ago
  [deleted]
• dkarl 1y ago
  Isn't this entirely to be expected? The people who dominate groups like these are the ones who put the most time and effort into them, and no sane person who appreciates both the value and the limitations of rational thinking is going to see as much value in a rationalist group, and devote as much time to it, as the kind of people who are attracted to the cultish aspect of achieving truth and power through pure thought. There's way more value there if you're looking to indulge in, or exploit, a cult-like spiral into shared fantasy than if you're just looking to sharpen your logical reasoning.
• gadders 1y ago
  They are literally the "ackchyually" meme made flesh.
• Isamu 1y ago
  So I like Steven Pinker’s book Rationality, to me it seems quite straightforward. But I have never been able to get into the Rationalist stuff, to me it’s all very meandering and peripheral and focused on… I don’t know what. Is it just me?
  • ameliaquining 1y ago
    Depends very much on what you're hoping to get out of it. There isn't really one "rationalist" thing at this point, it's now a whole bunch of adjacent social groups with overlapping-but-distinct goals and interests.
  • handoflixue 1y ago
    https://www.lesswrong.com/highlights https://www.lesswrong.com/highlights this is the ostensible "Core Highlights", curated by major members of the community, and I believe Eliezer would endorse it. If you don't get anything out of reading the list itself, then you're probably not going to get anything out of the rest of the community either. If you poke around and find a few neat ideas there, you'll probably find a few other neat ideas. For some people, though, this is "wait, holy shit, you can just DO that? And it WORKS?", in which case probably read all of this but then also go find a few other sources to counter-balance it. (In particular, probably 90% of the useful insights already exist elsewhere in philosophy, and often more rigorously discussed - LessWrong will teach you the skeleton, the general sense of "what rationality can do", but you need to go elsewhere if you want to actually build up the muscles)
• biophysboy 1y ago
  > “There’s this belief [among rationalists],” she said, “that society has these really bad behaviors, like developing self-improving AI, or that mainstream epistemology is really bad–not just religion, but also normal ‘trust-the-experts’ science. That can lead to the idea that we should figure it out ourselves. And what can show up is that some people aren't actually smart enough to form very good conclusions once they start thinking for themselves.” I see this arrogant attitude all the time on HN: reflexive distrust of the "mainstream media" and "scientific experts". Critical thinking is a very healthy idea, but its dangerous when people use it as a license to categorically reject sources. Its even worse when extremely powerful people do this; they can reduce an enormous sub-network of thought into a single node for many many people. So, my answer for "Why Are There So Many Rationalist Cults?" is the same reason all cults exist: humans like to feel like they're in on the secret. We like to be in secret clubs.
  • ameliaquining 1y ago
    Sure, but that doesn't say anything about why one particular social scene would spawn a bunch of cults while others do not, which is the question that the article is trying to answer.
    • biophysboy 1y ago
      Maybe I was too vague. My argument is that cults need a secret. The secret of the rationalist community is "nobody is rational except for us". Then the rituals would be endless probability/math/logic arguments about sci-fi futures.
      • saalweachter 1y ago
        I think the promise of secret knowledge is important, but I think cults also need a second thing: "That thing you fear? You're right to fear it, and only we can protect you from it. If you don't do what we say, it's going to be so much worse than it is now, but if you do, everything will be good and perfect." In the rationalist cults, you typically have the fear of death and non-existence, coupled with the promise of AGI, the Singularity and immortality, weighed against the AI Apocalypse.
• the_third_wave 1y ago
  Gott ist tot! Gott bleibt tot! Und wir haben ihn getötet! Wie trösten wir uns, die Mörder aller Mörder? Das Heiligste und Mächtigste, was die Welt bisher besaß, es ist unter unseren Messern verblutet. The average teenager who reads Nietzsches proclamation on the death of God thinks of this as an accomplishment, finally we got rid of those thousands of years old and thereby severely outdated ideas and rules. Somewhere along the march to maturity they may start to wonder whether that which has replaced those old rules and ideas were good replacements but most of them never come to the realisation that there were rebellious teenagers during all those centuries when the idea of a supreme being to which or whom even the mightiest were to answer to still held sway. Nietzsche saw the peril in letting go off that cultural safety valve and warned for what might come next. We are currently living in the world he warned us about and for that I, atheist as I am, am partly responsible. The question to be answered here is whether it is possible to regain the benefits of the old order without getting back the obvious excesses, the abuse, the sanctimoniousness and all the other abuses of power and privilege which were responsible for turning people away from that path.
• digbybk 1y ago
  When I was looking for a group in my area to meditate with, it was tough finding one that didn't appear to be a cult. And yet I think Buddhist meditation is the best tool for personal growth humanity has ever devised. Maybe the proliferation of cults is a sign that Yudkowsky was on to something.
  • ivm 1y ago
    None of them are practicing Buddhist meditation though, same for the "personal growth" oriented meditation styles. Buddhist meditation exists only in the context of the Four Noble Truths and the rest of the Buddha's Dhamma. Throwing them away means it stops being Buddhist.
    • digbybk 1y ago
      I disagree, but we'd be arguing semantics. In any case, the point still stands: you can just as easily argue that these rationalist offshoots aren't really Rationalist.
      • ivm 1y ago
        I'm not familiar enough with their definitions to argue about them, but meditations techniques predate Buddhism. In fact, the Buddha himself learned them from two teachers before developing his own path. Also, the style of meditation taught nowadays (accepting non-reactive awareness) is not how it's described in the Pali Canon. This isn’t just a "must come from the Champagne region of France, otherwise it’s sparkling wine" bickering, but actual widespread misconceptions of what counts as Buddhism. Many ideas floating in Western discourse are basically German Romanticism wrapped in Orientalist packaging, not matching neither Theravada nor Mahayana teachings (for example, see the Fake Buddha Quotes project). So the semantics are extremely important when it comes to spiritual matters. Flip one or two words and the whole metaphysical model goes in a completely different direction. Even translations add distortions, so there’s no room to be careless.
• os2warpman 1y ago
  Rationalists are, to a man (and they’re almost all men) arrogant dickheads and arrogant dickheads do not see what they’re doing to be “a cult” but “the right and proper way of things because I am right and logical and rational and everyone else isn’t”.
  • IX-103 1y ago
    That's an unnecessary charicaterature. I have met many rationalists of both genders and found most of them quite pleasant. But it seems that the proportion of "arrogant dickheads" unfortunately matches that of the general population. Whether it's "irrational people" or "liberal elites" these assholes always seem to find someone to look down on.
• jancsika 1y ago
  > And what can show up is that some people aren't actually smart enough to form very good conclusions once they start thinking for themselves. It's mostly just people who aren't very experienced talking about and dealing honestly with their emotions, no? I mean, suppose someone is busy achieving and, at the same time, proficient in balancing work with emotional life, dealing head-on with interpersonal conflicts, facing change, feeling and acknowledging hurt, knowing their emotional hangups, perhaps seeing a therapist, perhaps even occasionally putting personal needs ahead of career... :) Tell that person they can get a marginal (or even substantial) improvement from some rationalist cult practice. Their first question is going to be, "What's the catch?" Because at the very least they'll suspect that adjusting their work/life balance will bring a sizeable amount of stress and consequent decrease in their emotional well-being. And if the pitch is that this rationalist practice works equally well at improving emotional well-being, that smells to them. They already know they didn't logic themselves into their current set of emotional issues, and they are highly unlikely to logic themselves out of them. So there's not much value here to offset the creepy vibes of the pitch. (And again-- being in touch with your emotions means quicker and deeper awareness of creepy vibes!) Now, take a person whose unexplored emotional well-being tacitly depends on achievement. Even a marginal improvement in achievement could bring perceptible positive changes in their holistic selves! And you can step through a well-specified, logical process to achieve change? Sign HN up!
• scythe 1y ago
  One of the hallmarks of cults — if not a necessary element — is that they tend to separate their members from the outside society. Rationalism doesn't directly encourage this, but it does facilitate it in a couple of ways: - Idiosyncratic language used to describe ordinary things ("lightcone" instead of "future", "prior" instead of "belief" or "prejudice", etc) - Disdain for competing belief systems - Insistence on a certain shared interpretation of things most people don't care about (the "many-worlds interpretation" of quantum uncertainty, self-improving artificial intelligence, veganism, etc) - I'm pretty sure polyamory makes the list somehow, just because it isn't how the vast majority of people want to date. In principle it's a private lifestyle choice, but it's obviously a community value here. So this creates an opportunity for cult-like dynamics to occur where people adjust themselves according to their interactions within the community but not interactions outside the community. And this could seem — to the members — like the beliefs themselves are the problem, but from a sociological perspective, it might really be the inflexible way they diverge from mainstream society.
• kazinator 1y ago
  The only way you can hope to get a gathering of nothing but paragons of critical thinking and skepticism is if the gathering has an entrance exam in critical thinking and skepticism (and a pretty tough one, if they are to be paragons). Or else, it's invitation-only.
• VonGuard 1y ago
  This is actually a known pattern in tech, going back to Engelbart and SRI. While not 1-to-1, you could say that the folks who left SRI for Xerox PARC did so because Engelbart and his crew became obsessed with EST: https://en.wikipedia.org/wiki/Erhard_Seminars_Training https://en.wikipedia.org/wiki/Erhard_Seminars_Training EST-type training still exists today. You don't eat until the end of the whole weekend, or maybe you get rice and little else. Everyone is told to insult you day one until you cry. Then day two, still having not eaten, they build you up and tell you how great you are and have a group hug. Then they ask you how great you feel. Isn't this a good feeling? Don't you want your loved ones to have this feeling? Still having not eaten, you're then encouraged to pay for your family and friends to do the training, without their knowledge or consent. A friend of mine did this training after his brother paid for his mom to do it, and she paid for him to do it. Let's just say that, though they felt it changed their lives at the time, their lives in no way shape or form changed. Two are in quite a bad place, in fact... Anyway, point is, the people who invented everything we are using right now were also susceptible to cult-like groups with silly ideas and shady intentions.
  • nobody9999 1y ago
    >EST-type training still exists today It's called the "Landmark"[0] now. Several of my family members got sucked into that back in the early 80s and quite a few folks I knew socially as well. I was quite skeptical, especially because of the cult-like fanaticism of its adherents. They would go on for as long as you'd let them (often needing to just walk away to get them to stop) try to get you to join. The goal appears to be to obtain as much legal tender as can be pried from those who are willing to part with it. Hard sell, abusive and deceptive tactics are encouraged -- because it's so important for those who haven't "gotten it" to do so, justifying just about anything. But if you don't pay -- you get bupkis. It's a scam, and an abusive one at that. [0] https://en.wikipedia.org/wiki/Landmark_Worldwide https://en.wikipedia.org/wiki/Landmark_Worldwide
  • ziknard 1y ago
    There is a words for people who go to EST: EST-holes.
• Mizza 1y ago
  It's amphetamine. All of these people are constantly tweaking. They're annoying people to begin with, but they're all constantly yakked up and won't stop babbling. It's really obvious, I don't know why it isn't highlighted more in all these post Ziz articles.
  • Muromec 1y ago
    How do you know?
    • ajkjk 1y ago
      Presumably they mean Adderall. Plausible theory tbh. Although it's just a factor not an explanation.
    • tbrake 1y ago
      having known dozens of friends, family, roommates, coworkers etc both before and after they started them. The two biggest telltale signs - 1. tendency to produce - out of no necessity whatsoever, mind - walls of text. walls of speech will happen too but not everyone rambles. 2. Obnoxiously confident that they're fundamentally correct about whatever position they happen to be holding during a conversation with you. No matter how subjective or inconsequential. Even if they end up changing it an hour later. Challenging them on it gets you more of #1.
      • Muromec 1y ago
        I mean, I know the effects of adderall/ritalin and it's plausible, what I'm asking is whether if gp knows that for a fact or deduces from what is known.
      • MinimalAction 1y ago
        Pretty much spot on! It is frustrating to talk with these when they never admit they are wrong. They find new levels of abstractions to deal with your simpler counterarguments and it is a never ending deal unless you admit they were right.
      • 1y ago
• JKCalhoun 1y ago
  > Many of them also expect that, without heroic effort, AGI development will lead to human extinction. Odd to me. Not biological warfare? Global warming? All-out nuclear war? I guess The Terminator was a formative experience for them. (For me perhaps it was The Andromeda Strain.)
  • mitthrowaway2 1y ago
    These aren't mutually exclusive. Even in The Terminator, Skynet's method of choice is nuclear war. Yudkowsky frequency expressses concern that a malevolent AI might synthesize a bioweapon. I personally worry that destroying the ozone layer might be an easy opening volley. Either way, I don't want a really smart computer spending its time figuring out plans to end the human species, because I think there are too many ways to be successful.
    • staticman2 1y ago
      Terminator descends from a tradition of science fiction cold war parables. Even in Terminator 2 there's a line suggesting the movie isn't really about robots: John:We're not gonna make it, are we? People, I mean. Terminator: It's in your nature to destroy yourselves. Seems odd to worry about computers shooting the ozone when there's plenty of real existential threats loaded in missles aimed at you right now.
      • mitthrowaway2 1y ago
        I'm not in any way discounting the danger represented by those missiles. In fact I think AI only makes it more likely that they might someday be launched. But I will say that in my experience the error-condition that causes a system to fail is usually the one that didn't seem likely to happen, because the more obvious failure modes were taken seriously from the beginning. Is it so unusual to be able to consider more than one risk at a time?
  • myaccountonhn 1y ago
    That's what was so strange with EA and rationalist movements. A highly theoretical model that AGI could wipe us all out vs the very real issue of global warming and pretty much all emphasis was on AGI.
• 1970-01-01 1y ago
  I find it ironic that the question is asked unempirically. Where is the data stating there are many more than before? Start there, then go down the rabbit hole. Otherwise, you're concluding on something that may not be true, and trying to rationalize the answer, just as a cultist does.
  • arduanika 1y ago
    Oh come on. Anyone who's ever seen the sky knows it's blue. Anyone who's spent much time around rationalism knows the premise of this article is real. It would make zero sense to ban talking about about a serious and obvious problem in their community until some double blind peer reviewed data can be gathered. It would be what they call an "isolated demand for rigor".
• akomtu 1y ago
  It's a religion of an overdeveloped mind that hides from everything it cannot understand. It's an anti-religion, in a sense, that puts your mind on the pedestal. Note the common pattern in major religions: they tell you that thoughts and emotions obscure the light of intuition, like clouds obscure sunlight. Rationalism is the opposite: it denies the very idea of intuition, or anything above the sphere of thoughts, and tells to create as many thoughts as possible. Rationalists deny anything spiritual, good or evil, because they don't have evidence to think otherwise. They remain in this state of neutral nihilism until someone bigger than them sneaks into their ranks and casually introduces them to evil with some undeniable evidence. Their minds quickly pass the denial-anger-acceptance stages and being faithful to their rationalist doctrine they update their beliefs with what they know. From that point they are a cult. That's the story of Scientology, which has too many many parallels with Rationalism.
  • deleted 1y ago
    [deleted]
• skrebbel 1y ago
  This article is beautifully written, and it's full of proper original research. I'm sad that most comments so far are knee-jerk "lol rationalists" type responses. I haven't seen any comment yet that isn't already addressed in much more colour and nuance in the article itself.
  • mm263 1y ago
    [flagged]
    • teh_klev 1y ago
      I have a link for you: https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html Scroll to the bottom of the page.
      • mm263 1y ago
        Thank you for the link - I'm familiar with the commenting rules. I just don't think that people follow the spirit of those rules.
  • knallfrosch 1y ago
    I think it's perfectly fine to read these articles, think "definitely a cult" and ignore whether they believe in spaceships, or demons, or AGI. The key takeaway from the article is that if you have a group leader who cuts you off from other people, that's a red flag – not really a novel, or unique, or situational insight.
    • andrewflnr 1y ago
      That's a side point of the article, acknowledged as an old idea. The central points of this article are actually quite a bit more interesting than that. He even summarized his conclusions concisely at the end, so I don't know what your excuse is for trivializing it.
    • ethbr1 1y ago
• SpaceManNabs 1y ago
  Cause they all read gwern and all eugenics leads into cults because conspiracy adjacent garbo always does.
• dav_Oz 1y ago
  For me largley shaped by the westering old Europe creaking and breaking (after 2 WWs) under its heavy load of philosophical/metaphysical inheritance (which at this point in time can be considered effectively americanized). It is still fascinating to trace back the divergent developments like american-flavoured christian sects or philosophical schools of "pragmatism", "rationalism" etc. which get super-charged by technological disruptions. In my youth I was heavily influenced by the so-called Bildung which can be functionally thought of as a form of ersatz religion and is maybe better exemplified in the literary tradition of the Bildungsroman. I've grappled with and wildly fantasized about all sorts of things, experimented mindlessly with all kinds of modes of thinking and consciousness amidst my coming-of-age, in hindsight without this particular frame of Bildung left by myself I would have been left utterly confused and maybe at some point acted out on it. By engaging with books like Der Zauberberg by Thomas Mann or Der Mann ohne Eigenschaften by Robert Musil, my apparent madness was calmed down and instead of breaking the dam of a forming social front of myself with the vastness of the unconsciousness, over time I was guided to develop my own way into slowly operating it appropriately without completely blowing myself up into a messiah or finding myself eternally trapped in the futility and hopelessness of existence. Borrowing from my background, one effective vaccination which spontaneously came up in my mind against rationalists sects described here, is Schopenhauer's Die Welt als Wille und Vorstellung which can be read as a radical continuation of Kant's Critique of Pure Reason which was trying to stress test the ratio itself. [To demonstrate the breadth of Bildung in even something like the physical sciences e.g. Einstein was familiar with Kant's a priori framework of space and time, Heisenberg's autobiographical book Der Teil und das Ganze was motivated by: "I wanted to show that science is done by people, and the most wonderful ideas come from dialog".] Schopenhauer arrives at the realization because of the groundwork done by Kant (which he heavily acknowledges): that there can't even exist a rational basis for rationality itself, that it is simply an exquisitely disguised tool in the service of the more fundamental will i.e. by its definition an irrational force. Funny little thought experiment but what consequences does this have? Well, if you are declaring the ratio as your ultima ratio you are just fooling yourself in order to be able to rationalize anything you want. Once internalized Schopenhauer's insight gets you overwhelmed by Mitleid for every conscious being, inoculating you against the excesses of your own ratio. It instantly hit me with the same force as MDMA but several years before.
• jameslk 1y ago
  Over rationalizing is paperclip maximizing
• idontwantthis 1y ago
  Does anyone else feel that “rationality” is the same as clinical anxiety? I’m hyper rational when I don’t take my meds. I’m also insane. But all of my thoughts and actions follow a carefully thought out sequence.
• rogerkirkness 1y ago
  Because they have serious emotional maturity issues leading to lobotomizing their normal human emotional side of their identity and experience of life.
• AndrewKemendo 1y ago
  I was on LW when it emerged from the OB blog and back then it was a interesting and engaging group, though even then there were like 5 “major” contributors - most of which had no coherent academic or commercial success. As soon as those “sequences” were being developed it was clearly turning into a cult around EY, that I never understood and still don’t. This article did a good job of covering the history since and was really well written. Water finds its own level
• gwbas1c 1y ago
  Many years ago I met Eliezer Yudkowsky. He handed me a pamphlet extolling the virtues of rationality. The whole thing came across as a joke, as a parody of evangelizing. We both laughed. I glanced at it once or twice and shoved it into a bookshelf. I wish I kept it, because I never thought so much would happen around him.
  • yubblegum 1y ago
    imo These people are promoted. You look at their backgrounds and there is nothing that justifies their perches. Eliezer Yudkowsky is (iirc) a Thiel baby, isn't he?
    • next_xibalba 1y ago
      Yep. Thiel funded Yudkowsky’s Singularity Institute. Thiel seems to have soured on the rationalists though as he has repeatedly criticized “the East Bay rationalists” in his public remarks. He also apparently thinks he helped create a Black Pill monster in Yudkowsky and his disciples which ultimately led to Sam Altman’s brief ousting from Open AI.
  • quickthrowman 1y ago
    I only know Eliezer Yudkowsky from his Harry Potter fanfiction, most notably Harry Potter and the Methods of Rationality. Is he known publicly for some other reason?
    • meowface 1y ago
      He's considered the father of rationalism and the father of AI doomerism. He wrote this famous article in Time magazine a few years ago: https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-not-enough/ https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-no... His book If Anyone Builds It, Everyone Dies comes out in a month: https://www.amazon.com/Anyone-Builds-Everyone-Dies-Superhuman/dp/0316595640 https://www.amazon.com/Anyone-Builds-Everyone-Dies-Superhuma... You can find more info here: https://en.wikipedia.org/wiki/Eliezer_Yudkowsky https://en.wikipedia.org/wiki/Eliezer_Yudkowsky
• deleted 1y ago
  [deleted]
• Liftyee 1y ago
  Finally, something that properly articulates my unease when encountering so-called "rationalists" (especially the ones that talk about being "agentic", etc.). For some reason, even though I like logical reasoning, they always rubbed me the wrong way - probably just a clash between their behavior and my personal values (mainly humility).
• psunavy03 1y ago
  A problem with this whole mindset is that humans, all of us, are only quasi-rational beings. We all use System 1 ("The Elephant") and System 2 ("The Rider") thinking instinctively. So if you end up in deep denial about your own capacity for irrationality, I guess it stands to reason you could end up getting led down some deep dark rabbit holes.
  • Muromec 1y ago
    Wasn't the "fast&slow" thingy debunked as another piece of popscience?
    • psunavy03 1y ago
      The point remains. People are not 100 percent rational beings, never have been, never will be, and it's dangerous to assume that this could ever be the case. Just like any number of failed utopian political movements in history that assumed people could ultimately be molded and perfected.
      • simpaticoder 1y ago
        Those of us who accept this limitation can often fail to grasp how much others perceive it as a profound attack on the self. To me, it is a basic humility - that no matter how much I learn, I cannot really transcend the time and place of my birth, the biology of my body, the quirks of my culture. Rationality, though, promises that transcendence, at least to some people. And look at all the trouble such delusion has caused, for example "presentism". Science fiction often introduces a hidden coordinate system, one of language and predicate, upon which reason can operate, but system itself did not come from reason, but rather a storyteller's aesthetic.
    • navane 1y ago
      I think duality gets debunked every couple of hundred years
    • lmm 1y ago
      No?
• vehemenz 1y ago
  I get the impression that these people desperately want to study philosophy but for some reason can't be bothered to get formal training because it would be too humbling for them. I call it "small fishbowl syndrome," but maybe there's a better term for it.
  • 1attice 1y ago
    My thoughts exactly! I'm a survivor of ten years in the academic philosophy trenches and it just sounds to me like what would happen if you left a planeload of undergraduates on a _Survivor_ island with an infinite supply of pizza pockets and adderall
    • joeblubaugh 1y ago
      Funny that this also describes these cult rationalist groups very well.
  • username332211 1y ago
    The reason why people can't be bothered to get formal training is that modern philosophy doesn't seem that useful. It was a while ago, but take the infamous story of the 2006 rape case in Duke University. If you check out coverage of that case, you get the impression every member of faculty that joined in the hysteria was from some humanities department, including philosophy. And quite a few of them refused to change their mind even as the prosecuting attorney was being charged with misconduct. Compare that to Socrates' behavior during the trial of the admirals in 406 BC. Meanwhile, whatever meager resistence was faced by that group seems to have come from economists, natural scientist or legal scholars. I wouldn't blame people for refusing to study in a humanities department where they can't tell right from wrong.
    • freejazz 1y ago
      >The reason why people can't be bothered to get formal training is that modern philosophy doesn't seem that useful. But rationalism is?
• caycep 1y ago
  Granted, admitted from what little I've read on the outside, the "rational" part just seems to be mostly the writing style - this sort of dispassionate, eloquently worded prose that makes weird ideas seem more "rational" and logical than they really are.
  • knallfrosch 1y ago
    Yes, they're not rational at all. They're just a San-Francisco/Bay area cult who use that word.
• mordnis 1y ago
  I really like your suggestions, even for non-rationalists.
• Atlas667 1y ago
  Narcissism and Elitism justified by material wealth. What else? Rationalism isn't any more "correct" and "proper" thinking than Christianity and Buddhism claim to espouse.
• jmull 1y ago
  I think rationalist cults work exactly the same as religious cults. They promise to have all the answers, to attract the vulnerable. The answers are convoluted and inscrutable, so a leader/prophet interprets them. And doom is neigh, providing motivation and fear to hold things together. It's the same wolf in another sheep's clothing. And people who wouldn't join a religious cult -- e.g. because religious cults are too easy to recognize since we're all familiar with them, or because religions hate anything unusual about gender -- can join a rationalist cult instead.
• mathattack 1y ago
  On a recent Mindscape podcast Sean Carroll mentioned that rationalists are rational about everything except accusations that they're not being rational.
  • doubleunplussed 1y ago
    I mean you have to admit that that's a bit of a kafkatrap
• Jtsummers 1y ago
  > Many of them also expect that, without heroic effort, AGI development will lead to human extinction. > These beliefs can make it difficult to care about much of anything else: what good is it to be a nurse or a notary or a novelist, if humanity is about to go extinct? Replace AGI causing extinction with the Rapture and you get a lot of US Christian fundamentalists. They often reject addressing problems in the environment, economy, society, etc. because the Rapture will happen any moment now. Some people just end up stuck in a belief about something catastrophic (in the case of the Rapture, catastrophic for those left behind but not those raptured) and they can't get it out of their head. For individuals who've dealt with anxiety disorder, catastrophizing is something you learn to deal with (and hopefully stop doing), but these folks find a community that reinforces the belief about the pending catastrophe(s) and so they never get out of the doom loop.
  • tines 1y ago
    The Rapture isn't doom for the people who believe in it though (except in the lost sense of the word), whereas the AI Apocalypse is, so I'd put it in a different category. And even in that category, I'd say that's a pretty small number of Christians, fundamentalist or no, who abandon earthly occupations for that reason.
    • Jtsummers 1y ago
      Yes, I removed a parenthetical "(or euphoria loop for the Rapture believers who know they'll be saved)". But I removed it because not all who believe in the Rapture believe they will be saved (or have such high confidence) and, for them, it is a doom loop. Both communities, though, end up reinforcing the belief amongst their members and tend towards increasing isolation from the rest of the world (leading to cultish behavior, if not forming a cult in the conventional sense), and a disregard for the here and now in favor of focusing on this impending world changing (destroying or saving) event.
    • JohnMakin 1y ago
      I don't mean to well ackshually you here, but there are several different theological beliefs around the Rapture, some of which believe Christians will remain during the theoretical "end times." The megachurch/cinema version of this very much believes they won't, but, this is not the only view, either in modern times or historically. Some believe it's already happened, even. It's a very good analogy.
• pavlov 1y ago
  A very interesting read. My idea of these self-proclaimed rationalists was fifteen years out of date. I thought they’re people who write wordy fan fiction, but turns out they’ve reached the point of having subgroups that kill people and exorcise demons. This must be how people who had read one Hubbard pulp novel in the 1950s felt decades later when they find out he’s running a full-blown religion now. The article seems to try very hard to find something positive to say about these groups, and comes up with: “Rationalists came to correct views about the COVID-19 pandemic while many others were saying masks didn’t work and only hypochondriacs worried about covid; rationalists were some of the first people to warn about the threat of artificial intelligence.” There’s nothing very unique about agreeing with the WHO, or thinking that building Skynet might be bad… (The rationalist Moses/Hubbard was 12 when that movie came out — the most impressionable age.) In the wider picture painted by the article, these presumed successes sound more like a case of a stopped clock being right twice a day.
  • qcnguy 1y ago
    [flagged]
    • skybrian 1y ago
      It was genuinely difficult to persuade people to wear masks before everyone started doing it and it became normal.
      • qcnguy 1y ago
        Nobody was persuaded, they were forced by law exactly because it was obvious to everyone with their brain switched on that masks didn't work. Remember how when the rules demanding masks on planes were rescinded there were videos of whole planes ripping off their masks and celebrating mid-flight? Literally the second the law changed, people stopped wearing masks. That's because masks were a mass hysteria. They did not work. Everyone could see it.
        • n4r9 1y ago
• xpe 1y ago
  > The Sequences make certain implicit promises. ... Some meta-commentary first... How would one go about testing if this is true? If true, then such "promises" are not written down -- they are implied. So one would need to ask at least two questions: 1. Did the author intend to make these implicit promises? 2. What portion of readers perceive them as such? > ... There is an art of thinking better ... First, this isn't _implicit_ in the Sequences; it is stated directly. In any case, the quote does not constitute a promise: so far, it is a claim. And yes, rationalists do think there are better and worse ways of thinking, in the sense of "what are more effective ways of thinking that will help me accomplish my goals?" > ..., and we’ve figured it out. Codswallop. This is not a message of the rationality movement -- quite the opposite. We share what we've learned and why we believe it to be true, but we don't claim we've figured it all out. It is better to remain curious. > If you learn it, you can solve all your problems... Bollocks. This is not claimed implicitly or explicitly. Besides, some problems are intractable. > ... become brilliant and hardworking and successful and happy ... Rubbish. > ..., and be one of the small elite shaping not only society but the entire future of humanity. Bunk. For those who haven't read it, I'll offer a relevant extended quote from Yudkowsky's 2009 "Go Forth and Create the Art!" [1], the last post of the Sequences: ## Excerpt from Go Forth and Create the Art But those small pieces of rationality that I've set out... I hope... just maybe... I suspect—you could even call it a guess—that there is a barrier to getting started, in this matter of rationality. Where by default, in the beginning, you don't have enough to build on. Indeed so little that you don't have a clue that more exists, that there is an Art to be found. And if you do begin to sense that more is possible—then you may just instantaneously go wrong. As David Stove observes—I'm not going to link it, because it deserves its own post—most "great thinkers" in philosophy, e.g. Hegel, are properly objects of pity. That's what happens by default to anyone who sets out to develop the art of thinking; they develop fake answers. When you try to develop part of the human art of thinking... then you are doing something not too dissimilar to what I was doing over in Artificial Intelligence. You will be tempted by fake explanations of the mind, fake accounts of causality, mysterious holy words, and the amazing idea that solves everything. It's not that the particular, epistemic, fake-detecting methods that I use, are so good for every particular problem; but they seem like they might be helpful for discriminating good and bad systems of thinking. I hope that someone who learns the part of the Art that I've set down here, will not instantaneously and automatically go wrong, if they start asking themselves, "How should people think, in order to solve new problem X that I'm working on?" They will not immediately run away; they will not just make stuff up at random; they may be moved to consult the literature in experimental psychology; they will not automatically go into an affective death spiral around their Brilliant Idea; they will have some idea of what distinguishes a fake explanation from a real one. They will get a saving throw. It's this sort of barrier, perhaps, which prevents people from beginning to develop an art of rationality, if they are not already rational. And so instead they... go off and invent Freudian psychoanalysis. Or a new religion. Or something. That's what happens by default, when people start thinking about thinking. I hope that the part of the Art I have set down, as incomplete as it may be, can surpass that preliminary barrier—give people a base to build on; give them an idea that an Art exists, and somewhat of how it ought to be developed; and give them at least a saving throw before they instantaneously go astray. That's my dream—that this highly specialized-seeming art of answering confused questions, may be some of what is needed, in the very beginning, to go and complete the rest. [1]: https://www.lesswrong.com/posts/aFEsqd6ofwnkNqaXo/go-forth-and-create-the-art https://www.lesswrong.com/posts/aFEsqd6ofwnkNqaXo/go-forth-a...
• IX-103 1y ago
  Is it really that surprising that a group of humans who think they have some special understanding of reality compared to others tend to separate and isolate themselves until they fall into an unguided self-reinforcing cycle? I'd have thought that would be obvious since it's the history of many religions (which seem to just be cults that survived the bottleneck effect to grow until they reached a sustainable population). In other words, humans are wired for tribalism, so don't be surprised when they start forming tribes...
• guerrilla 1y ago
  > One way that thinking for yourself goes wrong is that you realize your society is wrong about something, don’t realize that you can’t outperform it, and wind up even wronger. I've been there myself. > And without the steadying influence of some kind of external goal you either achieve or don’t achieve, your beliefs can get arbitrarily disconnected from reality — which is very dangerous if you’re going to act on them. I think this and the entire previous two paragraphs preceding it are excellent arguments for philosophical pragmatism and empiricism. It's strange to me that the community would not have already converged on that after all their obsessions with decision theory. > The Zizians and researchers at Leverage Research both felt like heroes, like some of the most important people who had ever lived. Of course, these groups couldn’t conjure up a literal Dark Lord to fight. But they could imbue everything with a profound sense of meaning. All the minor details of their lives felt like they had the fate of humanity or all sentient life as the stakes. Even the guilt and martyrdom could be perversely appealing: you could know that you’re the kind of person who would sacrifice everything for your beliefs. This helps me understand what people mean by "meaning". A sense that their life and actions actually matter. I've always struggled to understand this issue but this helps make it concrete, the kind of thing people must be looking for. > One of my interviewees speculated that rationalists aren’t actually any more dysfunctional than anywhere else; we’re just more interestingly dysfunctional. "We're"? The author is a rationalist too? That would definitely explain why this article is so damned long. Why are rationalists not able to write less? It sounds like a joke but this is seriously a thing. [EDIT: Various people further down in the comments are saying it's amphetamines and yes, I should have known that from my own experience. That's exactly what it is.] > Consider talking about “ethical injunctions:” things you shouldn’t do even if you have a really good argument that you should do them. (Like murder.) This kind of defeats the purpose, doesn't it? Also, this is nowhere justified in the article, just added on as the very last sentence.
  • s1artibartfast 1y ago
    >I think this and the entire previous two paragraphs preceding it are excellent arguments for philosophical pragmatism and empiricism. It's strange to me that the community would not have already converged on that after all their obsessions with decision theory They did! One of the great ironies inside the community is that they are and openly admit to being empiricists. They reject most of the French/European rationalist cannon. >Why are rationalists not able to write less? The answer is a lot more boring. They like to write and they like to think. They also think by writing. It is written as much for themselves as for anyone else, probably more.
• RareAz 1y ago
  [flagged]
• duckmysick 1y ago
  > And yet, the rationalist community has hosted perhaps half a dozen small groups with very strange beliefs (including two separate groups that wound up interacting with demons). Some — which I won’t name in this article for privacy reasons — seem to have caused no harm but bad takes. So there's six questionable (but harmless) groups and then later the article names three of them as more serious. Doesn't seem like "many" to me. I wonder what percentage of all cults are the rationalist ones.
• hax0ron3 1y ago
  The premise of the article might just be nonsense. How many rationalists are there in the world? Of course it depends on what you mean by rationalist, but I'd guess that there are probably several tens of thousands, at very least, people in the world who either consider themselves rationalists or are involved with the rationalist community. With such numbers, is it surprising that there would be half a dozen or so small cults? There are certainly some cult-like aspects to certain parts of the rationalist community, and I think that those are interesting to explore, but come on, this article doesn't even bother to establish that its title is justified. To the extent that rationalism does have some cult-like aspects, I think a lot of it is because it tends to attract smart people who are deficient in the ability to use avenues other than abstract thinking to comprehend reality and who enjoy making loosely justified imaginative leaps of thought while overestimating their own abilities to model reality. The fact that a huge fraction of rationalists are sci-fi fans is not a coincidence. But again, one should first establish that there is anything actually unusual about the number of cults in the rationalist community. Otherwise this is rather silly.
• pizzadog 1y ago
  I have a lot of experience with rationalists. What I will say is: 1) If you have a criticism about them or their stupid name or how "'all I know is that I know nothing' how smug of them to say they're truly wise," rest assured they have been self flagellating over these criticisms 100x longer than you've been aware of their group. That doesn't mean they succeeded at addressing the criticisms, of course, but I can tell you that they are self aware. Especially about the stupid name. 2) They are actually well read. They are not sheltered and confused. They are out there doing weird shit together all the time. The kind of off-the-wall life experiences you find in this community will leave you wide eyed. 3) They are genuinely concerned with doing good. You might know about some of the weird, scary, or cringe rationalist groups. You probably haven't heard about the ones that are succeeding at doing cool stuff because people don't gossip about charitable successes. In my experience, where they go astray is when they trick themselves into working beyond their means. The basic underlying idea behind most rationalist projects is something like "think about the way people suffer everyday. How can we think about these problems in a new way? How can we find an answer that actually leaves everyone happy?" A cynic (or a realist, depending on your perspective) might say that there are many problems that fundamentally will leave some group unhappy. The overconfident rationalist will challenge that cynical/realist perspective until they burn themselves out, and in many cases they will attract a whole group of people who burn out alongside them. To consider an extreme case, the Zizians squared this circle by deciding that the majority of human beings didn't have souls and so "leaving everyone happy" was as simple as ignoring the unsouled masses. In less extreme cases this presents itself as hopeless idealism, or a chain of logic that becomes so divorced from normal socialization that it appears to be opaque. "This thought experiment could hypothetically create 9 quintillion cubic units of Pain to exist, so I need to devote my entire existence towards preventing it, because even a 1% chance of that happening is horrible. If you aren't doing the same thing then you are now morally culpable for 9 quintillion cubic units of Pain. You are evil." Most rationalists are weird but settle into a happy place far from those fringes where they have a diet of "plants and specifically animals without brains that cannot experience pain" and they make $300k annually and donate $200k of it to charitable causes. The super weird ones are annoying to talk to and nobody really likes them.
  • wredcoll 1y ago
• gwbas1c 1y ago
  One thing I'm having trouble with: The article assumes the reader knows some history about the rationalists. I listened to a podcast that covered some of these topics, so I'm not lost; but I think someone who's new to this topic will be very, very, confused.
  • clueless 1y ago
    I'm curious, what was the podcast episode?
    • gwbas1c 1y ago
      Stuff you should Know, Who are the Zizians. I can't find a direct link, but if you search for "Who are the Zizians?", you'll find it at https://stuffyoushouldknow.com/episodes/ https://stuffyoushouldknow.com/episodes/
  • guerrilla 1y ago
    Here you go. It has like 10 chapters, so keep going once you reach the end. https://aiascendant.substack.com/p/extropias-children-chapter-1-the-wunderkind https://aiascendant.substack.com/p/extropias-children-chapte...
• kanzure 1y ago
  Here are some other anti-lesswrong materials to consider: https://aiascendant.com/p/extropias-children-chapter-1-the-wunderkind https://aiascendant.com/p/extropias-children-chapter-1-the-w... https://davidgerard.co.uk/blockchain/2023/02/06/ineffective-altruism-ftx-and-the-future-robot-apocalypse/ https://davidgerard.co.uk/blockchain/2023/02/06/ineffective-... https://www.bloomberg.com/news/features/2023-03-07/effective-altruism-s-problems-go-beyond-sam-bankman-fried https://www.bloomberg.com/news/features/2023-03-07/effective... https://www.vox.com/future-perfect/23458282/effective-altruism-sam-bankman-fried-ftx-crypto-ethics https://www.vox.com/future-perfect/23458282/effective-altrui... https://qchu.substack.com/p/eliezer https://qchu.substack.com/p/eliezer https://x.com/kanzure/status/1726251316513841539 https://x.com/kanzure/status/1726251316513841539
  • meowface 1y ago
    Note that Asterisk magazine is basically the unofficial magazine for the rationalism community and the author is a rationalist blogger who is naturally very pro-LessWrong. This piece is not anti-Yudkowsky or anti-LessWrong. Here's a counter-piece on David Gerard and his portrayal of LessWrong and Effective Altruism: https://www.tracingwoodgrains.com/p/reliable-sources-how-wikipedia-admin https://www.tracingwoodgrains.com/p/reliable-sources-how-wik...
• antithesizer 1y ago
  Little on offer but cults these days. Take your pick. You probably already did long ago and now your own cult is the only one you'll never clock as such.
  • noqc 1y ago
    Same as it ever was, but with more of them, people are a little warier about their own, I think.
• crazydoggers 1y ago
  Trying to find life’s answers by giving over your self authority to another individual or group’s philosophy is not rational. Submitting oneself to an authority who’s role is telling people what’s best in life will always lead to attracting the type of people looking to control, take advantage and traumatize others.
• Lerc 1y ago
  Reading the other comments makes me wonder if they just misread the sign and they were looking for the rationalizationist meeting.
• fuzztester 1y ago
  Has anyone here ever been a part of a cult? If so, got anything to share - anecdotes, learnings, cautions, etc.? I am never planning to be part of one; just interested to know, partly because I have lived adjacent to what might be one, at times.
  • lyu07282 1y ago
    There was this interview with Diane Benscoter who talked about her experience and reasons for joining a cult that I found very insightful: https://www.youtube.com/watch?v=6Ibk5vJ-4-o https://www.youtube.com/watch?v=6Ibk5vJ-4-o The main point is that it isn't so much the cult (leader) so much as the victims being in a vulnerable mental state getting exploited.
    • fuzztester 1y ago
      will check out the video, thanks.
  • Viliam1234 1y ago
    There are entire books written by former cult members. Some examples (I just copied the list, didn't check each one of them): "Beyond Belief: My Secret Life Inside Scientology and My Harrowing Escape" by Jenna Miscavige Hill "A Billion Years: My Escape From a Life in the Highest Ranks of Scientology" by Mike Rinder "Blown for Good: Behind the Iron Curtain of Scientology" by Marc Headley "Satan Created the Cult: Memoirs of an Escapee" by Mona Vasquez "Escape" by Carolyn Jessop "The Witness Wore Red" by Rebecca Musser "Breaking Free" by Rachel Jeffs "Stolen Innocence" by Elissa Wall "The Sound of Gravel" by Ruth Wariner "Daughter of Gloriavale" by Lilia Tarawa "My Life in Orange" by Tim Guest "Crazy for God" by Christopher Edwards "In the Days of Rain: A Daughter, a Father, a Cult" by Rebecca Stott "Cartwheels in a Sari" by Jayanti Tamm "Seductive Poison" by Deborah Layton "Hollywood Park" by Mikel Jollett "Unfollow: A Journey from Hatred to Hope" by Megan Phelps-Roper "Crisis of Conscience" by Raymond Franz "Leaving the Saints" by Martha Beck "Forager: Field Notes for Surviving a Family Cult" by Michelle Dowd "Dinner for Vampires: Life on a Cult TV Show (While Also in an Actual Cult)" by Bethany Joy Lenz
• gverrilla 1y ago
  lol
• marstall 1y ago
  Harpers did an amazing cover story on these freaks in 2015 https://harpers.org/archive/2015/01/come-with-us-if-you-want-to-live/ https://harpers.org/archive/2015/01/come-with-us-if-you-want...
• homeonthemtn 1y ago
  This just sounds like any other community based around a niche interest. From kink to rock hounding, there's always people who base their identity on being a broker of status or power because they themselves are a powerless outsider once removed from the community
  • gchamonlive 1y ago
    > base their identity on being a broker of status or power because they themselves are a powerless outsider once removed from the community Who would ever maintain power when removed from their community? You mean to say they base their identity on the awareness of the power they possess within a certain group?
• duxup 1y ago
  Great read. I remember going to college and some graduate student, himself a philosophy major, telling me that nobody is as big a jerk as philosophy majors. I don't know if it is really true, but it certainly felt true that folks looking for deeper answers about a better way to think about things end up finding what they believe is the "right" way and that tends to lead to branding other options as "wrong". A search for certainty always seems to be defined or guided by people dealing with their own issues and experiences that they can't explain. It gets tribal and very personal and those kind of things become dark rabbit holes. ---- >Jessica Taylor, an AI researcher who knew both Zizians and participants in Leverage Research, put it bluntly. “There’s this belief [among rationalists],” she said, “that society has these really bad behaviors, like developing self-improving AI, or that mainstream epistemology is really bad–not just religion, but also normal ‘trust-the-experts’ science. That can lead to the idea that we should figure it out ourselves. And what can show up is that some people aren't actually smart enough to form very good conclusions once they start thinking for themselves.” Reminds me of some members of our government and conspiracy theorists who "research" and encourage people to figure it out themselves ...
• jmoggr 1y ago
  I think the comments here have been overly harsh. I have friends in the community and have visited the LessWrong "campus" several times. They seemed very welcoming, sincere, and were kind and patient even when I was basically asserting that several of their beliefs were dumb (in hopefully somewhat respectful manner). As for the AI doomerism, many in the community have more immediate and practical concerns about AI, however the most extreme voices are often the most prominent. I also know that there has been internal disagreement on the kind of messaging they should be using to raise concern. I think rationalists get plenty of things wrong, but I suspect that many people would benefit from understanding their perspective and reasoning.
  • cynicalkane 1y ago
    > They seemed very welcoming, sincere, and were kind and patient even when I was basically asserting that several of their beliefs were dumb I don't think LessWrong is a cult (though certainly some of their offshoots are) but it's worth pointing out this is very characteristic of cult recruiting. For cultists, recruiting cult fodder is of overriding psychological importance--they are sincere, yes, but the consequences are not what you and I would expect from sincere people. Devotion is not always advantageous.
    • directevolve 1y ago
      Does insincerity, cruelty, unfriendliness, and impatience make a community less likely to be a cult?
  • rsynnott 1y ago
    > They seemed very welcoming, sincere, and were kind and patient even when I was basically asserting that several of their beliefs were dumb I mean, I'm not sure what that proves. A cult which is reflexively hostile to unbelievers won't be a very effective cult, as that would make recruitment almost impossible.
• a_bonobo 1y ago
  This is a great article. There's so much in these group dynamics that repeats group dynamics of communist extremists of the 70s. A group that has found a 'better' way of life, all you have to do is believe in the group's beliefs. Compare this part from OP: >Here is a sampling of answers from people in and close to dysfunctional groups: “We spent all our time talking about philosophy and psychology and human social dynamics, often within the group.” “Really tense ten-hour conversations about whether, when you ate the last chip, that was a signal that you were intending to let down your comrades in selfish ways in the future.” This reeks of Marxist-Leninist self-criticism, where everybody tried to up each other in how ideologically pure they were. The most extreme outgrowing of self-criticism is when the Japanese United Red Army beat its own members to death as part of self-criticisms. >'These violent beatings ultimately saw the death of 12 members of the URA who had been deemed not sufficiently revolutionary.' https://en.wikipedia.org/wiki/United_Red_Army https://en.wikipedia.org/wiki/United_Red_Army History doesn't repeat, but it rhymes.
• narrator 1y ago
  I think everyone should be familiar with hermeticism because its various mystery cults have been with us since Hermes Trismegistus laid down its principles in Ancient Egypt on the Emerald Tablets. It was where early science like practices like alchemy originated, but that wheat got separated out from the chaff during the renaissance and the more coercive control aspects remained. That part, how to get people to follow you and fight for you and maintain a leadership hierarchy is extremely old technology. They essentially use this glitch in human psychology that gets exploited over and over again. The glitch is a more generalized version of the advanced fee scam. You tell people that if we just believe something can be true, it can be true. Then we discriminate against people who don't believe in that thing. We then say only the leader(s) can make that thing true, but first you must give them all your power and support so they can fight the people who don't believe in that thing. Unfortunately, reality is much more messy than the cult leaders would have you believe, and leaders often don't have their followers best interests at heart, especially those who follow blindly, or even the ability to make the thing true that everyone wants to be true, but use it as this white rabbit that everyone in the cult has to chase after forever.
• deleted 1y ago
  [deleted]
• Animats 1y ago
  What's the scale of this thing in the SF Bay Area? 100 people? 1000 people? 10,000 people?
• marcus_holmes 1y ago
  What is it about San Francisco that makes it the global center for this stuff? Reading this, I was reminded of the 60's hippy communes, that generally centered around SF, and the problems they reported. So similar, especially around that turning-inward group emotional dynamics problem, that such groups tend to become dysfunctional (as TFA says) by blowing up internal emotional group politics into huge problems that need the entire group to be involved in trying to heal (as opposed to, say, accepting that a certain amount of interpersonal conflict is inevitable in human groups and ignoring it). It's fascinating that the same kind of groups seem to encounter the same kind of problems despite being ~60 years apart and armed with a lot more tech and knowledge. And, yeah, why SF?
• sriram_malhar 1y ago
  Scott Aaronson had this view of one rationalist group. "Guess I’m A Rationalist Now" https://scottaaronson.blog/?p=8908 https://scottaaronson.blog/?p=8908
• sriram_malhar 1y ago
  It is so strange that this article would hijack the term "rationalist" to mean this extraordinarily specific set of people "drawn together by AI researcher Eliezer Yudkowsky’s blog post series The Sequences, a set of essays about how to think more rationally". As a counter example (with many many more people) is the Indian Rationalist Association (https://en.wikipedia.org/wiki/Indian_Rationalist_Association https://en.wikipedia.org/wiki/Indian_Rationalist_Association) to "promote scientific skepticism and critique supernatural claims". This isn't a cult of any kind, even if the members broadly agree about what it means to be rational with the set above.
• morpheos137 1y ago
  Because Yudkowskian rationalism is a scifi inspired cult it's. If someone believes in the singularliy my estimation of their intellectual capacity or at least maturity diminishes.
• BrenBarn 1y ago
  This is a very interesting article. It's surprising though to see it not use the term "certainty" at all. (It only uses "certain" in a couple instances of like "a certain X" and one use of "certainly" for generic emphasis.) Most of what the article says makes sense, but it seems to sidestep the issue that a major feature distinguishing the "good" rationalists from the "bad" is that the bad ones are willing to take very extreme actions in support of their beliefs. This is not coincidentally something that distinguishes good believers in various religions or philosophies from bad believers (e.g., people who say God told them to kill people). This is also lurking in the background of discussion of those who "muddled through" or "did the best they could". The difference is not so much in the beliefs as in the willingness to act on them, and that willingness is in turn largely driven by certainty. I think it's plausible there is a special dimension to rationalism that may exacerbate this, namely a tendency of rationalists to feel especially "proud" of their beliefs because of their meta-belief that they derived their beliefs rationally. Just like an amateur painter may give themselves extra brownie points because no one taught them how to paint, my impression of rationalists is that they sometimes give themselves an extra pat on the back for "pulling themselves up by their bootstraps" in the sense of not relying on faith or similar "crutches" to determine the best course of action. This can paradoxically increase their certainty in their beliefs when actually it's often a warning that those beliefs may be inadequately tested against reality. I always find it a bit odd that people who profess to be rationalists can propose or perform various extreme acts, because it seems to me that one of the strongest and most useful rational beliefs is that your knowledge is incomplete and your beliefs are almost surely not as well-grounded as you think they are. (Certainly no less an exponent of reason than Socrates was well aware of this.) This on its own seems sufficient to me to override some of the most absurd "rationalist" conclusions (like that you should at all costs become rich or fix Brent Dill's depression). It's all the more so when you combine it with some pretty common-sense forecasts of what might happen if you're wrong. (As in, if you devote your life to curing Brent Dill's depression on the theory that he will then save the world, and he turns out to be just an ordinary guy or worse, you wasted your life curing one person's depression when you yourself could have done more good with your own abilities, just by volunteering at a soup kitchen or something.) It's never made sense to me that self-described rationalists could seriously consider some of these possible courses of action in this light. Sort of related is the claim at the end that rationalists "want to do things differently from the society around them". It's unclear why this would be a rational desire. It might be rational in a sense to say you want to avoid being influenced by the society around you, but that's different from affirmatively wanting to differ from it. This again suggests a sort of "psychological greed" to reach a level of certainty that allows you to confidently, radically diverge from society, rather than accepting that you may never reach a level of certainty that allows you to make such deviations on a truly rational basis. It's also interesting to me that the article focuses a lot not on rationalist belief per se, but on the logistics and practices of rationalist communities. This in itself seems like a warning that the rationality of rationalism is not all it's cracked up to be. It's sort of like, you can try to think as logically as possible, but if you hit yourself in the head with a hammer every day you're likely going to make mistakes anyway. And some of the "high demand" practices mentioned seem like slightly less severe psychological versions of that.
• low_tech_love 1y ago
  The thing with identifying yourself with an “ism” (e.g. rationalism, feminism, socialism) is that, even though you might not want that, you’re inherently positioning yourself in a reductionist and inaccurate corner of the world. Or in other words you’re shielding yourself in a comfortable, but wrong, bubble. To call yourself an -ist means that you consider that you give more importance to that concept than other people—-you’re more rational than most, or care more about women than most, or care more about social issues than most. That is wrong both because there are many irrational rationalists and also because there are many rational people who don’t associate with the group (same with the other isms). The thing is that the very fact of creating the label and associating yourself with it will ruin the very thing that you strive for. You will attract a bunch of weirdos who want to be associated with the label without having to do the job that it requires, and you will become estranged from those who prefer to walk the walk instead of talk the talk. In both ways, you failed. The fact is that every ism is a specific set of thoughts and ideas that is not generic, and not broad enough to carry the weight of its name. Being a feminist does not mean you care about women; it means you are tied to a specific set of ideologies and behaviours that may or may not advance the quality of life of women in the modern world, and are definitely not the only way to achieve that goal (hence the inaccuracy of the label).
• precompute 1y ago
  Because purporting to be extra-rational about decisions is effective nerd-bait.
• user94wjwuid 1y ago
  Ted Talks became religions, podcasts sermons
• PeterStuer 1y ago
  "Nihilists! F* me. I mean, say what you will about the tenets of National Socialism, Dude, at least it's an ethos."
• m101 1y ago
  Rationality is a broken tool for understanding the world. The complexity of the world is such that there are a plethora of reasons for anything which means our ability to be sure of any relationship is limited, and hence rationality leads to an unfounded confidence in our beliefs, which is more harmful than helpful.
• rsynnott 1y ago
  > The Sequences [posts on LessWrong, apparently] make certain implicit promises. There is an art of thinking better, and we’ve figured it out. If you learn it, you can solve all your problems, become brilliant and hardworking and successful and happy, and be one of the small elite shaping not only society but the entire future of humanity. Ooh, a capital S and everything. I mean, I feel like it is fairly obvious, really. 'Rationalism' is a new religion, and every new religion spawns a bunch of weird, generally short-lived, cults. You might as well ask, in 100AD, "why are there so many weird Christian cults all of a sudden"; that's just what happens whenever any successful new religion shows up. Rationalism might be particularly vulnerable to this because it lacks a strong central authority (much like early Christianity), but even with new religions which _did_ have a strong central authority from the start, like Mormonism or Scientology, you still saw this happening to some extent.
• PicassoCTs 1y ago
  My pet theory is - that as a rationalist, you have a idealized view of humanity by nature. Your mirror neurons copy your own mind for interpolating other peoples behavior and character. Which results in a constant state of cognitive dissonance, as the people of normal society around you behave very differently and often more "rustically" then expected. The education is there- all the learning sources are there- and yet are rejected. The lessons learned from history go unlearned and are often repeated. You are in a out-group by definition and life-long, so you band together with others and get conned by cult-con-artists into foolish projects. For the "rational" are nothing but another deluded project to hijack by the sociopaths of our society. The most rational being- in fact a being so capable to predate us, society had to develop anti-bodies against socio-paths, we call religion and laws!
• coldtea 1y ago
  >There’s a lot to like about the Rationalist community Like what? Never saw anything worth while there...
• ysofunny 1y ago
  because the sacred simplicity problem, yet another label I had to coin due to necessity for example, lambda calculus, it's too simple. to the point that it's real power is immediately unbelievable. the simplest 'solution', is to make it "sacred", to infuse an aura of mystery and ineffability around the ideas. that way people will give it the proper respect proportional to the mathematical elegance without necessarily having to really grasp the details. I'm reflecting on how, for example, lambda calculus is really easy to learn to do by rote. but this does not help in truly grasping the significance that even LLM can be computed by (an inhuman) amount of symbol substitutions on paper. and how it is easy to trivialize what this really entails (fleshing out all the entailment is difficult; it's easier to act as if they have been fleshed out and mimic the awe) therefore, rationalist cults are the legacy, the latest leaf in the long succession of the simplest solution to the simplicity of the truly sacred mathematical ideas with which we can "know" (and nod to each other who also "know") what numbers fucking mean
• wolvesechoes 1y ago
  The thing I always found funny about those self-proclaimed Rationalists, or Skeptics, is how their value logic and reasoning, or science, and yet seriously lack knowledge in any of these, what leads to very naive views in that aspects.
• propter_hoc 1y ago
  The major problem I see in this group is that they have constructed a self-identity of being intelligent. This means that by being part of a Rationalist group, a person can claim to have insight into things that "the rest of the world doesn't understand." Which, because (1) self-identifying as intelligent/insightful does not mean you are actually so; (2) you are following the "brain reprogramming" processes of some kind of comvincing leader; is a straight shot to NXIVM style apocalyptic cultism.
• loning 1y ago
  [dead]