6 ms·

Rationalism? The term has been used a lot of times since Pythagoras [0], but the combination of Bay Area, Oxford, existential risks, AI safety makes it sound li

by t_mann 2y ago

Rationalism? The term has been used a lot of times since Pythagoras [0], but the combination of Bay Area, Oxford, existential risks, AI safety makes it sound like this particular movement could have formed in the same mold as Effective Altruism and Long-Termism (ie, the "it's objectively better for humanity if you give us money to buy a castle in France than whatever you'd do with it" crowd that SBF sprung from). Can somebody in know weigh in? [0] https://en.wikipedia.org/wiki/Rationalism#History https://en.wikipedia.org/wiki/Rationalism#History

hollerith 2y ago
Just like HN grew around the writing of Paul Graham, the "rationalist community" grew around the writings of Eliezer Yudkowsky. Similar to how Paul Graham no longer participates on HN, Eliezer rarely participates on http://lesswrong.com http://lesswrong.com anymore, and the benevolent dictator for life of lesswrong.com is someone other than Eliezer. Eliezer's career has always been centered around AI. At first Eliezer was wholly optimistic about AI progress. In fact, in the 1990s, I would say that Eliezer was the loudest voice advocating for the development of AI technology that would greatly exceed human cognitive capabilities. "Intentionally causing a technological singularity," was the way he phrased it in the 1990s IIRC. (Later "singularity" would be replaced by "intelligence explosion".) From 2001 to 2004 he started to believe that AI has a strong tendency to become very dangerous once it starts exceeding the human level of cognitive capabilities. Still, he hoped that before AI starts exceeding human capabilities, he and his organization could develop a methodology to keep it safe. As part of that effort, he coined the term "alignment". The meaning of the term has broadened drastically: when Eliezer coined it, he meant the creation of an AI that stays aligned with human values and human preferences even as its capabilities greatly exceed human capabilities. In contrast, these days, when you see the phrase "aligned AI", it is usually being applied to an AI system that is not a threat to people only because it's not cognitively capable enough to dis-empower human civilization. By the end of 2015, Eliezer had lost most of the hope he initially had for the alignment project in part because of conversations he had with Elon Musk and Sam Altman at an AGI conference in Puerto Rico followed by Elon and Sam's actions later that year, which actions included the founding of OpenAI. Eliezer still considers the alignment problem solvable in principle if a sufficiently-smart and sufficiently-careful team attacks it, but considers it extremely unlikely any team will manage a solution before the AI labs cause human extinction. In April 2022 he went public with his despair and announced that his organization (MIRI) will cease technical work on the alignment project and will focus on lobbying the governments of the world to ban AI (or at least the deep-learning paradigm, which he considers too hard to align) before it is too late. The rationalist movement began in November 2006 when Eliezer began posting daily about human rationality on overcomingbias.com. (The community moved to lesswrong.com in 2009, at which time overcomingbias.com became the personal blog of Robin Hanson.) The rationalist movement was always seen by Eliezer as secondary to the AI-alignment enterprise. Specifically, Eliezer hoped that by explaining to people how to become more rational, he could increase the number of people who are capable of realizing that AI research was a potent threat to human survival. To help advance this secondary project, the Center for Applied Rationality (CFAR) was founded as a non-profit in 2012. Eliezer is neither an employee nor a member of the board of this CFAR. He is employed by and on the board of the non-profit Machine Intelligence Research Institute (MIRI) which was founded in 2000 as the Singularity Institute for Artificial Intelligence. I stress that believing that AI research is dangerous has never been a requirement for posting on lesswrong.com or for participating in workshops run by CFAR. Effective altruism (EA) has separate roots, but the two communities have become close over the years, and EA organizations have donated millions to MIRI.
- EA-3167 2y ago
  A great example of superficially smart people creating echo chambers which then turn sour, but they can't escape. There's a very good reason that, "Buying your own press" is a cliched pejorative, and this is an extreme end of that. More generally it's just a depressing example of how rationalism in the LW sense has become a sort of cult-of-cults, with the same old existential dread packaged in a new "rational" form. No god here, just really unstable people.
  - hollerith 2y ago
    My explanation for why Eliezer went from vocal AI optimist to AI pessimist is that he became more knowledgeable about AI. What is your explanation? I've seen the explanation that AI pessimism helped Eliezer attract donations, but that does not work because his biggest donor when he started going public with his pessimism (2003 through 2006) was Peter Theil, who responded to his turn to pessimism by refusing to continue to donate (except for donations earmarked for studying the societal effects of AI, which is not the object of Eliezer's pessimism and not something Eliezer particularly wanted to study). I suspect that most of the accusations to the effect that MIRI or Less Wrong is a cult are lazy ad-hominems by people who have a personal interest in the AI industry or an ideological attachment to technological progress.
    - EA-3167 2y ago
      > My explanation for why Eliezer went from vocal AI optimist to AI pessimist is that he became more knowledgeable about AI. What is your explanation? He spoke to businessmen posing as experts, became increasingly self-referential, and frankly the quasi-religious subtext became text.
      - hollerith 2y ago
        Businessmen like Elon and Sam Altman, you mean?
        2y ago
nullc 2y ago
[flagged]
- deleted 2y ago
  [deleted]
- nejsjsjsbsb 2y ago
  Wow. Didn't they learn about ecosystems at school. And who says they are suffering?
  - throwaway314155 2y ago
    [flagged]
rachofsunshine 2y ago
You're correct. Those communities heavily overlap. Take, for example, 80,000 Hours, among the more prominent EA organizations. Their top donors (https://80000hours.org/about/donors/ https://80000hours.org/about/donors/) include: - SBF and Alameda Research (you probably knew this), - the Berkeley Existential Risk Initiative, founded (https://www.existence.org/team https://www.existence.org/team) by the same guy who founded CFAR (the Center for Applied Rationality, a major rationalist organization) - the "EA infrastructure fund", whose own team page (https://funds.effectivealtruism.org/team https://funds.effectivealtruism.org/team) contains the "project lead for LessWrong.com, where he tries to build infrastructure for making intellectual progress on global catastrophic risks" - the "long-term future fund", largely AI x-risk focused and so on.
throwawayk7h 2y ago
That castle was found to be more cost-effective than any other space the group could have purchased, for the simple reason that almost nobody wants castles anymore. It was chosen because it was the best calculation; the optics of it were not considered. It would be less disingenuous if you were to say EA is the "it's objectively better for humanity if you give us money to buy a conference space in France than whatever you'd do with it" crowd -- the fact that it was a castle shouldn't be relevant.
- zozbot234 2y ago
  The depressing part is that the "optics" of buying a castle are pretty good if you care about attracting interest from elite "respectable" donors, who might just look down on you if you give off the impression of being a bunch of socially inept geeks who are just obsessed with doing the most good they can for the world at large.
- nejsjsjsbsb 2y ago
  Why buy a conference space. Most pubs will give you a seperate room if you promise to spend some money at the bar. There are probably free spaces had they researched. If I am donating money and you are buying a conference space on day 1 I'd want it to be filled with experienced ex-UN field type of people and Nobel peace prize winners. Otherwise it looks like a grift.
  - bombcar 2y ago
    Somewhere between “once a year” conferences hosted at hotels and the continual conferences of a university lies the point where buying a building makes sense. The biggest downside, of course, is all your conferences are now in the same location
- wat10000 2y ago
  Nobody wants castles anymore because they’re impractical and difficult to maintain. It’s not some sort of taboo or psychological block, it’s entirely practical. Actually, the fact that people think castles are cool suggests that the going price for them is higher than their concrete utility would make it, since demand would be boosted by people who want a castle because it’s cool. Did these guys have some special use case where it made sense, or did they think they were the only ones smart enough to see that it’s actually worth buying?
bashmelek 2y ago
It bothers me also how the word “rationalist” suddenly means the LW crowd, while I keep thinking Leibniz
- washadjeffmad 2y ago
  Their sneers are longer than their memories Check your notifications Doom scroll Refresh Refre Ref
guerrilla 2y ago
Thet're virtually identical. Seven chapter history: https://aiascendant.substack.com/p/extropias-children-chapter-1-the-wunderkind https://aiascendant.substack.com/p/extropias-children-chapte...
FeepingCreature 2y ago
Rationalism is simply an error. The thing being referred to here is "LessWrong-style rationality", which is fundamentally in the empirical, not rational school. People calling it rationalism are simply confused because the words sound similar. (Of course, the actual thing is more closely "Zizian style cultish insanity", which honestly has very very little to do with LessWrong style rationality either.)
deleted 2y ago
[deleted]

by t_mann 2y ago

hollerith 2y ago

Just like HN grew around the writing of Paul Graham, the "rationalist community" grew around the writings of Eliezer Yudkowsky. Similar to how Paul Graham no longer participates on HN, Eliezer rarely participates on http://lesswrong.com http://lesswrong.com anymore, and the benevolent dictator for life of lesswrong.com is someone other than Eliezer. Eliezer's career has always been centered around AI. At first Eliezer was wholly optimistic about AI progress. In fact, in the 1990s, I would say that Eliezer was the loudest voice advocating for the development of AI technology that would greatly exceed human cognitive capabilities. "Intentionally causing a technological singularity," was the way he phrased it in the 1990s IIRC. (Later "singularity" would be replaced by "intelligence explosion".) From 2001 to 2004 he started to believe that AI has a strong tendency to become very dangerous once it starts exceeding the human level of cognitive capabilities. Still, he hoped that before AI starts exceeding human capabilities, he and his organization could develop a methodology to keep it safe. As part of that effort, he coined the term "alignment". The meaning of the term has broadened drastically: when Eliezer coined it, he meant the creation of an AI that stays aligned with human values and human preferences even as its capabilities greatly exceed human capabilities. In contrast, these days, when you see the phrase "aligned AI", it is usually being applied to an AI system that is not a threat to people only because it's not cognitively capable enough to dis-empower human civilization. By the end of 2015, Eliezer had lost most of the hope he initially had for the alignment project in part because of conversations he had with Elon Musk and Sam Altman at an AGI conference in Puerto Rico followed by Elon and Sam's actions later that year, which actions included the founding of OpenAI. Eliezer still considers the alignment problem solvable in principle if a sufficiently-smart and sufficiently-careful team attacks it, but considers it extremely unlikely any team will manage a solution before the AI labs cause human extinction. In April 2022 he went public with his despair and announced that his organization (MIRI) will cease technical work on the alignment project and will focus on lobbying the governments of the world to ban AI (or at least the deep-learning paradigm, which he considers too hard to align) before it is too late. The rationalist movement began in November 2006 when Eliezer began posting daily about human rationality on overcomingbias.com. (The community moved to lesswrong.com in 2009, at which time overcomingbias.com became the personal blog of Robin Hanson.) The rationalist movement was always seen by Eliezer as secondary to the AI-alignment enterprise. Specifically, Eliezer hoped that by explaining to people how to become more rational, he could increase the number of people who are capable of realizing that AI research was a potent threat to human survival. To help advance this secondary project, the Center for Applied Rationality (CFAR) was founded as a non-profit in 2012. Eliezer is neither an employee nor a member of the board of this CFAR. He is employed by and on the board of the non-profit Machine Intelligence Research Institute (MIRI) which was founded in 2000 as the Singularity Institute for Artificial Intelligence. I stress that believing that AI research is dangerous has never been a requirement for posting on lesswrong.com or for participating in workshops run by CFAR. Effective altruism (EA) has separate roots, but the two communities have become close over the years, and EA organizations have donated millions to MIRI.

EA-3167 2y ago
A great example of superficially smart people creating echo chambers which then turn sour, but they can't escape. There's a very good reason that, "Buying your own press" is a cliched pejorative, and this is an extreme end of that. More generally it's just a depressing example of how rationalism in the LW sense has become a sort of cult-of-cults, with the same old existential dread packaged in a new "rational" form. No god here, just really unstable people.
- hollerith 2y ago
  My explanation for why Eliezer went from vocal AI optimist to AI pessimist is that he became more knowledgeable about AI. What is your explanation? I've seen the explanation that AI pessimism helped Eliezer attract donations, but that does not work because his biggest donor when he started going public with his pessimism (2003 through 2006) was Peter Theil, who responded to his turn to pessimism by refusing to continue to donate (except for donations earmarked for studying the societal effects of AI, which is not the object of Eliezer's pessimism and not something Eliezer particularly wanted to study). I suspect that most of the accusations to the effect that MIRI or Less Wrong is a cult are lazy ad-hominems by people who have a personal interest in the AI industry or an ideological attachment to technological progress.
  - EA-3167 2y ago
    > My explanation for why Eliezer went from vocal AI optimist to AI pessimist is that he became more knowledgeable about AI. What is your explanation? He spoke to businessmen posing as experts, became increasingly self-referential, and frankly the quasi-religious subtext became text.
    - hollerith 2y ago
      Businessmen like Elon and Sam Altman, you mean?
      - 2y ago

nullc 2y ago

[flagged]

deleted 2y ago
[deleted]
nejsjsjsbsb 2y ago
Wow. Didn't they learn about ecosystems at school. And who says they are suffering?
- throwaway314155 2y ago
  [flagged]

rachofsunshine 2y ago

You're correct. Those communities heavily overlap. Take, for example, 80,000 Hours, among the more prominent EA organizations. Their top donors (https://80000hours.org/about/donors/ https://80000hours.org/about/donors/) include: - SBF and Alameda Research (you probably knew this), - the Berkeley Existential Risk Initiative, founded (https://www.existence.org/team https://www.existence.org/team) by the same guy who founded CFAR (the Center for Applied Rationality, a major rationalist organization) - the "EA infrastructure fund", whose own team page (https://funds.effectivealtruism.org/team https://funds.effectivealtruism.org/team) contains the "project lead for LessWrong.com, where he tries to build infrastructure for making intellectual progress on global catastrophic risks" - the "long-term future fund", largely AI x-risk focused and so on.

throwawayk7h 2y ago

That castle was found to be more cost-effective than any other space the group could have purchased, for the simple reason that almost nobody wants castles anymore. It was chosen because it was the best calculation; the optics of it were not considered. It would be less disingenuous if you were to say EA is the "it's objectively better for humanity if you give us money to buy a conference space in France than whatever you'd do with it" crowd -- the fact that it was a castle shouldn't be relevant.

zozbot234 2y ago
The depressing part is that the "optics" of buying a castle are pretty good if you care about attracting interest from elite "respectable" donors, who might just look down on you if you give off the impression of being a bunch of socially inept geeks who are just obsessed with doing the most good they can for the world at large.
nejsjsjsbsb 2y ago
Why buy a conference space. Most pubs will give you a seperate room if you promise to spend some money at the bar. There are probably free spaces had they researched. If I am donating money and you are buying a conference space on day 1 I'd want it to be filled with experienced ex-UN field type of people and Nobel peace prize winners. Otherwise it looks like a grift.
- bombcar 2y ago
  Somewhere between “once a year” conferences hosted at hotels and the continual conferences of a university lies the point where buying a building makes sense. The biggest downside, of course, is all your conferences are now in the same location
wat10000 2y ago
Nobody wants castles anymore because they’re impractical and difficult to maintain. It’s not some sort of taboo or psychological block, it’s entirely practical. Actually, the fact that people think castles are cool suggests that the going price for them is higher than their concrete utility would make it, since demand would be boosted by people who want a castle because it’s cool. Did these guys have some special use case where it made sense, or did they think they were the only ones smart enough to see that it’s actually worth buying?

bashmelek 2y ago

It bothers me also how the word “rationalist” suddenly means the LW crowd, while I keep thinking Leibniz

washadjeffmad 2y ago
Their sneers are longer than their memories Check your notifications Doom scroll Refresh Refre Ref

guerrilla 2y ago

Thet're virtually identical. Seven chapter history: https://aiascendant.substack.com/p/extropias-children-chapter-1-the-wunderkind https://aiascendant.substack.com/p/extropias-children-chapte...

FeepingCreature 2y ago

Rationalism is simply an error. The thing being referred to here is "LessWrong-style rationality", which is fundamentally in the empirical, not rational school. People calling it rationalism are simply confused because the words sound similar. (Of course, the actual thing is more closely "Zizian style cultish insanity", which honestly has very very little to do with LessWrong style rationality either.)

deleted 2y ago

[deleted]