8 ms·

I'm currently reading Yudkowsky's "Rationality: from AI to zombies". Not my first try, since the book is just a collection of blog posts and I found it a bit ha

by samuel 1y ago

I'm currently reading Yudkowsky's "Rationality: from AI to zombies". Not my first try, since the book is just a collection of blog posts and I found it a bit hard to swallow due its repetitiveness, so I gave up after the first 50 "chapters" the first time I tried. Now I'm enjoying it way more, probably because I'm more interested in the topic now. For those who haven't delved(ha!) into his work or have been pushed back by the cultish looks, I have to say that he's genuinelly onto something. There are a lot of practical ideas that are pretty useful for everyday thinking ("Belief in Belief", "Emergence", "Generalizing from fiction", etc...). For example, I recall being in lot of arguments that are purely "semantical" in nature. You seem to disagree about something but it's just that both sides aren't really referring to the same phenomenon. The source of the disagreement is just using the same word for different, but related, "objects". This is something that seems obvious, but the kind of thing you only realize in retrospect, and I think I'm much better equipped now to be aware of it in real time. I recommend giving it a try.

greener_grass 1y ago
I think there is an arbitrage going on where STEM types who lack background in philosophy, literature, history are super impressed by basic ideas from those subjects being presented to them by stealth. Not saying this is you, but these topics have been discussed for thousands of years, so it should at least be surprising that Yudkowsky is breaking new ground.
- samuel 1y ago
  I don't claim that his work is original (the AI related probably is, but it's just tangentially related to rationalism), but it's clearly presented and is practical. And, BTW, I could just be ignorant in a lot of these topics, I take no offense in that. Still I think most people can learn something from an unprejudiced reading.
- elt895 1y ago
  Are there other philosophy- or history-grounded sources that are comparable? If so, I’d love some recommendations. Yudkowsky and others have their problems, but their texts have an interesting points, are relatively easy to read and understand, and you can clearly see which real issues they’re addressing. From my experience, alternatives tend to fall into two categories: 1. Genuine classical philosophy, which is usually incredibly hard to read and after 50 pages I have no idea what the author is even talking about anymore. 2. Basically self help books that take one or very few idea and repeat them ad nouseam for 200 pages.
  - NoGravitas 1y ago
    I don't know if there's anything like a comprehensive high-level guide to philosophy that's any good, though of course there are college textbooks. If you want real/academic philosophy that's just more readable, I might suggest Eugene Thacker's "The Horror of Philosophy" series (starting with "In The Dust Of This Planet"), especially if you are a horror fan already.
  - ashwinsundar 1y ago
    I don't have an answer here either, but after suffering through the first few chapters of HPMOR, I've found that Yudk and others tech-bros posing as philosophers are basically like leaky, dumbed-down abstractions for core philosophical ideas. Just go to the source and read about utilitarianism and deontology directly. Yudk is like the Wix of web development - sure you can build websites but you're not gonna be a proper web developer unless you learn HTML, CSS and Javascript. Worst of all, crappy abstractions train you in some actively bad patterns that are hard to unlearn It's almost offensive - are technologists so incapable of understanding philosophy that Yudk has to reduce it down to the least common denominator they are all familiar with - some fantasy world we read about as children?
    - AnimalMuppet 1y ago
      I'd like what the original sources would have written if someone had fed them some speak-clearly pills. Yudkowsky and company may have the dumbing-down problem, but the original sources often have a clarity problem. (That's why people are still arguing about what they meant centuries later. Not just whether they were right - though they argue about that too - but what they meant.) Even better, I'd like some filtering out of the parts that are clearly wrong.
      - chipsrafferty 1y ago
        Have you considered that the ideas that philosophy discusses are very complex, and words are not a sufficient medium to describe those ideas? That's why you have people arguing over what someone meant. Dumbing it down or trying to write something unambiguous doesn't actually make it better.
- bnjms 1y ago
  I think you’re mostly right. But also that it isn’t what the Yudkowsky is (was?) trying to do with it. I think he’s trying to distill useful tools which increase baseline rationality. Religions have this. It’s what the original philosophers are missing. (At least as taught, happy to hear counter examples)
  - ashwinsundar 1y ago
    I think I'd rather subscribe to an actual religion, than listen to these weird rationalist types of people who seem to have solved the problem that is "everything". At least there is some interesting history to learn about with religion
    - bnjms 1y ago
      I would too if I could but organized religions make me uncomfortable even though I admire parts of them. Similar to my admiration you don’t need to like the rationality types or believe in their program to find one or more of their tools useful. I’ll also respond to the silent downvoters apparent disagreement. CFAR holds workshops and a summer camp for teaching rationality tools. In HPMoR Harry discusses the way he thinks and why. I read it as more of a way to discuss EY’s views in fiction as much as fiction itself.
- sixo 1y ago
  To the Stem-enlightened mind, the classical understanding and pedagogy of such ideas is underwhelming, vague, and riddled with language-game problems, compared to the precision a mathematically-rooted idea has. They're rederiving all this stuff not out of obstinacy, but because they prefer it. I don't really identify with rationalism per se, but I'm with them on this--the humanities are over-cooked and a humanity education tends to be a tedious slog through outmoded ideas divorced from reality
  - throwawaymaroon 1y ago
    [dead]
  - biofox 1y ago
    If you contextualise the outmoded ideas as part of the Great Conversation [1], and the story of how we reached our current understanding, rather than objective statements of fact, then they becomes a lot more valuable and worthy of study. [1] https://en.wikipedia.org/wiki/Great_Conversation https://en.wikipedia.org/wiki/Great_Conversation
    - jstanley 1y ago
      But isn't the content of LessWrong part of the Great Conversation too?
  - jay_kyburz 1y ago
    I have kids in high school. We sometimes talk about the difference between the black and white of math or science, and the wishy washy grey of the humanities. You can be right or wrong in math. You have can an opinion in English.
    - HDThoreaun 1y ago
      You can definitely have wrong opinions in the humanities.
    - bmacho 1y ago
      You can be right or wrong in math and philosophy. You have can an opinion in any other sciences, physics, chemistry, biology, medical sciences, history, you name it.
- HDThoreaun 1y ago
  Rationalism largely rejects continental philosophy in favor of a more analytic approach. Yes these ideas are not new, but they’re not really the mainstream stuff you’d see in philosophy, literature, or history studies. You’d have to seek out these classes specifically to find them.
  - TimorousBestie 1y ago
    They largely reject analytic philosophy as well. Austin and Whitehead are roughly as detestable to a Rationalist as Foucault and Marx. Carlyle, Chesterton and Thoreau are about the limit of their philosophical knowledge base.
  - TheAncientGeek 1y ago
    Analytical philosophy is rationalism done right.
- FeepingCreature 1y ago
  In AI finetuning, there's a theory that the model already contains the right ideas and skills, and the finetuning just raises them to prominence. Similarly in philosophic pedagogy, there's huge value in taking ideas that are correct but unintuitive and maybe have 30% buy-in and saying "actually, this is obviously correct, also here's an analysis of why you wouldn't believe it anyway and how you have to think to become able to believe it". That's most of what the Sequences are: they take from every field of philosophy the ideas that are actually correct, and say "okay actually, we don't need to debate this anymore, this just seems to be the truth because so-and-so." (Though the comments section vociferously disagrees.) And it turns out if you do this, you can discard 90% of philosophy as historical detritus. You're still taking ideas from philosophy, but which ideas matters, and how you present them matters. The massive advantage of the Sequences is they have justified and well-defended confidence where appropriate. And if you manage to pick the right answers again and again, you get a system that actually hangs together, and IMO it's to philosophy's detriment that it doesn't do this itself much more aggressively. For instance, 60% of philosophers are compatibilists. Compatibilism is really obviously correct. "What are you complaining about, that's a majority, isn't that good?" What is wrong with those 40% though? If you're in those 40%, what arguments may convince you? Repeat to taste.
  - margalabargala 1y ago
    > And it turns out if you do this, you can discard 90% of philosophy as historical detritus This is just the story of the history of philosophy. Going back hundreds of years. See Kant and Hegel for notable examples.
    - FeepingCreature 1y ago
      Sure, and I agree that LW is doing philosophy in that sense.
      - margalabargala 1y ago
        Sure, I just object to the characterisation of "actually correct", as though each of those ideas has not gone back and forth on philosophers thinking that particular idea is "actually correct" for centuries. LW does not appear to have much if any novel insight; just much better marketing.
deleted 1y ago
[deleted]
Bjartr 1y ago
Yeah, the whole community side to rationality is, at best, questionable. But the tools of thought that the literature describes are invaluable with one very important caveat. The moment you think something like "I am more correct than this other person because I am a rationalist" is the moment you fail as a rationalist. It is an incredibly easy mistake to make. To make effective use of the tools, you need to become more humble than before you were using them or you just turn into an asshole who can't be reasoned with. If you're saying "well actually, I'm right" more often than "oh wow, maybe I'm wrong", you've failed as a rationalist.
- the_af 1y ago
  > The moment you think something like "I am more correct than this other person because I am a rationalist" is the moment you fail as a rationalist. It's very telling that some of them went full "false modesty" by naming sites like "LessWrong", when you just know they actually mean "MoreRight". And in reality, it's just a bunch of "grown teenagers" posting their pet theories online and thinking themselves "big thinkers".
  - mariusor 1y ago
    > you just know they actually mean "MoreRight". I'm not affiliated with the rationalist community, but I always interpreted "Less Wrong" as word-play on how "being right" is an absolute binary: you can either be right, or not be right, while "being wrong" can cover a very large gradient. I expect the community wanted to emphasize how people employing the specific kind of Bayesian iterative reasoning they were proselytizing would arrive at slightly lesser degrees of wrong than the other kinds that "normal" people would use. If I'm right, your assertion wouldn't be totally inaccurate, but I think it might be missing the actual point.
    - the_af 1y ago
      > I'm not affiliated with the rationalist community, but I always interpreted "Less Wrong" as word-play on how "being right" is an absolute binary: you can either be right, nor not be right, while "being wrong" can cover a very large gradient. I know that's what they mean at the surface level, but you just know it comes with a high degree of smugness and false modesty. "I only know that I know nothing" -- maybe, but they ain't no modern day Socrates, they are just a bunch of nerds going online with their thoughts.
hiAndrewQuinn 1y ago
If you're in it just to figure out the core argument for why artificial intelligence is dangerous, please consider reading the first few chapters of Nick Bostom's Superintelligence instead. You'll get a lot more bang for your buck that way.
quickthrowman 1y ago
Your time would probably be better spent reading his magnum opus, Harry Potter and the Methods of Rationality. https://hpmor.com/ https://hpmor.com/
turtletontine 1y ago
For example, I recall being in lot of arguments that are purely "semantical" in nature. I believe this is what Wittgenstein called “language games”
- throwaway314155 1y ago
  In spirit of playing said game, I believe you can just use the word "pedantic" these days.
  - deleted 1y ago
    [deleted]
ramon156 1y ago
Sounds close to Yuval's book nexus which talks about the history of information gathering

by samuel 1y ago

greener_grass 1y ago

I think there is an arbitrage going on where STEM types who lack background in philosophy, literature, history are super impressed by basic ideas from those subjects being presented to them by stealth. Not saying this is you, but these topics have been discussed for thousands of years, so it should at least be surprising that Yudkowsky is breaking new ground.

samuel 1y ago
I don't claim that his work is original (the AI related probably is, but it's just tangentially related to rationalism), but it's clearly presented and is practical. And, BTW, I could just be ignorant in a lot of these topics, I take no offense in that. Still I think most people can learn something from an unprejudiced reading.
elt895 1y ago
Are there other philosophy- or history-grounded sources that are comparable? If so, I’d love some recommendations. Yudkowsky and others have their problems, but their texts have an interesting points, are relatively easy to read and understand, and you can clearly see which real issues they’re addressing. From my experience, alternatives tend to fall into two categories: 1. Genuine classical philosophy, which is usually incredibly hard to read and after 50 pages I have no idea what the author is even talking about anymore. 2. Basically self help books that take one or very few idea and repeat them ad nouseam for 200 pages.
- NoGravitas 1y ago
  I don't know if there's anything like a comprehensive high-level guide to philosophy that's any good, though of course there are college textbooks. If you want real/academic philosophy that's just more readable, I might suggest Eugene Thacker's "The Horror of Philosophy" series (starting with "In The Dust Of This Planet"), especially if you are a horror fan already.
- ashwinsundar 1y ago
  I don't have an answer here either, but after suffering through the first few chapters of HPMOR, I've found that Yudk and others tech-bros posing as philosophers are basically like leaky, dumbed-down abstractions for core philosophical ideas. Just go to the source and read about utilitarianism and deontology directly. Yudk is like the Wix of web development - sure you can build websites but you're not gonna be a proper web developer unless you learn HTML, CSS and Javascript. Worst of all, crappy abstractions train you in some actively bad patterns that are hard to unlearn It's almost offensive - are technologists so incapable of understanding philosophy that Yudk has to reduce it down to the least common denominator they are all familiar with - some fantasy world we read about as children?
  - AnimalMuppet 1y ago
    I'd like what the original sources would have written if someone had fed them some speak-clearly pills. Yudkowsky and company may have the dumbing-down problem, but the original sources often have a clarity problem. (That's why people are still arguing about what they meant centuries later. Not just whether they were right - though they argue about that too - but what they meant.) Even better, I'd like some filtering out of the parts that are clearly wrong.
    - chipsrafferty 1y ago
      Have you considered that the ideas that philosophy discusses are very complex, and words are not a sufficient medium to describe those ideas? That's why you have people arguing over what someone meant. Dumbing it down or trying to write something unambiguous doesn't actually make it better.
bnjms 1y ago
I think you’re mostly right. But also that it isn’t what the Yudkowsky is (was?) trying to do with it. I think he’s trying to distill useful tools which increase baseline rationality. Religions have this. It’s what the original philosophers are missing. (At least as taught, happy to hear counter examples)
- ashwinsundar 1y ago
  I think I'd rather subscribe to an actual religion, than listen to these weird rationalist types of people who seem to have solved the problem that is "everything". At least there is some interesting history to learn about with religion
  - bnjms 1y ago
    I would too if I could but organized religions make me uncomfortable even though I admire parts of them. Similar to my admiration you don’t need to like the rationality types or believe in their program to find one or more of their tools useful. I’ll also respond to the silent downvoters apparent disagreement. CFAR holds workshops and a summer camp for teaching rationality tools. In HPMoR Harry discusses the way he thinks and why. I read it as more of a way to discuss EY’s views in fiction as much as fiction itself.
sixo 1y ago
To the Stem-enlightened mind, the classical understanding and pedagogy of such ideas is underwhelming, vague, and riddled with language-game problems, compared to the precision a mathematically-rooted idea has. They're rederiving all this stuff not out of obstinacy, but because they prefer it. I don't really identify with rationalism per se, but I'm with them on this--the humanities are over-cooked and a humanity education tends to be a tedious slog through outmoded ideas divorced from reality
- throwawaymaroon 1y ago
  [dead]
- biofox 1y ago
  If you contextualise the outmoded ideas as part of the Great Conversation [1], and the story of how we reached our current understanding, rather than objective statements of fact, then they becomes a lot more valuable and worthy of study. [1] https://en.wikipedia.org/wiki/Great_Conversation https://en.wikipedia.org/wiki/Great_Conversation
  - jstanley 1y ago
    But isn't the content of LessWrong part of the Great Conversation too?
- jay_kyburz 1y ago
  I have kids in high school. We sometimes talk about the difference between the black and white of math or science, and the wishy washy grey of the humanities. You can be right or wrong in math. You have can an opinion in English.
  - HDThoreaun 1y ago
    You can definitely have wrong opinions in the humanities.
  - bmacho 1y ago
    You can be right or wrong in math and philosophy. You have can an opinion in any other sciences, physics, chemistry, biology, medical sciences, history, you name it.
HDThoreaun 1y ago
Rationalism largely rejects continental philosophy in favor of a more analytic approach. Yes these ideas are not new, but they’re not really the mainstream stuff you’d see in philosophy, literature, or history studies. You’d have to seek out these classes specifically to find them.
- TimorousBestie 1y ago
  They largely reject analytic philosophy as well. Austin and Whitehead are roughly as detestable to a Rationalist as Foucault and Marx. Carlyle, Chesterton and Thoreau are about the limit of their philosophical knowledge base.
- TheAncientGeek 1y ago
  Analytical philosophy is rationalism done right.
FeepingCreature 1y ago
In AI finetuning, there's a theory that the model already contains the right ideas and skills, and the finetuning just raises them to prominence. Similarly in philosophic pedagogy, there's huge value in taking ideas that are correct but unintuitive and maybe have 30% buy-in and saying "actually, this is obviously correct, also here's an analysis of why you wouldn't believe it anyway and how you have to think to become able to believe it". That's most of what the Sequences are: they take from every field of philosophy the ideas that are actually correct, and say "okay actually, we don't need to debate this anymore, this just seems to be the truth because so-and-so." (Though the comments section vociferously disagrees.) And it turns out if you do this, you can discard 90% of philosophy as historical detritus. You're still taking ideas from philosophy, but which ideas matters, and how you present them matters. The massive advantage of the Sequences is they have justified and well-defended confidence where appropriate. And if you manage to pick the right answers again and again, you get a system that actually hangs together, and IMO it's to philosophy's detriment that it doesn't do this itself much more aggressively. For instance, 60% of philosophers are compatibilists. Compatibilism is really obviously correct. "What are you complaining about, that's a majority, isn't that good?" What is wrong with those 40% though? If you're in those 40%, what arguments may convince you? Repeat to taste.
- margalabargala 1y ago
  > And it turns out if you do this, you can discard 90% of philosophy as historical detritus This is just the story of the history of philosophy. Going back hundreds of years. See Kant and Hegel for notable examples.
  - FeepingCreature 1y ago
    Sure, and I agree that LW is doing philosophy in that sense.
    - margalabargala 1y ago
      Sure, I just object to the characterisation of "actually correct", as though each of those ideas has not gone back and forth on philosophers thinking that particular idea is "actually correct" for centuries. LW does not appear to have much if any novel insight; just much better marketing.

deleted 1y ago

[deleted]

Bjartr 1y ago

Yeah, the whole community side to rationality is, at best, questionable. But the tools of thought that the literature describes are invaluable with one very important caveat. The moment you think something like "I am more correct than this other person because I am a rationalist" is the moment you fail as a rationalist. It is an incredibly easy mistake to make. To make effective use of the tools, you need to become more humble than before you were using them or you just turn into an asshole who can't be reasoned with. If you're saying "well actually, I'm right" more often than "oh wow, maybe I'm wrong", you've failed as a rationalist.

the_af 1y ago
> The moment you think something like "I am more correct than this other person because I am a rationalist" is the moment you fail as a rationalist. It's very telling that some of them went full "false modesty" by naming sites like "LessWrong", when you just know they actually mean "MoreRight". And in reality, it's just a bunch of "grown teenagers" posting their pet theories online and thinking themselves "big thinkers".
- mariusor 1y ago
  > you just know they actually mean "MoreRight". I'm not affiliated with the rationalist community, but I always interpreted "Less Wrong" as word-play on how "being right" is an absolute binary: you can either be right, or not be right, while "being wrong" can cover a very large gradient. I expect the community wanted to emphasize how people employing the specific kind of Bayesian iterative reasoning they were proselytizing would arrive at slightly lesser degrees of wrong than the other kinds that "normal" people would use. If I'm right, your assertion wouldn't be totally inaccurate, but I think it might be missing the actual point.
  - the_af 1y ago
    > I'm not affiliated with the rationalist community, but I always interpreted "Less Wrong" as word-play on how "being right" is an absolute binary: you can either be right, nor not be right, while "being wrong" can cover a very large gradient. I know that's what they mean at the surface level, but you just know it comes with a high degree of smugness and false modesty. "I only know that I know nothing" -- maybe, but they ain't no modern day Socrates, they are just a bunch of nerds going online with their thoughts.

hiAndrewQuinn 1y ago

If you're in it just to figure out the core argument for why artificial intelligence is dangerous, please consider reading the first few chapters of Nick Bostom's Superintelligence instead. You'll get a lot more bang for your buck that way.

quickthrowman 1y ago

Your time would probably be better spent reading his magnum opus, Harry Potter and the Methods of Rationality. https://hpmor.com/ https://hpmor.com/

turtletontine 1y ago

For example, I recall being in lot of arguments that are purely "semantical" in nature. I believe this is what Wittgenstein called “language games”

throwaway314155 1y ago
In spirit of playing said game, I believe you can just use the word "pedantic" these days.
- deleted 1y ago
  [deleted]

ramon156 1y ago

Sounds close to Yuval's book nexus which talks about the history of information gathering