4 ms·

So, to be perfectly clear, the main reason for studying rationality as you explain it, is the belief that many individuals are not skilled at determining whethe

So, to be perfectly clear, the main reason for studying rationality as you explain it, is the belief that many individuals are not skilled at determining whether their beliefs about the observable world are right or wrong (as in your umbrella example), and that this is something they would benefit from working on?

FeepingCreature 5y ago
Well, most people can generally recognize when a belief is wrong if they are directly confronted with the fact. I think it's a bit more meta - such as creating modes of thought that systematically (again, holding in mind the assumption that nature is regular) produce right and useful thoughts, and dismantling modes of thought that systematically produce wrong or useless thoughts. But in essence, yes. (Though of course, again, the ultimate goal is to be better able to achieve preferred outcomes in the world, and if thinking only the word "blue" on repeat produced outcomes perfectly in line with your preferences, then rationality would espouse blue blue blue blue. But at that point, most of the techniques gathered under the label would be useless, because yadda yadda ordered world etc.)
- Nav_Panel 5y ago
  Thanks for sticking with me as I clarified, I see the crux of our disagreement now. Basically, I am personally of the belief that any "wrong" or "useless" beliefs that one holds are always actually useful, as all belief is instrumental. Ignorance is maintained because it serves a purpose. I am also of the belief that the only way to determine why one holds a wrong belief is to understand the "emotional" or desire-based reason that underlies or supports it. Hence why I spent a lot of time above discussing emotions. We have seen in recent times the repeated failure of individuals to adjust their beliefs even in the face of overwhelming evidence. Why is this? My stance is that their "wrong" beliefs serve deep emotional needs for them. Most people have stronger desires than the desire to arrive at truth in their evaluation of the world. The difference in emphasis here is shifting from "the brain as a prediction-making device" to "the human body including the brain is a system that acts to relieve tensions placed upon it", and that prediction-making is one biological strategy for doing so. Of course, better predictions can and will relieve more tension than worse predictions, but my stance is that being able to conceptualize the causes and nature of tension directly will lead to better outcomes, as it permits stronger meta-cognition. I see the term "preference" doing a lot of work in rationalist thought, and I generally taboo it. The term comes from economics and fails IMO to adequately describe the nature of desire. But if I were to use the term in this instance, I would claim that disentangling one's preferences from internal conflict is the first step one should take prior to tackling the problem of rationality (because we need to know what we even want to get done before we figure out how to do it). This involves interrogating our beliefs about what our preferences are and why, which is something that the rationalist focus on the "world" seems uniquely unequipped for dealing with. I hope this all makes sense!
  - FeepingCreature 5y ago
    Yeah that makes sense. I really think where this is running apart is that... like, a lot of the people driving the rationalist project are in environments where holding strategically correct beliefs isn't just useful but vital. Eliezer famously believes on a gut level that he once held a wrong belief and that if he had acted on that wrong belief, the world would have ended. I don't know how that relieves a tension, but I can see how it would make you really really wary of cognitive mistakes. (I'm having a lot of fun with this debate too, it's really forced me to firm up my understanding of faith and reason.) > But if I were to use the term in this instance, I would claim that disentangling one's preferences from internal conflict is the first step one should take prior to tackling the problem of rationality (because we need to know what we even want to get done before we figure out how to do it). This involves interrogating our beliefs about what our preferences are and why, which is something that the rationalist focus on the "world" seems uniquely unequipped for dealing with. I agree with this take! Though I think it's less that this is something rationality cannot do as that, well, Eliezer when he was writing the Sequences, had an overriding belief that put all his other preferences at risk. This tends to focus one's attention on strategy. :) The most illuminating post on this, for me, is https://www.lesswrong.com/posts/SGR4GxFK7KmW7ckCB/something-to-protect https://www.lesswrong.com/posts/SGR4GxFK7KmW7ckCB/something-... . You see this notion of "rationality is a tool to come to conclusions when the conclusions really really matter" all over his writing. I especially like the phrasing in https://www.lesswrong.com/posts/wustx45CPL5rZenuo/no-safe-defense-not-even-science https://www.lesswrong.com/posts/wustx45CPL5rZenuo/no-safe-de... : > No one begins to truly search for the Way until their parents have failed them, their gods are dead, and their tools have shattered in their hand. Which implies that to most people, the pursuit of individual rationality, possibly even above mental wellbeing, is counteradaptive. (Man, does Eliezer have a way with words though.) (To me, personally, the idea of holding wrong beliefs is like a persistent itch or annoyance. I do not have a driving need, but I also have few attachments. And, of course, I am a nerd, and this is all nerd cocaine. All that sums up to create an ideological space in which I am comfortable.)
    - Nav_Panel 5y ago
      Thanks for sharing those two posts. These might be instrumental in helping a (more rationalist-leaning) friend and I move forward in a years-long debate about ethics, lmfao This part particularly resonated with me: > Your trust will not break, until you apply all that you have learned here and from other books, and take it as far as you can go, and find that this too fails you—that you have still been a fool, and no one warned you against it—that all the most important parts were left out of the guidance you received—that some of the most precious ideals you followed, steered you in the wrong direction— > —and if you still have something to protect, so that you must keep going, and cannot resign and wisely acknowledge the limitations of rationality— > —then you will be ready to start your journey as a rationalist. My trust broke years ago, and I found the rationalists, and then I realized, unlike Yudkowsky (I guess?), that I didn't have something to protect. So, my personal goals are in that sense very different. In my experience with the rationalist community as well, I found that most people I met didn't have a specific personal motivation beside their beliefs surrounding x-risk, and I personally find the set of axioms required to accept rationalist beliefs on x-risk a bit... difficult to swallow.

FeepingCreature 5y ago

Well, most people can generally recognize when a belief is wrong if they are directly confronted with the fact. I think it's a bit more meta - such as creating modes of thought that systematically (again, holding in mind the assumption that nature is regular) produce right and useful thoughts, and dismantling modes of thought that systematically produce wrong or useless thoughts. But in essence, yes. (Though of course, again, the ultimate goal is to be better able to achieve preferred outcomes in the world, and if thinking only the word "blue" on repeat produced outcomes perfectly in line with your preferences, then rationality would espouse blue blue blue blue. But at that point, most of the techniques gathered under the label would be useless, because yadda yadda ordered world etc.)

Nav_Panel 5y ago
Thanks for sticking with me as I clarified, I see the crux of our disagreement now. Basically, I am personally of the belief that any "wrong" or "useless" beliefs that one holds are always actually useful, as all belief is instrumental. Ignorance is maintained because it serves a purpose. I am also of the belief that the only way to determine why one holds a wrong belief is to understand the "emotional" or desire-based reason that underlies or supports it. Hence why I spent a lot of time above discussing emotions. We have seen in recent times the repeated failure of individuals to adjust their beliefs even in the face of overwhelming evidence. Why is this? My stance is that their "wrong" beliefs serve deep emotional needs for them. Most people have stronger desires than the desire to arrive at truth in their evaluation of the world. The difference in emphasis here is shifting from "the brain as a prediction-making device" to "the human body including the brain is a system that acts to relieve tensions placed upon it", and that prediction-making is one biological strategy for doing so. Of course, better predictions can and will relieve more tension than worse predictions, but my stance is that being able to conceptualize the causes and nature of tension directly will lead to better outcomes, as it permits stronger meta-cognition. I see the term "preference" doing a lot of work in rationalist thought, and I generally taboo it. The term comes from economics and fails IMO to adequately describe the nature of desire. But if I were to use the term in this instance, I would claim that disentangling one's preferences from internal conflict is the first step one should take prior to tackling the problem of rationality (because we need to know what we even want to get done before we figure out how to do it). This involves interrogating our beliefs about what our preferences are and why, which is something that the rationalist focus on the "world" seems uniquely unequipped for dealing with. I hope this all makes sense!
- FeepingCreature 5y ago
  Yeah that makes sense. I really think where this is running apart is that... like, a lot of the people driving the rationalist project are in environments where holding strategically correct beliefs isn't just useful but vital. Eliezer famously believes on a gut level that he once held a wrong belief and that if he had acted on that wrong belief, the world would have ended. I don't know how that relieves a tension, but I can see how it would make you really really wary of cognitive mistakes. (I'm having a lot of fun with this debate too, it's really forced me to firm up my understanding of faith and reason.) > But if I were to use the term in this instance, I would claim that disentangling one's preferences from internal conflict is the first step one should take prior to tackling the problem of rationality (because we need to know what we even want to get done before we figure out how to do it). This involves interrogating our beliefs about what our preferences are and why, which is something that the rationalist focus on the "world" seems uniquely unequipped for dealing with. I agree with this take! Though I think it's less that this is something rationality cannot do as that, well, Eliezer when he was writing the Sequences, had an overriding belief that put all his other preferences at risk. This tends to focus one's attention on strategy. :) The most illuminating post on this, for me, is https://www.lesswrong.com/posts/SGR4GxFK7KmW7ckCB/something-to-protect https://www.lesswrong.com/posts/SGR4GxFK7KmW7ckCB/something-... . You see this notion of "rationality is a tool to come to conclusions when the conclusions really really matter" all over his writing. I especially like the phrasing in https://www.lesswrong.com/posts/wustx45CPL5rZenuo/no-safe-defense-not-even-science https://www.lesswrong.com/posts/wustx45CPL5rZenuo/no-safe-de... : > No one begins to truly search for the Way until their parents have failed them, their gods are dead, and their tools have shattered in their hand. Which implies that to most people, the pursuit of individual rationality, possibly even above mental wellbeing, is counteradaptive. (Man, does Eliezer have a way with words though.) (To me, personally, the idea of holding wrong beliefs is like a persistent itch or annoyance. I do not have a driving need, but I also have few attachments. And, of course, I am a nerd, and this is all nerd cocaine. All that sums up to create an ideological space in which I am comfortable.)
  - Nav_Panel 5y ago
    Thanks for sharing those two posts. These might be instrumental in helping a (more rationalist-leaning) friend and I move forward in a years-long debate about ethics, lmfao This part particularly resonated with me: > Your trust will not break, until you apply all that you have learned here and from other books, and take it as far as you can go, and find that this too fails you—that you have still been a fool, and no one warned you against it—that all the most important parts were left out of the guidance you received—that some of the most precious ideals you followed, steered you in the wrong direction— > —and if you still have something to protect, so that you must keep going, and cannot resign and wisely acknowledge the limitations of rationality— > —then you will be ready to start your journey as a rationalist. My trust broke years ago, and I found the rationalists, and then I realized, unlike Yudkowsky (I guess?), that I didn't have something to protect. So, my personal goals are in that sense very different. In my experience with the rationalist community as well, I found that most people I met didn't have a specific personal motivation beside their beliefs surrounding x-risk, and I personally find the set of axioms required to accept rationalist beliefs on x-risk a bit... difficult to swallow.