4 ms·

Thanks for sticking with me as I clarified, I see the crux of our disagreement now. Basically, I am personally of the belief that any "wrong" or "useless" beli

Thanks for sticking with me as I clarified, I see the crux of our disagreement now. Basically, I am personally of the belief that any "wrong" or "useless" beliefs that one holds are always actually useful, as all belief is instrumental. Ignorance is maintained because it serves a purpose. I am also of the belief that the only way to determine why one holds a wrong belief is to understand the "emotional" or desire-based reason that underlies or supports it. Hence why I spent a lot of time above discussing emotions. We have seen in recent times the repeated failure of individuals to adjust their beliefs even in the face of overwhelming evidence. Why is this? My stance is that their "wrong" beliefs serve deep emotional needs for them. Most people have stronger desires than the desire to arrive at truth in their evaluation of the world. The difference in emphasis here is shifting from "the brain as a prediction-making device" to "the human body including the brain is a system that acts to relieve tensions placed upon it", and that prediction-making is one biological strategy for doing so. Of course, better predictions can and will relieve more tension than worse predictions, but my stance is that being able to conceptualize the causes and nature of tension directly will lead to better outcomes, as it permits stronger meta-cognition. I see the term "preference" doing a lot of work in rationalist thought, and I generally taboo it. The term comes from economics and fails IMO to adequately describe the nature of desire. But if I were to use the term in this instance, I would claim that disentangling one's preferences from internal conflict is the first step one should take prior to tackling the problem of rationality (because we need to know what we even want to get done before we figure out how to do it). This involves interrogating our beliefs about what our preferences are and why, which is something that the rationalist focus on the "world" seems uniquely unequipped for dealing with. I hope this all makes sense!

FeepingCreature 5y ago
Yeah that makes sense. I really think where this is running apart is that... like, a lot of the people driving the rationalist project are in environments where holding strategically correct beliefs isn't just useful but vital. Eliezer famously believes on a gut level that he once held a wrong belief and that if he had acted on that wrong belief, the world would have ended. I don't know how that relieves a tension, but I can see how it would make you really really wary of cognitive mistakes. (I'm having a lot of fun with this debate too, it's really forced me to firm up my understanding of faith and reason.) > But if I were to use the term in this instance, I would claim that disentangling one's preferences from internal conflict is the first step one should take prior to tackling the problem of rationality (because we need to know what we even want to get done before we figure out how to do it). This involves interrogating our beliefs about what our preferences are and why, which is something that the rationalist focus on the "world" seems uniquely unequipped for dealing with. I agree with this take! Though I think it's less that this is something rationality cannot do as that, well, Eliezer when he was writing the Sequences, had an overriding belief that put all his other preferences at risk. This tends to focus one's attention on strategy. :) The most illuminating post on this, for me, is https://www.lesswrong.com/posts/SGR4GxFK7KmW7ckCB/something-to-protect https://www.lesswrong.com/posts/SGR4GxFK7KmW7ckCB/something-... . You see this notion of "rationality is a tool to come to conclusions when the conclusions really really matter" all over his writing. I especially like the phrasing in https://www.lesswrong.com/posts/wustx45CPL5rZenuo/no-safe-defense-not-even-science https://www.lesswrong.com/posts/wustx45CPL5rZenuo/no-safe-de... : > No one begins to truly search for the Way until their parents have failed them, their gods are dead, and their tools have shattered in their hand. Which implies that to most people, the pursuit of individual rationality, possibly even above mental wellbeing, is counteradaptive. (Man, does Eliezer have a way with words though.) (To me, personally, the idea of holding wrong beliefs is like a persistent itch or annoyance. I do not have a driving need, but I also have few attachments. And, of course, I am a nerd, and this is all nerd cocaine. All that sums up to create an ideological space in which I am comfortable.)
- Nav_Panel 5y ago
  Thanks for sharing those two posts. These might be instrumental in helping a (more rationalist-leaning) friend and I move forward in a years-long debate about ethics, lmfao This part particularly resonated with me: > Your trust will not break, until you apply all that you have learned here and from other books, and take it as far as you can go, and find that this too fails you—that you have still been a fool, and no one warned you against it—that all the most important parts were left out of the guidance you received—that some of the most precious ideals you followed, steered you in the wrong direction— > —and if you still have something to protect, so that you must keep going, and cannot resign and wisely acknowledge the limitations of rationality— > —then you will be ready to start your journey as a rationalist. My trust broke years ago, and I found the rationalists, and then I realized, unlike Yudkowsky (I guess?), that I didn't have something to protect. So, my personal goals are in that sense very different. In my experience with the rationalist community as well, I found that most people I met didn't have a specific personal motivation beside their beliefs surrounding x-risk, and I personally find the set of axioms required to accept rationalist beliefs on x-risk a bit... difficult to swallow.
  - FeepingCreature 5y ago
    Yeah that all makes sense. I called myself a "rationality consumer" in a LW post a few years back, and I think that roughly still holds. Though I will say that being part of a community where wrong beliefs are corrigible through argumentative persuasion is an immense social relief for me. (Also, AI X-Risk makes a lot more sense if you're already habituated to thinking of the mind in terms of algorithm. 40% programmers: not a coincidence.)
    - Nav_Panel 5y ago
      This also makes sense. I did enjoy all the arguments I got into on the SSC discord. It was formative for me even though I found myself often frustrated when the arguments reached an impasse, often in relation to the desires of the interlocutors. I'm also a programmer and am very habituated in terms of thinking in algorithms, but when I had my "break", I realized that my beliefs were not working on a very deep level, and I needed something a lot stronger/deeper than rationality to push me out of whatever local extremum I was trapped in.

FeepingCreature 5y ago

Yeah that makes sense. I really think where this is running apart is that... like, a lot of the people driving the rationalist project are in environments where holding strategically correct beliefs isn't just useful but vital. Eliezer famously believes on a gut level that he once held a wrong belief and that if he had acted on that wrong belief, the world would have ended. I don't know how that relieves a tension, but I can see how it would make you really really wary of cognitive mistakes. (I'm having a lot of fun with this debate too, it's really forced me to firm up my understanding of faith and reason.) > But if I were to use the term in this instance, I would claim that disentangling one's preferences from internal conflict is the first step one should take prior to tackling the problem of rationality (because we need to know what we even want to get done before we figure out how to do it). This involves interrogating our beliefs about what our preferences are and why, which is something that the rationalist focus on the "world" seems uniquely unequipped for dealing with. I agree with this take! Though I think it's less that this is something rationality cannot do as that, well, Eliezer when he was writing the Sequences, had an overriding belief that put all his other preferences at risk. This tends to focus one's attention on strategy. :) The most illuminating post on this, for me, is https://www.lesswrong.com/posts/SGR4GxFK7KmW7ckCB/something-to-protect https://www.lesswrong.com/posts/SGR4GxFK7KmW7ckCB/something-... . You see this notion of "rationality is a tool to come to conclusions when the conclusions really really matter" all over his writing. I especially like the phrasing in https://www.lesswrong.com/posts/wustx45CPL5rZenuo/no-safe-defense-not-even-science https://www.lesswrong.com/posts/wustx45CPL5rZenuo/no-safe-de... : > No one begins to truly search for the Way until their parents have failed them, their gods are dead, and their tools have shattered in their hand. Which implies that to most people, the pursuit of individual rationality, possibly even above mental wellbeing, is counteradaptive. (Man, does Eliezer have a way with words though.) (To me, personally, the idea of holding wrong beliefs is like a persistent itch or annoyance. I do not have a driving need, but I also have few attachments. And, of course, I am a nerd, and this is all nerd cocaine. All that sums up to create an ideological space in which I am comfortable.)

Nav_Panel 5y ago
Thanks for sharing those two posts. These might be instrumental in helping a (more rationalist-leaning) friend and I move forward in a years-long debate about ethics, lmfao This part particularly resonated with me: > Your trust will not break, until you apply all that you have learned here and from other books, and take it as far as you can go, and find that this too fails you—that you have still been a fool, and no one warned you against it—that all the most important parts were left out of the guidance you received—that some of the most precious ideals you followed, steered you in the wrong direction— > —and if you still have something to protect, so that you must keep going, and cannot resign and wisely acknowledge the limitations of rationality— > —then you will be ready to start your journey as a rationalist. My trust broke years ago, and I found the rationalists, and then I realized, unlike Yudkowsky (I guess?), that I didn't have something to protect. So, my personal goals are in that sense very different. In my experience with the rationalist community as well, I found that most people I met didn't have a specific personal motivation beside their beliefs surrounding x-risk, and I personally find the set of axioms required to accept rationalist beliefs on x-risk a bit... difficult to swallow.
- FeepingCreature 5y ago
  Yeah that all makes sense. I called myself a "rationality consumer" in a LW post a few years back, and I think that roughly still holds. Though I will say that being part of a community where wrong beliefs are corrigible through argumentative persuasion is an immense social relief for me. (Also, AI X-Risk makes a lot more sense if you're already habituated to thinking of the mind in terms of algorithm. 40% programmers: not a coincidence.)
  - Nav_Panel 5y ago
    This also makes sense. I did enjoy all the arguments I got into on the SSC discord. It was formative for me even though I found myself often frustrated when the arguments reached an impasse, often in relation to the desires of the interlocutors. I'm also a programmer and am very habituated in terms of thinking in algorithms, but when I had my "break", I realized that my beliefs were not working on a very deep level, and I needed something a lot stronger/deeper than rationality to push me out of whatever local extremum I was trapped in.