5 ms·

Reevaluating how we reason

55 points by cwan 16y ago

swombat 16y ago
The final sentence is key: Unless we take our innate biases into account, the blessing of human reason can easily become a curse. So take your innate biases into account. Familiarise yourself with them. Read books like "Predictably Irrational". Follow Eliezer Yudkowsky most excellent articles on http://www.overcomingbias.com/ http://www.overcomingbias.com/ (and read his awesome "Harry Potter and the Methods of Rationality" at http://www.fanfiction.net/s/5782108/1/Harry_Potter_and_the_Methods_of_Rationality http://www.fanfiction.net/s/5782108/1/Harry_Potter_and_the_M... ). Be aware of those biases. Your subconscious has biases too. Don't take its answers for granted - test them out. Practice figuring out what your subconscious is biased about, and correct for that. Being a rational thinker is not about explaining/rationalising your conclusions, it's about using subconscious and conscious reasoning in tandem, being aware of the shortcomings of both, and building up to valid conclusions based on whatever evidence you can get your hands on (and acquiring more evidence where useful).
- radu_floricica 16y ago
  Speaking of Harry Potter's fic, Robert Cialdini's book was much better then I imagined.
greenlblue 16y ago
This is why all my major decisions are based on gut reactions and intuition and I'm not being sarcastic. As soon as something 'feels' wrong I know it's time to make a change. This strategy has been working out well for me so far and I'm happy to see there is research to back it up.
- dan00 16y ago
  I don't think, that your conclusion fits with the article. Because there's not a big disconnection between feeling and thinking. If you feel in a direction, then you will certainly also think in this direction. Because your feelings are colouring your thoughts. It's like you already feel the conclusion and are only searching for rational arguments. I think that the article doesn't go far enough, because you shouldn't only question your thinking, but also your feelings.
  - greenlblue 16y ago
    I think the point is to not think too much. Feelings certainly color thoughts but the article makes the point that if you do too much thinking then the color provided by your feelings is washed out and you are left with rationalization which can often work against your interests.
    - dan00 16y ago
      But that's not the point of the article. It's about your bias, and to see it, it's not enough to only look at your thoughts.
      - greenlblue 16y ago
        What I got from the article was that too much thinking makes people deviate from expert opinion. This is a bad thing because experts in general tend to be more objective in their domain of expertise and are better able to judge things based on merit instead of some other metric. I think it's less about bias and more about false rationalization being the default state of thinking when it comes to non-experts and one way to not fall into this trap is to not overthink. You could also avoid this trap by becoming an expert but that requires time and hard work and trusting your initial gut reactions and intuitions can often lead to better results when you don't have the time to become an expert in some domain.
        yters 16y ago
        That was the point of the study, the point of the article is that reasoning is just rationalization of what we already feel. However, this actually conflicts with the study, which shows reasoning leads us away from what we already feel. Obviously, the author spent too much time reasoning about the study.
- tarkin2 16y ago
  Gut feelings are also subject to bias.
  - yters 16y ago
    Is there anything not biased?
    - jerf 16y ago
      No. There is no such thing as unbiased. I don't just mean something like "to a first approximation there is no such thing as unbiased"; I mean that there is indeed no such thing. It is an English word with no existing actual referent, like the phrase "uniform distribution over the real number line". No such thing. One of the fun things about learning about machine learning is that you get a great and powerful definition of bias, and as a consequence it can clearly explain that what unbiased translates to is "brings no information about the possible set of experiences to the table", but the consequence of that is that it takes an infinite amount of information to come to any conclusions whatsoever, even probabilistic ones. No matter how finitely many bits you apply to your original N-dimensional infinite state space, you still have an infinite state space remaining. And in my experience, the ML definition of "bias" isn't just a cute mathematical construct, it actually has a direct (and somewhat profound) relationship to the word as used in English, so while I used math terminology in the previous paragraph this is not a case where theory and practice diverge. They are actually quite convergent here, and there really is no such thing as "unbiased".
      - Gormo 16y ago
        And if it was possible to be unbiased, what would be the point? We use our subjective reasoning to improve our subjective lives; our understanding and knowledge originates in and applies to our relationships with the world. If we somehow manage remove our selves from the equation, and form a purely objective understanding of things in their own right, what have we actually gained? What's the value of thoughts that exclude the ones thinking them from their application? Ideas are incredibly valuable tools, applied in their context and in pursuit of a valued goal. But I don't understand the value of regarding ideas as "truth" in some abstract and universal sense.
DLWormwood 16y ago
I found the main point of this article to be profoundly depressing... Our rational faculty isn’t a scientist – it’s a talk radio host ...what a good way to sap my enthusiasm and hope for the day.
- greenlblue 16y ago
  Objective thinking in any domain is hard. I don't see why pointing out that people don't get enough practice in actual rational thinking when it comes to their preferences is depressing?
tarkin2 16y ago
Although not emboldened, this sentence in the abstract best explained the jam tasting problem: "Proactively used reasoning also favors decisions that are easy to justify but not necessarily better." That is, the students focused on the quantity of reasons, not necessarily the most important reasons. It reminds me of the Bayesian stuff I've been reading about on lesswrong.com. And in that way, it best describes 'rationalisation': an overemphasis of less important aspects of a problem domain, often because of bias or because of difficulty in discovering the more important aspects, as in the jam example.
techiferous 16y ago
"...that human reason has nothing to do with finding the truth, or locating the best alternative. Instead, it’s all about argumentation." I've come to the conclusion that our brain is not trying to produce an accurate model of reality. It is trying to choose good experiences over bad experiences. If it ends up constructing an accurate model of some part of reality, that's a side effect.
Tycho 16y ago
Reminds me of how some people review music. They start to focus on some things which were minor decisions by the recording artists, requiring no special talent (the reviewers themselves think they could have done better), like say the sequencing of songs on a CD, and then attach the same importance they would afford the quality of songs themselves. It's sort of like they devise a number of criteria and then naively give each category equal weighting. Videogames reviews are also bad for this.
- chc 16y ago
  Videogame reviews are bad for a multitude of reasons.
msluyter 16y ago
Why is it so hard to eradicate confirmation bias? Well, for one, even if you're acutely aware of it, you'll just tend to believe that your beliefs are that much likely to be correct. "After all, I've taken confirmation bias into account!"
mkr-hn 16y ago
What bugs me about this story and the conclusions is that it tries to compare taste--something wholly subjective--to other things. It implies that people use the same criteria of judgment for everything, which isn't even addressed by the study. How we judge our food doesn't necessarily reflect how we judge more important things like our political and social views. Has a similar study been done for those subjects? This is the worst kind of science reporting because it's trying to take a single study and draw wildly unrelated conclusions.
isleyaardvark 16y ago
This explains the notorious confirmation bias. This bias is apparent not only when people are actually arguing but also when they are reasoning proactively with the perspective of having to defend their opinions. And overcoming bias could also be motivated by wanting to persuade. It's covering your bases in case the person you're trying to persuade is aware of confirmation bias. I'd hate for this paper to become support for the idea that you should just go by your gut and not by reason. It looked like this study only looked at highly subjective topics, which jam is better, which poster/car/jelly bean is better. Those are largely subject to taste, and are much different subjects than something like economics or science.

55 points by cwan 16y ago

The final sentence is key: Unless we take our innate biases into account, the blessing of human reason can easily become a curse. So take your innate biases into account. Familiarise yourself with them. Read books like "Predictably Irrational". Follow Eliezer Yudkowsky most excellent articles on http://www.overcomingbias.com/ http://www.overcomingbias.com/ (and read his awesome "Harry Potter and the Methods of Rationality" at http://www.fanfiction.net/s/5782108/1/Harry_Potter_and_the_Methods_of_Rationality http://www.fanfiction.net/s/5782108/1/Harry_Potter_and_the_M... ). Be aware of those biases. Your subconscious has biases too. Don't take its answers for granted - test them out. Practice figuring out what your subconscious is biased about, and correct for that. Being a rational thinker is not about explaining/rationalising your conclusions, it's about using subconscious and conscious reasoning in tandem, being aware of the shortcomings of both, and building up to valid conclusions based on whatever evidence you can get your hands on (and acquiring more evidence where useful).

radu_floricica 16y ago
Speaking of Harry Potter's fic, Robert Cialdini's book was much better then I imagined.

greenlblue 16y ago

This is why all my major decisions are based on gut reactions and intuition and I'm not being sarcastic. As soon as something 'feels' wrong I know it's time to make a change. This strategy has been working out well for me so far and I'm happy to see there is research to back it up.

dan00 16y ago
I don't think, that your conclusion fits with the article. Because there's not a big disconnection between feeling and thinking. If you feel in a direction, then you will certainly also think in this direction. Because your feelings are colouring your thoughts. It's like you already feel the conclusion and are only searching for rational arguments. I think that the article doesn't go far enough, because you shouldn't only question your thinking, but also your feelings.
- greenlblue 16y ago
  I think the point is to not think too much. Feelings certainly color thoughts but the article makes the point that if you do too much thinking then the color provided by your feelings is washed out and you are left with rationalization which can often work against your interests.
  - dan00 16y ago
    But that's not the point of the article. It's about your bias, and to see it, it's not enough to only look at your thoughts.
    - greenlblue 16y ago
      What I got from the article was that too much thinking makes people deviate from expert opinion. This is a bad thing because experts in general tend to be more objective in their domain of expertise and are better able to judge things based on merit instead of some other metric. I think it's less about bias and more about false rationalization being the default state of thinking when it comes to non-experts and one way to not fall into this trap is to not overthink. You could also avoid this trap by becoming an expert but that requires time and hard work and trusting your initial gut reactions and intuitions can often lead to better results when you don't have the time to become an expert in some domain.
      - yters 16y ago
        That was the point of the study, the point of the article is that reasoning is just rationalization of what we already feel. However, this actually conflicts with the study, which shows reasoning leads us away from what we already feel. Obviously, the author spent too much time reasoning about the study.
tarkin2 16y ago
Gut feelings are also subject to bias.
- yters 16y ago
  Is there anything not biased?
  - jerf 16y ago
    No. There is no such thing as unbiased. I don't just mean something like "to a first approximation there is no such thing as unbiased"; I mean that there is indeed no such thing. It is an English word with no existing actual referent, like the phrase "uniform distribution over the real number line". No such thing. One of the fun things about learning about machine learning is that you get a great and powerful definition of bias, and as a consequence it can clearly explain that what unbiased translates to is "brings no information about the possible set of experiences to the table", but the consequence of that is that it takes an infinite amount of information to come to any conclusions whatsoever, even probabilistic ones. No matter how finitely many bits you apply to your original N-dimensional infinite state space, you still have an infinite state space remaining. And in my experience, the ML definition of "bias" isn't just a cute mathematical construct, it actually has a direct (and somewhat profound) relationship to the word as used in English, so while I used math terminology in the previous paragraph this is not a case where theory and practice diverge. They are actually quite convergent here, and there really is no such thing as "unbiased".
    - Gormo 16y ago
      And if it was possible to be unbiased, what would be the point? We use our subjective reasoning to improve our subjective lives; our understanding and knowledge originates in and applies to our relationships with the world. If we somehow manage remove our selves from the equation, and form a purely objective understanding of things in their own right, what have we actually gained? What's the value of thoughts that exclude the ones thinking them from their application? Ideas are incredibly valuable tools, applied in their context and in pursuit of a valued goal. But I don't understand the value of regarding ideas as "truth" in some abstract and universal sense.

DLWormwood 16y ago

I found the main point of this article to be profoundly depressing... Our rational faculty isn’t a scientist – it’s a talk radio host ...what a good way to sap my enthusiasm and hope for the day.

greenlblue 16y ago
Objective thinking in any domain is hard. I don't see why pointing out that people don't get enough practice in actual rational thinking when it comes to their preferences is depressing?

tarkin2 16y ago

Although not emboldened, this sentence in the abstract best explained the jam tasting problem: "Proactively used reasoning also favors decisions that are easy to justify but not necessarily better." That is, the students focused on the quantity of reasons, not necessarily the most important reasons. It reminds me of the Bayesian stuff I've been reading about on lesswrong.com. And in that way, it best describes 'rationalisation': an overemphasis of less important aspects of a problem domain, often because of bias or because of difficulty in discovering the more important aspects, as in the jam example.

techiferous 16y ago

"...that human reason has nothing to do with finding the truth, or locating the best alternative. Instead, it’s all about argumentation." I've come to the conclusion that our brain is not trying to produce an accurate model of reality. It is trying to choose good experiences over bad experiences. If it ends up constructing an accurate model of some part of reality, that's a side effect.

Tycho 16y ago

Reminds me of how some people review music. They start to focus on some things which were minor decisions by the recording artists, requiring no special talent (the reviewers themselves think they could have done better), like say the sequencing of songs on a CD, and then attach the same importance they would afford the quality of songs themselves. It's sort of like they devise a number of criteria and then naively give each category equal weighting. Videogames reviews are also bad for this.

chc 16y ago
Videogame reviews are bad for a multitude of reasons.

msluyter 16y ago

Why is it so hard to eradicate confirmation bias? Well, for one, even if you're acutely aware of it, you'll just tend to believe that your beliefs are that much likely to be correct. "After all, I've taken confirmation bias into account!"

mkr-hn 16y ago

What bugs me about this story and the conclusions is that it tries to compare taste--something wholly subjective--to other things. It implies that people use the same criteria of judgment for everything, which isn't even addressed by the study. How we judge our food doesn't necessarily reflect how we judge more important things like our political and social views. Has a similar study been done for those subjects? This is the worst kind of science reporting because it's trying to take a single study and draw wildly unrelated conclusions.

isleyaardvark 16y ago

This explains the notorious confirmation bias. This bias is apparent not only when people are actually arguing but also when they are reasoning proactively with the perspective of having to defend their opinions. And overcoming bias could also be motivated by wanting to persuade. It's covering your bases in case the person you're trying to persuade is aware of confirmation bias. I'd hate for this paper to become support for the idea that you should just go by your gut and not by reason. It looked like this study only looked at highly subjective topics, which jam is better, which poster/car/jelly bean is better. Those are largely subject to taste, and are much different subjects than something like economics or science.