4 ms·

Hmm... So here is an interesting thought experiment. Suppose you take a person with some appreciable intelligence (at least average) but no particular knowledg

Hmm... So here is an interesting thought experiment. Suppose you take a person with some appreciable intelligence (at least average) but no particular knowledge about a certain topic. In this instance, we'll let that topic be psycology. Now we present this person with an unfortunate dilemma. For a particular hypothesis, they observe a significant amount of peer reviewed literature asserting empirical evidence in the affirmative. They don't have any real familiarity with any individual papers, but they can understand that there's an established consensus. On the other hand they are given an article like this one, which mounts a critical refutation of all the established literature. Furthermore, this lone paper is presented to our stalwart examiner amidst the zeitgeist of a reproducibility crisis. Thus we have a mountain of peer reviewed but undigestible evidence on one side, and one readily digestible paper on the other which specifically rebuts the mountain of evidence. The fundamental dilemma is this: how should this person examine the available evidence to maximize their chances of coming to the correct conclusion? Should they abstain from trying to discern the truth of the matter, and strike out any opinion they could have as unqualified? Should they read through the most comprehensive surveys of available evidence to come to a full understanding? Should they take the new, contrarian paper at face value? There are a few dimensions here which (from my view) make the dilemma nearly intractable unless you 1) abstain from an opinion, or 2) become a subject matter expert. The cloud of the reproducibility crisis is a first dimension of uncertainty - not only does that muddy the waters for existig research, but we also have to be careful not take contradictory research at face value simply because it's contrarian. For another, we have to figure out how to weight the reliability of evidence against our own time. It's tempting to discount the evidence of existing research if an especially compelling critique is released, because we can more easily read it and follow its arguments. It seems like this is enormously difficulty all around. How do we rate the critical correctness of differing amounts of conflicting literature published under academic uncertainty?

jadedhacker 8y ago
Usually the solution is to have a graduate student write a review paper. ;)
jjeaff 8y ago
In your thoughts experiment, make your refuting paper a slick Netflix documentary and you have the state we live in now.
- throwawaymath 8y ago
  That's a good point. In some social circles I've observed, it's become fashionable to have the latest contrarian research ready to cite for popsci topics. It seems like the new way to be well-read at a cocktail party. Documentaries providing new narratives to cleanly refute the old ones - all the while promoting social awareness - seem to be very "in" these days.
raverbashing 8y ago
My bet is that the "mountain of evidence" is more specific and less generalizable than assumed. The reproducibility "crisis" is not a crisis, it is a fundamental limitation of the scientific method for things that depend on a greater number of variables, on which a lot are unobservable and/or not known.
vkou 8y ago
There is no alternative to 1) or 2). You either have to follow consensus of the experts, or, to go contrary to consensus, you must understand the consensus well enough to be one of the experts. Down any other road lies pop-sci nonsense. It's incredibly easy to be a wrong contrarian, when you don't actually understand what you are attacking.
- giergirey 8y ago
  I think you're missing option 3, which is to avoid drawing a conclusion either way - assume that there's no definite answer, and that the experts' expertise is too narrow to support using their consensus outside small studies on college students buying and selling chocolate bars. In this case I'm happy to conclude that it's not clear if people do systematically make poor judgements on important issues due to an in-built "loss aversion" heuristic.
- darkerside 8y ago
  What you call popsci nonsense is the way most of the people on this planet live. Making decisions based on the available data and living with the fact that we may be wrong. Nerds are a special class, and not better nor worse.
  - vkou 8y ago
    They are absolutely better at things they are experts on. You don't hire a plumber to do a colonoscopy.
    - darkerside 8y ago
      Not better or worse in terms of value as human beings
gboudrias 8y ago
Your question is certainly interesting in an epistemological sense, in a way it is the basis of relativism. Having read Larry Laudan recently however, I'm a big fan of his pragmatism which trumps the question a bit: just use whatever works. In this case, we don't all need to be aware of loss aversion, and really I suspect barely anyone was using it in a practical sense. A pragmatist might say that it was only ever a theory, and this is why it thrived in largely theoretical exercises (or in the case of economics, in a context where the tangible consequences were extremely far removed from the application of the theory). But to me loss aversion still seems like an observation after the fact, though I did believe it at the time, which are the worst kind of observations ;) In short, from where I stand there is no such thing as "correctness", only what has been successfully applied and in what context, or temporary applicability if you will. Any further interpretation is usually a case of extrapolating knowledge from a vastly incomplete picture. Being a psychology student, I look at its early history as a tragic example of why this is counterproductive. Some habits are hard to break though.
- Vinnl 8y ago
  So what if the "certain topic" is something with no immediately obvious thing that "works", e.g. climate change?
  - gboudrias 8y ago
    I'm afraid I don't know enough to talk about it, but to me it seems more of a collection of observations than a "theory" in the speculative sense. As far as I'm aware though, we have observed that cities with less greenhouse gases tend to be colder, so in that sense lowering CO2 levels "works" to reduce temperature, though we can't speak of global "correctness" until we manage the same with the global temperature. This is a special case however, in that we have to assume correctness because otherwise we'll all be much worse off, and the possible cost of reducing pollution are slim in comparison. But if anything I think this supports my point that correctness itself doesn't matter, only the material consequences do.
    - Vinnl 8y ago
edanm 8y ago
> The fundamental dilemma is this: how should this person examine the available evidence to maximize their chances of coming to the correct conclusion? Should they abstain from trying to discern the truth of the matter, and strike out any opinion they could have as unqualified? > There are a few dimensions here which (from my view) make the dilemma nearly intractable unless you 1) abstain from an opinion I think the majority of the time, you should abstain. I mean, you can certainly debate things, it's fun to do and improves your thinking. But you should be sure of very few "controversial" things. You should be very ready to change your mind about most things. There is value in just saying "I'm not sure" about most things. How do I define controversial? Well yeah this becomes a circular problem very quickly. As others have said, practically speaking, you usually don't really need to use the latest social science research for practical purposes. Sometimes you do - but hopefully then you are an expert. However, sometimes there are things you need to decide. E.g., what diet maximizes your health/fitness goals? A classic case where there is a huge lack of expert consensus, etc. In these situations, I usually try to defer to who sounds smartest in general to me, but keep a very open mind to the idea that I might be totally wrong.
jpttsn 8y ago
Abstaining from the general case truth frees up time for figuring out some interesting special cases. I don’t know the best diet for the general population. I know roughly what works for me. I don’t know how microwaves work. I know how my microwave oven works.
ilyagr 8y ago
I think your goal should be not to judge to what degree each side is correct, but to learn a little about what people in the area are doing, what they did in the past, what they find important, and why. That way, there is a lot more you can learn and gain.
MarkMc 8y ago
For me it works like this: Before reading this HN item, my understanding was that 'Loss aversion' was very likely to be correct - let's say 98%. After reading the headline and a few comments I lower the 'chance of correctness' a little to say 93%. Two points on this approach: 1. My assessment lies on a gradient rather than a simple true/false. 2. I rarely read the source evidence - my assessment relies mostly on the assessment of others.

jadedhacker 8y ago

Usually the solution is to have a graduate student write a review paper. ;)

jjeaff 8y ago

In your thoughts experiment, make your refuting paper a slick Netflix documentary and you have the state we live in now.

throwawaymath 8y ago
That's a good point. In some social circles I've observed, it's become fashionable to have the latest contrarian research ready to cite for popsci topics. It seems like the new way to be well-read at a cocktail party. Documentaries providing new narratives to cleanly refute the old ones - all the while promoting social awareness - seem to be very "in" these days.

raverbashing 8y ago

My bet is that the "mountain of evidence" is more specific and less generalizable than assumed. The reproducibility "crisis" is not a crisis, it is a fundamental limitation of the scientific method for things that depend on a greater number of variables, on which a lot are unobservable and/or not known.

vkou 8y ago

There is no alternative to 1) or 2). You either have to follow consensus of the experts, or, to go contrary to consensus, you must understand the consensus well enough to be one of the experts. Down any other road lies pop-sci nonsense. It's incredibly easy to be a wrong contrarian, when you don't actually understand what you are attacking.

giergirey 8y ago
I think you're missing option 3, which is to avoid drawing a conclusion either way - assume that there's no definite answer, and that the experts' expertise is too narrow to support using their consensus outside small studies on college students buying and selling chocolate bars. In this case I'm happy to conclude that it's not clear if people do systematically make poor judgements on important issues due to an in-built "loss aversion" heuristic.
darkerside 8y ago
What you call popsci nonsense is the way most of the people on this planet live. Making decisions based on the available data and living with the fact that we may be wrong. Nerds are a special class, and not better nor worse.
- vkou 8y ago
  They are absolutely better at things they are experts on. You don't hire a plumber to do a colonoscopy.
  - darkerside 8y ago
    Not better or worse in terms of value as human beings

gboudrias 8y ago

Your question is certainly interesting in an epistemological sense, in a way it is the basis of relativism. Having read Larry Laudan recently however, I'm a big fan of his pragmatism which trumps the question a bit: just use whatever works. In this case, we don't all need to be aware of loss aversion, and really I suspect barely anyone was using it in a practical sense. A pragmatist might say that it was only ever a theory, and this is why it thrived in largely theoretical exercises (or in the case of economics, in a context where the tangible consequences were extremely far removed from the application of the theory). But to me loss aversion still seems like an observation after the fact, though I did believe it at the time, which are the worst kind of observations ;) In short, from where I stand there is no such thing as "correctness", only what has been successfully applied and in what context, or temporary applicability if you will. Any further interpretation is usually a case of extrapolating knowledge from a vastly incomplete picture. Being a psychology student, I look at its early history as a tragic example of why this is counterproductive. Some habits are hard to break though.

Vinnl 8y ago
So what if the "certain topic" is something with no immediately obvious thing that "works", e.g. climate change?
- gboudrias 8y ago
  I'm afraid I don't know enough to talk about it, but to me it seems more of a collection of observations than a "theory" in the speculative sense. As far as I'm aware though, we have observed that cities with less greenhouse gases tend to be colder, so in that sense lowering CO2 levels "works" to reduce temperature, though we can't speak of global "correctness" until we manage the same with the global temperature. This is a special case however, in that we have to assume correctness because otherwise we'll all be much worse off, and the possible cost of reducing pollution are slim in comparison. But if anything I think this supports my point that correctness itself doesn't matter, only the material consequences do.
  - Vinnl 8y ago

edanm 8y ago

> The fundamental dilemma is this: how should this person examine the available evidence to maximize their chances of coming to the correct conclusion? Should they abstain from trying to discern the truth of the matter, and strike out any opinion they could have as unqualified? > There are a few dimensions here which (from my view) make the dilemma nearly intractable unless you 1) abstain from an opinion I think the majority of the time, you should abstain. I mean, you can certainly debate things, it's fun to do and improves your thinking. But you should be sure of very few "controversial" things. You should be very ready to change your mind about most things. There is value in just saying "I'm not sure" about most things. How do I define controversial? Well yeah this becomes a circular problem very quickly. As others have said, practically speaking, you usually don't really need to use the latest social science research for practical purposes. Sometimes you do - but hopefully then you are an expert. However, sometimes there are things you need to decide. E.g., what diet maximizes your health/fitness goals? A classic case where there is a huge lack of expert consensus, etc. In these situations, I usually try to defer to who sounds smartest in general to me, but keep a very open mind to the idea that I might be totally wrong.

jpttsn 8y ago

Abstaining from the general case truth frees up time for figuring out some interesting special cases. I don’t know the best diet for the general population. I know roughly what works for me. I don’t know how microwaves work. I know how my microwave oven works.

ilyagr 8y ago

I think your goal should be not to judge to what degree each side is correct, but to learn a little about what people in the area are doing, what they did in the past, what they find important, and why. That way, there is a lot more you can learn and gain.

MarkMc 8y ago

For me it works like this: Before reading this HN item, my understanding was that 'Loss aversion' was very likely to be correct - let's say 98%. After reading the headline and a few comments I lower the 'chance of correctness' a little to say 93%. Two points on this approach: 1. My assessment lies on a gradient rather than a simple true/false. 2. I rarely read the source evidence - my assessment relies mostly on the assessment of others.