6 ms·

Isn't this the very definition of confirmation bias?

by rfugger 14y ago

lotharbot 14y ago
It's more like Bayesian inference. In confirmation bias, you specifically seek out information that confirms your views and avoid/ignore information that might contradict it. The criteria you use to filter information is "do I want to believe this?" To use the example from the article, you might choose to only read articles from homeopathy sites, or to relentlessly nitpick articles from critics. In a Bayesian framework, you compare the strength of information that contradicts your views to the strength of the information that supports those views. The criteria you use to filter information is "is this good evidence in relation to the rest of the body of evidence?" In the homeopathy example, you'd study science and then note that there's a lot of strong evidence against it, and therefore require that evidence given to support it be stronger than "5% of placebos would give this same effect". It's the difference between ignoring contrary evidence and giving it the proper weighting.
pradocchia 14y ago
Others can quibble over definitions, but yes, it's effectively a fudge factor in favor of whatever we believe ought to be so, and against whatever we believe ought not be so. So on one hand, given the nature of statistics, it is necessary to cut down on the noise. On the other hand, it puts the model before reality, and necessarily obscures anything unexpected or contradictory.
- lotharbot 14y ago
  It's not a matter of ought versus ought not. It's a matter of evidence. If we have strong evidence against hypothesis X, we should require at least moderately strong evidence for hypothesis X before we accept it. We can continue to explore it, but we shouldn't give a lot of weight to weak evidence (such as likely false positives) in the face of considerably stronger evidence against. This does, necessarily, obscure certain types of surprising results when the data supporting them is limited. That's OK. Science sometimes converges slowly.
  - hej 14y ago
    Not so much evidence (from trials, say), rather plausibility. We do not understand how Homeopathy works, but it goes even a level deeper: There is a certain and very specific process for creating those homeopathic remedies and from what we currently know about, say, physics, that process cannot leave you with some sort of active ingredient. If homeopathy were to work, our understanding of physics (which we have lots of evidence for) would most certainly be wrong in some ways. That doesn’t mean that homeopathy is definitely wrong, but it does mean that the signal should be very strong for us to even be considering it to be not wrong. There are also levels to this: I’m pretty sure there are remedies about which we don’t know how they work, but them working would at least not require us to overhaul the laws of physics. That’s still worse than remedies for which we already have a model of how they work in mind, but at least it’s something. Although you can’t really quantify that, I think such plausibility arguments are vitally important for any research. In fact, since time and money is limited, all researchers (in any field) will always try to limit themselves to what is plausible. Explaining why your model and your hypotheses are plausible is a big part of any scientific work. Plausibility is already a big part of science.
    - lotharbot 14y ago
      The reason we say homeopathy is implausible is because we have a great deal of evidence to the contrary -- as you say, not clinical trials, but laws of physics that are very well understood and that are completely incompatible with homeopathy. The very idea of plausibility must always go back to evidence; otherwise it's merely wishful thinking.
  - pradocchia 14y ago
    If we have strong evidence against hypothesis X, we should require at least moderately strong evidence for hypothesis X before we accept it. That's a normative position: "We should require..." And a normative position with an excellent pedigree is still a normative position. That's not a bad thing, though. Good science requires good judgement. Good hypotheses do not spring wholly formed from logical deduction, there is necessarily an element of informed speculation and guess-making. Evidence must then be interpreted, weighted even, and credibility assigned to third-party results. So yes, it is very much a matter of ought and ought not.

Isn't this the very definition of confirmation bias?

by rfugger 14y ago

Isn't this the very definition of confirmation bias?

It's more like Bayesian inference. In confirmation bias, you specifically seek out information that confirms your views and avoid/ignore information that might contradict it. The criteria you use to filter information is "do I want to believe this?" To use the example from the article, you might choose to only read articles from homeopathy sites, or to relentlessly nitpick articles from critics. In a Bayesian framework, you compare the strength of information that contradicts your views to the strength of the information that supports those views. The criteria you use to filter information is "is this good evidence in relation to the rest of the body of evidence?" In the homeopathy example, you'd study science and then note that there's a lot of strong evidence against it, and therefore require that evidence given to support it be stronger than "5% of placebos would give this same effect". It's the difference between ignoring contrary evidence and giving it the proper weighting.

pradocchia 14y ago

Others can quibble over definitions, but yes, it's effectively a fudge factor in favor of whatever we believe ought to be so, and against whatever we believe ought not be so. So on one hand, given the nature of statistics, it is necessary to cut down on the noise. On the other hand, it puts the model before reality, and necessarily obscures anything unexpected or contradictory.

lotharbot 14y ago
It's not a matter of ought versus ought not. It's a matter of evidence. If we have strong evidence against hypothesis X, we should require at least moderately strong evidence for hypothesis X before we accept it. We can continue to explore it, but we shouldn't give a lot of weight to weak evidence (such as likely false positives) in the face of considerably stronger evidence against. This does, necessarily, obscure certain types of surprising results when the data supporting them is limited. That's OK. Science sometimes converges slowly.
- hej 14y ago
  Not so much evidence (from trials, say), rather plausibility. We do not understand how Homeopathy works, but it goes even a level deeper: There is a certain and very specific process for creating those homeopathic remedies and from what we currently know about, say, physics, that process cannot leave you with some sort of active ingredient. If homeopathy were to work, our understanding of physics (which we have lots of evidence for) would most certainly be wrong in some ways. That doesn’t mean that homeopathy is definitely wrong, but it does mean that the signal should be very strong for us to even be considering it to be not wrong. There are also levels to this: I’m pretty sure there are remedies about which we don’t know how they work, but them working would at least not require us to overhaul the laws of physics. That’s still worse than remedies for which we already have a model of how they work in mind, but at least it’s something. Although you can’t really quantify that, I think such plausibility arguments are vitally important for any research. In fact, since time and money is limited, all researchers (in any field) will always try to limit themselves to what is plausible. Explaining why your model and your hypotheses are plausible is a big part of any scientific work. Plausibility is already a big part of science.
  - lotharbot 14y ago
    The reason we say homeopathy is implausible is because we have a great deal of evidence to the contrary -- as you say, not clinical trials, but laws of physics that are very well understood and that are completely incompatible with homeopathy. The very idea of plausibility must always go back to evidence; otherwise it's merely wishful thinking.
- pradocchia 14y ago
  If we have strong evidence against hypothesis X, we should require at least moderately strong evidence for hypothesis X before we accept it. That's a normative position: "We should require..." And a normative position with an excellent pedigree is still a normative position. That's not a bad thing, though. Good science requires good judgement. Good hypotheses do not spring wholly formed from logical deduction, there is necessarily an element of informed speculation and guess-making. Evidence must then be interpreted, weighted even, and credibility assigned to third-party results. So yes, it is very much a matter of ought and ought not.