4 ms·

Others can quibble over definitions, but yes, it's effectively a fudge factor in favor of whatever we believe ought to be so, and against whatever we believe ou

Others can quibble over definitions, but yes, it's effectively a fudge factor in favor of whatever we believe ought to be so, and against whatever we believe ought not be so. So on one hand, given the nature of statistics, it is necessary to cut down on the noise. On the other hand, it puts the model before reality, and necessarily obscures anything unexpected or contradictory.

lotharbot 14y ago
It's not a matter of ought versus ought not. It's a matter of evidence. If we have strong evidence against hypothesis X, we should require at least moderately strong evidence for hypothesis X before we accept it. We can continue to explore it, but we shouldn't give a lot of weight to weak evidence (such as likely false positives) in the face of considerably stronger evidence against. This does, necessarily, obscure certain types of surprising results when the data supporting them is limited. That's OK. Science sometimes converges slowly.
- hej 14y ago
  Not so much evidence (from trials, say), rather plausibility. We do not understand how Homeopathy works, but it goes even a level deeper: There is a certain and very specific process for creating those homeopathic remedies and from what we currently know about, say, physics, that process cannot leave you with some sort of active ingredient. If homeopathy were to work, our understanding of physics (which we have lots of evidence for) would most certainly be wrong in some ways. That doesn’t mean that homeopathy is definitely wrong, but it does mean that the signal should be very strong for us to even be considering it to be not wrong. There are also levels to this: I’m pretty sure there are remedies about which we don’t know how they work, but them working would at least not require us to overhaul the laws of physics. That’s still worse than remedies for which we already have a model of how they work in mind, but at least it’s something. Although you can’t really quantify that, I think such plausibility arguments are vitally important for any research. In fact, since time and money is limited, all researchers (in any field) will always try to limit themselves to what is plausible. Explaining why your model and your hypotheses are plausible is a big part of any scientific work. Plausibility is already a big part of science.
  - lotharbot 14y ago
    The reason we say homeopathy is implausible is because we have a great deal of evidence to the contrary -- as you say, not clinical trials, but laws of physics that are very well understood and that are completely incompatible with homeopathy. The very idea of plausibility must always go back to evidence; otherwise it's merely wishful thinking.
- pradocchia 14y ago
  If we have strong evidence against hypothesis X, we should require at least moderately strong evidence for hypothesis X before we accept it. That's a normative position: "We should require..." And a normative position with an excellent pedigree is still a normative position. That's not a bad thing, though. Good science requires good judgement. Good hypotheses do not spring wholly formed from logical deduction, there is necessarily an element of informed speculation and guess-making. Evidence must then be interpreted, weighted even, and credibility assigned to third-party results. So yes, it is very much a matter of ought and ought not.
  - lotharbot 14y ago
    but it's not a matter of what we believe ought be so vs ought not be so -- that is, it's not a matter of what we want to be true. It's a matter of the strength of evidence we ought to have in order to draw conclusions, given other sets of evidence. It's a normative position, but it's not the specific normative position (based on confirmation bias) you described above. Think of it as a spectrum. On one end is the pure flake -- always giving too much weight to the newest evidence, no matter how weak, and therefore swinging wildly between different beliefs (I've heard this called "regressive bias"). On the other end is the pure dogmatist -- always giving too much weight to prior evidence, and therefore holding fast to a prior conclusion, ignoring or dismissing even strong contrary evidence ("confirmation bias"). Somewhere in the middle is the proper level of evidential weighting -- giving both old and new evidence the appropriate level of consideration, and therefore changing beliefs exactly as much as is warranted.

lotharbot 14y ago

It's not a matter of ought versus ought not. It's a matter of evidence. If we have strong evidence against hypothesis X, we should require at least moderately strong evidence for hypothesis X before we accept it. We can continue to explore it, but we shouldn't give a lot of weight to weak evidence (such as likely false positives) in the face of considerably stronger evidence against. This does, necessarily, obscure certain types of surprising results when the data supporting them is limited. That's OK. Science sometimes converges slowly.

hej 14y ago
Not so much evidence (from trials, say), rather plausibility. We do not understand how Homeopathy works, but it goes even a level deeper: There is a certain and very specific process for creating those homeopathic remedies and from what we currently know about, say, physics, that process cannot leave you with some sort of active ingredient. If homeopathy were to work, our understanding of physics (which we have lots of evidence for) would most certainly be wrong in some ways. That doesn’t mean that homeopathy is definitely wrong, but it does mean that the signal should be very strong for us to even be considering it to be not wrong. There are also levels to this: I’m pretty sure there are remedies about which we don’t know how they work, but them working would at least not require us to overhaul the laws of physics. That’s still worse than remedies for which we already have a model of how they work in mind, but at least it’s something. Although you can’t really quantify that, I think such plausibility arguments are vitally important for any research. In fact, since time and money is limited, all researchers (in any field) will always try to limit themselves to what is plausible. Explaining why your model and your hypotheses are plausible is a big part of any scientific work. Plausibility is already a big part of science.
- lotharbot 14y ago
  The reason we say homeopathy is implausible is because we have a great deal of evidence to the contrary -- as you say, not clinical trials, but laws of physics that are very well understood and that are completely incompatible with homeopathy. The very idea of plausibility must always go back to evidence; otherwise it's merely wishful thinking.
pradocchia 14y ago
If we have strong evidence against hypothesis X, we should require at least moderately strong evidence for hypothesis X before we accept it. That's a normative position: "We should require..." And a normative position with an excellent pedigree is still a normative position. That's not a bad thing, though. Good science requires good judgement. Good hypotheses do not spring wholly formed from logical deduction, there is necessarily an element of informed speculation and guess-making. Evidence must then be interpreted, weighted even, and credibility assigned to third-party results. So yes, it is very much a matter of ought and ought not.
- lotharbot 14y ago
  but it's not a matter of what we believe ought be so vs ought not be so -- that is, it's not a matter of what we want to be true. It's a matter of the strength of evidence we ought to have in order to draw conclusions, given other sets of evidence. It's a normative position, but it's not the specific normative position (based on confirmation bias) you described above. Think of it as a spectrum. On one end is the pure flake -- always giving too much weight to the newest evidence, no matter how weak, and therefore swinging wildly between different beliefs (I've heard this called "regressive bias"). On the other end is the pure dogmatist -- always giving too much weight to prior evidence, and therefore holding fast to a prior conclusion, ignoring or dismissing even strong contrary evidence ("confirmation bias"). Somewhere in the middle is the proper level of evidential weighting -- giving both old and new evidence the appropriate level of consideration, and therefore changing beliefs exactly as much as is warranted.