4 ms·

Assume that there are always errors in the data, and that the errors in the data are unbiased - i.e. that half of the errors are biased towards the model and ha

by rm-rf 17y ago

Assume that there are always errors in the data, and that the errors in the data are unbiased - i.e. that half of the errors are biased towards the model and half are biased away from the model. In cases where the errors are biased away from the model (the errors do not support the hypothesis that the model is based on), the scientist will have a tendency to double-check the data, comb through it and correct the errors. That's certainly what I'd do. In cases where the errors are biased toward the model however, there is no incentive for the scientists to comb the data and correct the errors. Why would she, as the data re-enforces the model and the hypothesis. Publish the paper, get tenure and take a sabbatical. It sounds to me like there will always be a bias toward the model - only because the data will only get the extra scrutiny that it needs to eliminate measurement errors in cases where it disagrees with the model, thereby taking unbiased errors and turning them into biased errors, with the bias always toward the model. That's my hypothesis, and to make sure it's valid, I'll accept without question any data that supports it, and I'll double-extra analyze and carefully correct any data that doesn't.

gjm11 17y ago
If the model is very good then random errors will not be "unbiased" in your sense: they will all be "away from the model". In fact, if the model is any good at all then random errors will more often be away from it than towards it. There is an incentive for scientists to check for errors that make their models look better. You're more likely to get famous for finding an error in a widely used model than for yet another confirmation that it works OK. (Even more likely if you can come up with a better model. More likely still if that better model is different in illuminating ways.) I expect there still is a tendency for measurements to get distorted towards better fit with models. But it's not as one-sided as you make it sound.
- rm-rf 17y ago
  "There is an incentive for scientists to check for errors..." Unless your career, grants and funding are dependent on the validity of the model, or if the model is highly politicized. In those cases, there is no incentive for finding errors that undermine the model. Richard Feynmans paper on Cargo Cult Science has a explanation of how Millikans electron charge measurements drifted over time because of measurement bias. In that case, there almost certainly wasn't any politics involved, so the follow on experiments drifted toward a more accurate measurement. But as Feynman indicates in his paper, the fact that the measurements drifted slowly and incrementally, rather than in a corrective step indicates that the scientists who made the follow on measurements were biased towards Millikans' original results - even though his original measurements were not as accurate as the follow on measurements.

by rm-rf 17y ago

gjm11 17y ago

If the model is very good then random errors will not be "unbiased" in your sense: they will all be "away from the model". In fact, if the model is any good at all then random errors will more often be away from it than towards it. There is an incentive for scientists to check for errors that make their models look better. You're more likely to get famous for finding an error in a widely used model than for yet another confirmation that it works OK. (Even more likely if you can come up with a better model. More likely still if that better model is different in illuminating ways.) I expect there still is a tendency for measurements to get distorted towards better fit with models. But it's not as one-sided as you make it sound.

rm-rf 17y ago
"There is an incentive for scientists to check for errors..." Unless your career, grants and funding are dependent on the validity of the model, or if the model is highly politicized. In those cases, there is no incentive for finding errors that undermine the model. Richard Feynmans paper on Cargo Cult Science has a explanation of how Millikans electron charge measurements drifted over time because of measurement bias. In that case, there almost certainly wasn't any politics involved, so the follow on experiments drifted toward a more accurate measurement. But as Feynman indicates in his paper, the fact that the measurements drifted slowly and incrementally, rather than in a corrective step indicates that the scientists who made the follow on measurements were biased towards Millikans' original results - even though his original measurements were not as accurate as the follow on measurements.