5 ms·

But when the data are complicated --- by which I mean, there are many different effects that must be accounted for in order to interpret the raw data as a measu

by miked 17y ago

But when the data are complicated --- by which I mean, there are many different effects that must be accounted for in order to interpret the raw data as a measurement of a parameter of interest --- then it's not necessarily a surprise to find problems with the data, and to find that when those problems are fixed the result is better agreement with a model. "The problems are fixed" based on what? You can't use the model to "fix" them, since the model has validity only insofar as it tracks data points. In short, this is simply circular reasoning: the model has no validity beyond it's ability to explain and predict data values.

Luc 17y ago
Why do you assume the model is needed to fix the data? You are jumping to the wrong conclusion, both in the general case and in the specific case of this article (where the data was fixed after taking into account that a satellite had dropped 20km in orbit, or after excluding faulty thermometers).
- greendestiny 17y ago
  But only the data that doesn't fit the model is fixed, and attempts to fix it stop when it does fit the data. A lesson in machine learning I'll never forget is this: no matter how careful you are about not testing on training data, running different parameters until you get the results you want is doing just that.
  - tel 17y ago
    I didn't get the impression only bad data was fixed, instead they used model misfit to highlight deficiencies in the model (it didn't account for all information, namely the motion of the satellite) and then updated the model. The question becomes at what point does adding more degrees of freedom to your model lead to overfitting, though for reasons I've never felt fully confident in, Gelman tends to assume that Bayesian methods don't overfit.
  - sesqu 17y ago
    There is some amount of theory to account for this effect, but I'm not familiar enough with any of it to restate particulars. In general, the two approaches I know of are to strongly favor simpler models, or to increase expected error as you develop new hypotheses. A related effect is the tendency of finite random samples to express spurious structure, which is guarded against by simply assuming the general structure of the effect.
ezy 17y ago
You figure out what the problem in the measurements might be and correct for that based on your hypothesis of what the problem might be -- then compare the new results to the model predictions (and other data). The tacit assumption is that model should predict the data, if it doesn't either the data or the model is borked. If other data supports the model, then you're going to look at the data first -- especially if it's new data collected in a new way. The satellite example is perfect, they didn't take the model output and add a factor based on that. No, they hypothesized that the satellite height was different and that was affecting the data, checked that, found ample evidence for it. Adjusted the data based on the height (not the model) and lo and behold, the data more closely matched the existing model. The likelihood that this kind of independent factor makes all the difference for a bad model is quite small. Where the bias might come in is that one would have a tendency to only look for errors in data not fitting the model(s) or other data. But no one is suggesting that you readjust the data to match the model output -- what they may be saying is that if your data doesn't match the model -- you better be able to justify it.. and that starts with the data and proceeds to a new iteration of the model as you gain more confidence in the data.
- yummyfajitas 17y ago
  Everything you describe sounds fine, but in the aggregate it privileges the hypothesis. When data supports the model, it is accepted. When it contradicts the model, extra scrutiny is applied. In short, a higher standard of evidence is required when you disagree with the hypothesis. It's kind of like accepting p-values of 0.05 when you agree with the model, but 0.01 when you disagree. It creates a bias in favor of the model. When huge arbitrary amounts of new evidence are available (e.g., an experimental field), this won't matter. In fields without experiments (climate science, oceanography, economics), this effect can protect a bad hypothesis.

by miked 17y ago

Luc 17y ago

Why do you assume the model is needed to fix the data? You are jumping to the wrong conclusion, both in the general case and in the specific case of this article (where the data was fixed after taking into account that a satellite had dropped 20km in orbit, or after excluding faulty thermometers).

greendestiny 17y ago
But only the data that doesn't fit the model is fixed, and attempts to fix it stop when it does fit the data. A lesson in machine learning I'll never forget is this: no matter how careful you are about not testing on training data, running different parameters until you get the results you want is doing just that.
- tel 17y ago
  I didn't get the impression only bad data was fixed, instead they used model misfit to highlight deficiencies in the model (it didn't account for all information, namely the motion of the satellite) and then updated the model. The question becomes at what point does adding more degrees of freedom to your model lead to overfitting, though for reasons I've never felt fully confident in, Gelman tends to assume that Bayesian methods don't overfit.
- sesqu 17y ago
  There is some amount of theory to account for this effect, but I'm not familiar enough with any of it to restate particulars. In general, the two approaches I know of are to strongly favor simpler models, or to increase expected error as you develop new hypotheses. A related effect is the tendency of finite random samples to express spurious structure, which is guarded against by simply assuming the general structure of the effect.

ezy 17y ago

You figure out what the problem in the measurements might be and correct for that based on your hypothesis of what the problem might be -- then compare the new results to the model predictions (and other data). The tacit assumption is that model should predict the data, if it doesn't either the data or the model is borked. If other data supports the model, then you're going to look at the data first -- especially if it's new data collected in a new way. The satellite example is perfect, they didn't take the model output and add a factor based on that. No, they hypothesized that the satellite height was different and that was affecting the data, checked that, found ample evidence for it. Adjusted the data based on the height (not the model) and lo and behold, the data more closely matched the existing model. The likelihood that this kind of independent factor makes all the difference for a bad model is quite small. Where the bias might come in is that one would have a tendency to only look for errors in data not fitting the model(s) or other data. But no one is suggesting that you readjust the data to match the model output -- what they may be saying is that if your data doesn't match the model -- you better be able to justify it.. and that starts with the data and proceeds to a new iteration of the model as you gain more confidence in the data.

yummyfajitas 17y ago
Everything you describe sounds fine, but in the aggregate it privileges the hypothesis. When data supports the model, it is accepted. When it contradicts the model, extra scrutiny is applied. In short, a higher standard of evidence is required when you disagree with the hypothesis. It's kind of like accepting p-values of 0.05 when you agree with the model, but 0.01 when you disagree. It creates a bias in favor of the model. When huge arbitrary amounts of new evidence are available (e.g., an experimental field), this won't matter. In fields without experiments (climate science, oceanography, economics), this effect can protect a bad hypothesis.