5 ms·
But only the data that doesn't fit the model is fixed, and attempts to fix it stop when it does fit the data. A lesson in machine learning I'll never forget is
by greendestiny 17y ago
But only the data that doesn't fit the model is fixed, and attempts to fix it stop when it does fit the data.
A lesson in machine learning I'll never forget is this: no matter how careful you are about not testing on training data, running different parameters until you get the results you want is doing just that.
- tel 17y agoI didn't get the impression only bad data was fixed, instead they used model misfit to highlight deficiencies in the model (it didn't account for all information, namely the motion of the satellite) and then updated the model. The question becomes at what point does adding more degrees of freedom to your model lead to overfitting, though for reasons I've never felt fully confident in, Gelman tends to assume that Bayesian methods don't overfit.
- sesqu 17y agoThere is some amount of theory to account for this effect, but I'm not familiar enough with any of it to restate particulars. In general, the two approaches I know of are to strongly favor simpler models, or to increase expected error as you develop new hypotheses. A related effect is the tendency of finite random samples to express spurious structure, which is guarded against by simply assuming the general structure of the effect.