5 ms·

One of the must-read papers in this area is a very short, very readable piece by Chris Achen called "Let's Put Garbage-Can Regressions and Garbage-Can Probits W

by p4wnc6 11y ago

One of the must-read papers in this area is a very short, very readable piece by Chris Achen called "Let's Put Garbage-Can Regressions and Garbage-Can Probits Where They Belong." [0] The basic idea is that if your data experiences non-linear coding errors due to some external process before it gets to you, then as related to the true causal relationship between your variables, you're actually fitting a (possibly) non-linear transformation. If you do this and still use the classical cook-book t-stat / p-value stuff, you can run into big problems. As the paper shows, even for an extremely simple data set where there is merely a non-linear coding error, the coefficient can be very statistically significant (t-stat magnitude > 2) and yet it will have the wrong sign. This is a really remarkable thing. By just jittering your data a bit, you can obtain a result that passes the usual, simplistic significance tests but for which the effect size is negative when it should be positive (or vice versa). I remember when I worked in quant finance and we were using a terrible, shamefully bad automated framework for frequentist model fitting, the sense of dread was huge. If the data we used had been jittered by the data vendor, or if there was a slight bug in the code that did variable cleaning, outlier manipulation, or scoring, or any of a hundred other steps where tiny non-linearities could be introduced, then it would be entirely possible for us to see "significant" results that pointed in the opposite direction from the truth. There are many other papers which point out the pitfalls of naive dump-it-all-on-the-right-hand-side regression models, but I think this one from Achen is unique in that it is extremely short, extremely simple to follow, and yet it is fully devastating to this entire technique of modeling. [0] http://www.columbia.edu/~gjw10/achen04.pdf http://www.columbia.edu/~gjw10/achen04.pdf

thyrsus 11y ago
I have often wondered at linearity problems. E.g., why is the BMI formula based on weight divided by a square of height when geometry says that weight is proportional to the cube of (similarly proportioned) height? Probably, in the region of interest, the one curve approximates the other, but why fight locally Euclidean fidelity? Feedback mechanisms are common; unless you look for the correct function (whether polynomial, exponential, factorial, etc.), as implied by the underlying mechanism, throwing regression at the problem is going to obfuscate.
jhbadger 11y ago
There certainly are poorly done regression analyses out there and yes, things like coding-errors and correlations of variables can screw things up. But the point of regression (at least in the natural/medical sciences) isn't to give a definitive answer but to suggest actual experiments to be done. I think it isn't coincidence that both your link and the original link are from social scientists. There it is much harder (and often impossible) to conduct followup experimental studies based on the regression and so the regression often ends up as the end result.
- p4wnc6 11y ago
  I don't think very many practitioners share your view that regressions are meant to be simplistic first steps on the path to more sophisticated analyses or experiments. If that were true, then I would definitely agree with you that we can forgive most of the shortcommings of frequentist regressions. In practice and in academia, though, I've only ever experienced researchers who see regressions as the main result, and they intend on exclusively using regression models even before they have any sense of what the data is or what the challenges will be. In a lot of cases, they even go out of their way to build frameworks that hard-code in the assumption that whatever model is used must be representable as a set of coefficients and their standard errors. It's painful to see it happen: people have been using regression models for so long (due to the easy manipulation of what the results "mean") that they can't even imagine any other way of doing things. Even in cases where someone might want to earnestly prioritize a move to a more accurate or more flexible modeling framework, they often can't because decades of legacy code has been tailored to work solely with coefficient-based regression models. The software development cost of switching would be so high that they would rather just find excuses to hide the model's inaccuracy and "sell" the results than to bite the bullet and invest in systems where the results are actually statistically rigorous.

by p4wnc6 11y ago

thyrsus 11y ago

I have often wondered at linearity problems. E.g., why is the BMI formula based on weight divided by a square of height when geometry says that weight is proportional to the cube of (similarly proportioned) height? Probably, in the region of interest, the one curve approximates the other, but why fight locally Euclidean fidelity? Feedback mechanisms are common; unless you look for the correct function (whether polynomial, exponential, factorial, etc.), as implied by the underlying mechanism, throwing regression at the problem is going to obfuscate.

jhbadger 11y ago

There certainly are poorly done regression analyses out there and yes, things like coding-errors and correlations of variables can screw things up. But the point of regression (at least in the natural/medical sciences) isn't to give a definitive answer but to suggest actual experiments to be done. I think it isn't coincidence that both your link and the original link are from social scientists. There it is much harder (and often impossible) to conduct followup experimental studies based on the regression and so the regression often ends up as the end result.

p4wnc6 11y ago
I don't think very many practitioners share your view that regressions are meant to be simplistic first steps on the path to more sophisticated analyses or experiments. If that were true, then I would definitely agree with you that we can forgive most of the shortcommings of frequentist regressions. In practice and in academia, though, I've only ever experienced researchers who see regressions as the main result, and they intend on exclusively using regression models even before they have any sense of what the data is or what the challenges will be. In a lot of cases, they even go out of their way to build frameworks that hard-code in the assumption that whatever model is used must be representable as a set of coefficients and their standard errors. It's painful to see it happen: people have been using regression models for so long (due to the easy manipulation of what the results "mean") that they can't even imagine any other way of doing things. Even in cases where someone might want to earnestly prioritize a move to a more accurate or more flexible modeling framework, they often can't because decades of legacy code has been tailored to work solely with coefficient-based regression models. The software development cost of switching would be so high that they would rather just find excuses to hide the model's inaccuracy and "sell" the results than to bite the bullet and invest in systems where the results are actually statistically rigorous.