6 ms·

Mindblowing. No wonder so much science doesn't replicate. I read scientific papers fairly often compared to the average person, and this is probably the most i

Mindblowing. No wonder so much science doesn't replicate. I read scientific papers fairly often compared to the average person, and this is probably the most important I've read in years. It feels like a similar problem to Spectre in the CPU design space: a problem nobody noticed for decades, that breaks fundamental assumptions in ways that appear nearly impossible to fix. Except this is way bigger and more important than Spectre. Here's a quick summary of the paper and its conclusions: 1. Virtually all claims of the form "We have found X is an important factor when controlling for other factors" are wrong, because they ignore the possibility of measurement error, which has unintuitive and enormous effects. 2. As a consequence, vast swathes of so-called findings in medical and social sciences are spurious to an even greater extent than previously realised. The paper cites Google Scholar searches to show that hundreds of thousands of papers contain trigger phrases like "after controlling for". 3. The problem is fundamental and can't be fixed. For (3) I should explain, because the paper does propose what initially looks like a fix: an improved form of statistical modelling that incorporates measurement error into the outcomes (SEM). This can restore statistical validity. Unfortunately as we study this idea closer it becomes clear that there's no way scientists are going to adopt it: • It explodes the sample sizes and therefore cost. • Frequently we have no idea what the measurement error actually is. • Therefore it becomes impossible to say with any confidence whether a claim is true or false. Even very small amounts of measurement error can push the confidence level of the outcome under the already arbitrary and probably too lax 95% threshold. To detect a weak effect size of 0.1, handling realistic error rates could easily take you from needing hundreds of samples (grad student populations) to tens of thousands. To clarify the term "measurement error" here, what the paper gets at is that in social sciences they are often trying to measure things indirectly via e.g. surveys which have some inherent noisiness due to people lying, not understanding the question, not being sure of the answer but being unwilling to admit it etc. And then on top of that they are often trying to measure something ill-defined to begin with, e.g. mood, intelligence, wealth. But as noise rates go up, the chances of spuriously detecting correlations that don't exist reaches 100%. The paper does various simulations to show that this can easily happen with sample sizes, p values and error rates that are standard. The part that really nailed it for me was where they use a hypothetical example of a correlation between ice cream sales and swimming pool deaths. Common sense tells us ice cream doesn't cause people to die in pools, but rather, on hot days more people go swimming and buy ice cream: it's a confounding variable. In the case where the confounding variable is accurately measured with a thermometer, measurement errors are zero and the typical sort of regression analysis scientists use shows no correlation as expected. But if temperature is measured using a survey where people self report perceived body temperatures on the Likert scale, then you get an error rate of about 0.4. At that point a regression analysis shows a strong direct correlation between eating ice cream and dying in a pool with p < 0.001 - much lower than the p < 0.05 threshold needed to trigger publishing. A big question is what does this mean for science and the academy? I can see nothing happening in the short run: it seems this paper wasn't picked up by journalists, or at least I've not seen any mention of it even in discussions of the replication crisis. Scientists themselves can't really pay attention to it without breaking the entire funding model of non-STEM sciences. In the long run this could possibly lead to a major breakdown in society's notions of expertise, trustworthiness of scientists and the value of government funded academia (corporate funding of social sciences is close to zero). How many decades have people been reading stories of the form "$foo causes $bar" or "$baz (good thing) is highly correlated with $zap (bad thing)"? Many people have already clocked that this kind of science is unreliable - now we seem to know precisely why.

graycat 8y ago
For measurements, the social scientists have been considering what they call reliability and validity for a long time. So, they do consider the accuracy of measurements. In science we collect data then use the data. If the data has some measurement error, no big surprise. Instead just say that the model is in terms of the measured values with their errors instead of some much more accurate values that don't have. Or, if have more accurate values, then use them. Else don't feel guilty and beat up on self. See also my earlier post in this thread on cross tabulation.
- tomrod 8y ago
  Why isn't total least squares used to control for measurement error in regressors?
  - graycat 8y ago
    By "total least squares" you mean to pick regression coefficients that minimize the squared errors between the observed values of the dependent variable and the predicted value from the regression? The predicted values are from a perpendicular projection, and we do get the Pythagorean theorem with total sum of squares = regression sum of squares + error sum of squares So, we minimize the error sum of squares. Does that really control on measurement error? Not in any simple way I can see. Or, if we have errors in the measurements of the independent variables, then we are facing one of the facts of life, we don't have the error-free, true values. Not good but usually not as bad as having your marriage fail or losing your pet dog or cat! Like the video clip of a Heifetz master class where a student tried to play the D flat minor scale and at the end Heifetz assured the student that they were still alive! Maybe what you are saying is that with enough mathematical assumptions, e.g., the famous heteroscedasticity with independent, identically distributed mean zero Gaussian errors, as the number of observations goes to infinity the errors wash out much as in the weak/strong laws of large numbers and we get to f'get about the errors -- maybe there is such a theorem, I should get out my copy of the old C. Radhakrishna Rao, Linear Statistical Inference and Its Applications: Second Edition, ISBN 0-471-70823-2, John Wiley and Sons, New York. and look or do some such derivations for myself. But, to what end if we don't believe the mathematical assumptions for the mathematical theorems, e.g., heteroscedasticity with independent, identically distributed mean zero Gaussian errors? Sorry, from 50,000 feet up, it seems to me that having control variables in regression is shaky stuff. And without some careful derivations, we should not be surprised at the effects of various errors. Also, the usual derivations of the math are in the context of just some one regression model where we make all those assumptions. Instead, given one dependent variable and 10 independent variables plus five more we believe are causes plus 10 more we want to use as controls, the dependent variable plus 25 more variables in all, last time I checked we were short on how to pick the 2^25 sets of independent variables and make sense of the different, maybe wildly different, coefficients we get. Here's a simple view: If we have 5 independent variables and they are all orthogonal, then we can get the regression coefficients one at a time just from 5 projections, covariances, inner products (all essentially the same things except how we scale things) and have those coefficient just the same for any of the 2^5 regression analyses. That is, if we have orthogonal independent variables U, V, W, X, and Y and dependent variable Z, then we can get the coefficients one at a time and be done -- have all the regression coefficients for all 2^5 regressions. Otherwise, without othogonality, we face some possibly tricky math derivations -- maybe they are in Rao's book, it's thick enough -- and are asking a bit too much from regression analysis. Others have seen this swamp, and a current idea from the machine learning community, and going back at least to L. Breiman, is that we are not really looking for coefficients, t-tests on the coefficients, F-ratio tests on the regressions, confidence intervals on the predictions (for that might try some resampling ideas), importance of coefficients, causes, control variables, etc. but are just looking for a fit that can predict: To this end we put the data into at least two buckets, fit to one bucket and test in another one. Our main criterion is just that the model predicts well for the data we have. That is, all the data in all the buckets has all the same statistical assumptions, whatever the heck they are, and we are just fitting and then testing (confirming) on simple random samples of data all from some one big bucket. Yes, we still run into the issue of overfitting, fit well in the first bucket but flop terribly testing on the second bucket. Okay, a bit crude, uncouth, vulgar, primitive, ..., etc. but maybe useful in some cases -- apparently Breiman made it useful in some cases of medical data.

graycat 8y ago

For measurements, the social scientists have been considering what they call reliability and validity for a long time. So, they do consider the accuracy of measurements. In science we collect data then use the data. If the data has some measurement error, no big surprise. Instead just say that the model is in terms of the measured values with their errors instead of some much more accurate values that don't have. Or, if have more accurate values, then use them. Else don't feel guilty and beat up on self. See also my earlier post in this thread on cross tabulation.

tomrod 8y ago
Why isn't total least squares used to control for measurement error in regressors?
- graycat 8y ago
  By "total least squares" you mean to pick regression coefficients that minimize the squared errors between the observed values of the dependent variable and the predicted value from the regression? The predicted values are from a perpendicular projection, and we do get the Pythagorean theorem with total sum of squares = regression sum of squares + error sum of squares So, we minimize the error sum of squares. Does that really control on measurement error? Not in any simple way I can see. Or, if we have errors in the measurements of the independent variables, then we are facing one of the facts of life, we don't have the error-free, true values. Not good but usually not as bad as having your marriage fail or losing your pet dog or cat! Like the video clip of a Heifetz master class where a student tried to play the D flat minor scale and at the end Heifetz assured the student that they were still alive! Maybe what you are saying is that with enough mathematical assumptions, e.g., the famous heteroscedasticity with independent, identically distributed mean zero Gaussian errors, as the number of observations goes to infinity the errors wash out much as in the weak/strong laws of large numbers and we get to f'get about the errors -- maybe there is such a theorem, I should get out my copy of the old C. Radhakrishna Rao, Linear Statistical Inference and Its Applications: Second Edition, ISBN 0-471-70823-2, John Wiley and Sons, New York. and look or do some such derivations for myself. But, to what end if we don't believe the mathematical assumptions for the mathematical theorems, e.g., heteroscedasticity with independent, identically distributed mean zero Gaussian errors? Sorry, from 50,000 feet up, it seems to me that having control variables in regression is shaky stuff. And without some careful derivations, we should not be surprised at the effects of various errors. Also, the usual derivations of the math are in the context of just some one regression model where we make all those assumptions. Instead, given one dependent variable and 10 independent variables plus five more we believe are causes plus 10 more we want to use as controls, the dependent variable plus 25 more variables in all, last time I checked we were short on how to pick the 2^25 sets of independent variables and make sense of the different, maybe wildly different, coefficients we get. Here's a simple view: If we have 5 independent variables and they are all orthogonal, then we can get the regression coefficients one at a time just from 5 projections, covariances, inner products (all essentially the same things except how we scale things) and have those coefficient just the same for any of the 2^5 regression analyses. That is, if we have orthogonal independent variables U, V, W, X, and Y and dependent variable Z, then we can get the coefficients one at a time and be done -- have all the regression coefficients for all 2^5 regressions. Otherwise, without othogonality, we face some possibly tricky math derivations -- maybe they are in Rao's book, it's thick enough -- and are asking a bit too much from regression analysis. Others have seen this swamp, and a current idea from the machine learning community, and going back at least to L. Breiman, is that we are not really looking for coefficients, t-tests on the coefficients, F-ratio tests on the regressions, confidence intervals on the predictions (for that might try some resampling ideas), importance of coefficients, causes, control variables, etc. but are just looking for a fit that can predict: To this end we put the data into at least two buckets, fit to one bucket and test in another one. Our main criterion is just that the model predicts well for the data we have. That is, all the data in all the buckets has all the same statistical assumptions, whatever the heck they are, and we are just fitting and then testing (confirming) on simple random samples of data all from some one big bucket. Yes, we still run into the issue of overfitting, fit well in the first bucket but flop terribly testing on the second bucket. Okay, a bit crude, uncouth, vulgar, primitive, ..., etc. but maybe useful in some cases -- apparently Breiman made it useful in some cases of medical data.