Article summary: Many scientists (particularly social scientists) think they’re doing Popperian falsification of their hypotheses by using null hypothesis testi

Article summary: Many scientists (particularly social scientists) think they’re doing Popperian falsification of their hypotheses by using null hypothesis testing. That is, they posit a hypothesis that e.g. IQ is correlated with happiness and then setup a null hypothesis that says IQ Has zero correlation to happpiness and reject it if their data does not have zero correlation. Gelman argues this is confirmation-seeking not falsificationist since the scientist never posited a precise model of how IQ correlated with happiness that could be falsified. They only reject a straw man null hypothesis and then claim that this supports their hypothesis. He also notes that this has nothing to do with frequentist vs Bayesian statistics since null testing can be done using either approach. My own added view: Gelman never elaborates on what “making a hypothesis precise” means in this article but I think this is the key. A lot of social science is really about estimating model parameters and not positing new causal explanations (theories) of phenomena. If you hypothesize that IQ correlates with happiness, that is not a theory, one could say it’s not even a hypothesis, it’s really asking “how much does IQ correlate with happiness?” Anything can correlate with anything else, in principle so this “hypothesis” is not a new causal model of reality, it’s just a parameter estimation. So it doesn’t make sense to use falsificationist reasoning here since there’s no theory to falsify only a parameter to estimate. This is why null hypothesis significance testing (NHST) is so wrongheaded because 1) most social science is not about new causal models but about parameter estimation 2) when you do posit a new causal model you should falsify the predictions of your model not some straw man null hypothesis.