11 ms·
Not Frequentist Enough
- hammock 4y agoA lot of words to say something that essentially boils down to, "Make sure your findings replicate."
- yellowstuff 4y agoMost people care about their findings replicating. The hard part is knowing what you need to do to improve your chances.
- hackerlight 4y agoMore like "make sure your test power is what you think it is". There will still be results that fail to replicate by virtue of the rejection of the null hypothesis by chance, but that should only happen 1 in 20 times at an alpha of 0.05. With all the bad practices that alter test power, such as p-hacking and the file drawer effect, that 1 in 20 blows up to 1 in 2.
- hackandthink 4y agoGelman makes a distinction between p-hacking and choosing the test based on your data. p-hacking is confirming your preferred hypothesis - just try again. the latter let's you write a paper. (test power is different: if there's really an effect, would you detect it)
- hackerlight 4y agoOops, I meant to say test size, not test power. A test size of 0.05 is only really 0.05 if the entire process you used was free from p-hacking. And as a reader of that paper that process includes the file drawer effect. All these things have an impact on distorting the test size away from what it is advertised to be.
- jldugger 4y agoIn principle the robustness approach (redoing the analysis with some different parameters) tries to show good faith. This matters because observational studies have no meaningful replication, no second collection of data. In practice it just means practitioners gravitate towards methods more likely to show false positives. Like, this discontinuity at zero approach seems geared to show large effects from noise. When the linear regression across the dataset is a flat line, splitting at the discontinuity pretty much mathematically requires _any_ slope have an opposite slope on the other side.
- civilized 4y agoAs if that was a thing you could just do? "Oh, forgot to make sure my findings replicated. I'll be sure to check that box next time, easy enough!"
- hackandthink 4y agoA famous Bayesian arguing for frequentist statistics? Gelman tries to steal the concept "Frequentism" from simple minded frequentist statisticians. His argument seems to be: Simple minded frequentists statisticians perform a statistical procedure once - they do not think about performing the procedure many times. They fall into this trap (from Gelman's paper): "3. Researcher degrees of freedom without fishing: computing a single test based on the data, but in an environment where a different test would have been performed given different data"
- rwilson4 4y agoGelman is one of the few self-proclaimed Bayesians who doesn't seem to outright hate frequentist approaches. They're complementary approaches. Bayesian methods are great for combining different sources of information. Frequentist methods are great for validating that a method is working well. (For example, Gelman often recommends running simulations to see if models give sensible predictions, but that is itself a pretty frequentist thing to do.) Frequentism is mostly about how to evaluate a methodology. It's pretty agnostic about what that methodology is. Bayesian methods are about combining different sources of information. In a situation where you only have one source of information, Bayesian and Frequentist methods usually give the same answer. People say you might as well always use Bayesian methods then. But no matter what, you should always try to validate or poke holes in your model, and Frequentist techniques are great for that. So it's best to be familiar with both!
- hackandthink 4y agoyes https://stats.stackexchange.com/questions/115157/what-are-posterior-predictive-checks-and-what-makes-them-useful https://stats.stackexchange.com/questions/115157/what-are-po...
- kgwgk 4y ago> running simulations to see if models give sensible predictions, but that is itself a pretty frequentist thing to do Is looking at probability distributions “a pretty frequentist thing to do”? Even when those models and simulations include _prior_ probability distributions? Sure, one can (re)define frequentist to include Bayesian models - as Gelman seems to want to do in that post. I just don’t see how this helps to clarify anything.
- youainti 4y agoMy understanding of the post is that given a small actual effect size, for a fixed experiment, you are more likely to get a significant p-value on a "large" measured effect size.
- konschubert 4y agoWhen studying statistics, I never understood the apparent conflict between frequentism and bayesianism. To me, they seemed like complementary tools, each with their own strengths and weaknesses. You can use one or the other depending on what your goal is. Do you want to figure out how much you should believe something? Or are you trying to figure out how compatible a hypothesis is with reality? In the first case, go bayesian, in the second, frequentist.
- kgwgk 4y agoNothing prevents you from "going Bayesian" in figuring out how compatible a hypothesis is with reality. Bayesians have no issues with probabilities representing frequencies even though frequentists cannot understand probabilities representing uncertainty.
- jxy 4y agoSeriously? Frequentists simply build infinite amount of parallel Universes and ask what percentage of the Universes It occurs with certainty.
- layer8 4y agoThat sounds like a pretty sensible approach if you believe in Many-Worlds.
- konschubert 4y agoIt has nothing to do with that. Even if there is a single world, it is enough to make the thought experiment.
- layer8 4y agoMy point is, in Many-Words the parallel universes aren’t just a thought experiment, they are the actual real future of your current self. The computed probabilities do not describe potentialities then, but actualities.
- tpoacher 4y agoThere's nothing stopping the definition of a p-value from incorporating bayesian priors. There. I said it.
- mgraczyk 4y agoP values already incorporate assumptions about the distribution under the null hypothesis, so it's not such a stretch to report bayesian statistics
- whatshisface 4y agoThere's nothing stopping the Bayesian posterior from being the degree to which you can reject the null hypothesis except for its definition. There's nothing except their definition stopping real numbers from including i, too.
- tpoacher 4y agoNo, that's redefining. The p-value's ACTUAL definition, as accepted today in the statistical sense, can incorporate bayesian priors easily by making the model-data compatibility score upon which it acts compatible with Bayesian priors. Such as, e.g., a posterior probability.