4 ms·
I agree that you shouldn't look only at effect sizes any more than you should look only at p-values. (What I would actually prefer you to do, where you can figu
by gjm11 4y ago
I agree that you shouldn't look only at effect sizes any more than you should look only at p-values. (What I would actually prefer you to do, where you can figure out a good way to do it, is to compute a posterior probability distribution and look at the whole distribution. Then you can look at its mean or median or mode or something to get a point estimate of effect size, you can look at how much of the distribution is > 0 to get something a bit like a p-value but arguably more useful, etc.
If anything I wrote appeared to be saying "just look at effect size, it's the only thing that matters" then that was an error on my part. I definitely didn't intend to say that.
But I was responding to an article saying "p<0.05 considered harmful" that never mentions effect sizes at all. I think that's enough to demonstrate that, in context, "it's bad to look at p-values and ignore effect sizes" is not in fact a straw man.
Incidentally, I am not convinced that the p-value as such is often a good way to assess how likely it is that your results are due to random chance. Suppose you see an effect size of 1 unit with p=0.05. OK, so there's a 5% chance of getting these results if the true effect size is zero. But you should also care what the chance is of getting these results if the true effect size is +0.1. (Maybe the distribution of errors is really weird and these results are very likely with a positive but much smaller effect size; then you have good evidence against the null hypothesis but very weak evidence for an effect size of the magnitude you measured.) In fact, what you really want to know is what the probability is for every possible effect size, because that gives you the likelihood ratios you can use to decide how likely you think any given effect size is after seeing the results. For sure, having the p-value is better than having nothing, but if you were going to pick one statistic to know in addition to (say) a point estimate of the effect size, it's not at all clear that the p-value is what you should choose.
- whimsicalism 4y agoFair enough, I don’t disagree with anything you just said. It would be cool if interfaces caught up so I could just draw a basic estimate of my prior and then see the posterior graph afterwards.
- nextos 4y agoWhat I have discovered after working in medicine for pretty long is that many biologists and MDs think p-values are a measure of effect sizes. Even a reviewer from Nature thought that, which is incredibly disturbing. p-values were created to facilitate rigorous inference with minimal computation, which was the norm during the first half of the 20th century. For those who work on a frequentist framework, inference should be done using a likelihood-based approach plus model selection, e.g. AIC. It's makes it much harder to lie.
- spekcular 4y agoAIC is an estimate of prediction error. I would caution against using it for selecting a model for the purpose of inference of e.g. population parameters from some dataset (without producing some additional justification that this is a sensible thing to do). Also, uncertainty quantification after data-dependent model selection can be tricky. Best practice (as I understand it) is to fix the model ahead of time, before seeing the data, if possible (as in a randomized controlled trial of a new medicine, etc.).
- ramblenode 4y agoAnd it is not uncommon that an intentionally bad model (low AIC) will be used for inference on a parameter when one wants to test the robustness of the parameter to covariates.