3 ms·
I think the article is confusing three things. In Bayesian terminology, suppose we've found a posterior distribution for effect size. Then there are three thing
by datastoat 5y ago
I think the article is confusing three things. In Bayesian terminology, suppose we've found a posterior distribution for effect size. Then there are three things we might consider: (a) how tightly concentrated the posterior distribution is, (b) how much probability mass there is in a given part of the space, Prob(effect size >= thresh), (c) how we're choosing to incorporate other data.
To illustrate: suppose we've found a posterior distribution for effect size, and we've found that 95% of the posterior probability mass is >= 0. It might be a broad flat distribution (huge MAP effect size, massive uncertainty about effect size), or it might be a tight distribution (small MAP effect size, small uncertainty about effect size). There's nothing intrinsically wrong or suspicious about either of these two cases.
The headline of the linked article has the words "higher than is plausible". In other words, the premise of the article is that there is some other evidence which tells us what effect size is plausible. We could incorporate that evidence by adjusting our prior, which will obviously push the posterior distribution closer to what prior belief says it should be. Alternatively we could incorporate that evidence as a subsequent Bayesian conditioning step, i.e. treat the other evidence as further observations. The magic of Bayes's rule means that both of these approaches give the same answer. (The frequentist approach is pretty much like the latter -- it says "here's the result of my experiment, and I'll leave the reader to incorporate it with their own prior beliefs, in a meta-analysis." There's nothing intrinsically wrong with this.)
The article says "I find it frustrating when researchers don't think about their effect sizes." That's the wrong conclusion. The proper conclusion is (1) always report confidence intervals for your effect sizes, (2) don't pay any attention to a statement about effect sizes unless it comes with confidence / credible intervals -- and so the article's "painful email exchange" is missing the point, (3) when you report your results, make sure there's enough information for the reader to incorporate it with their own priors -- and the whole point of confidence intervals is to let us do this.