4 ms·

> The article is all about why "0.05" might be a bad value to choose. No. It's more about why choosing a value to serve as the default choice is a bad idea in

> The article is all about why "0.05" might be a bad value to choose. No. It's more about why choosing a value to serve as the default choice is a bad idea in the first place. The specific value chosen as the default itself (i.e., 0.05 in this case) is irrelevant. The idea is that the value you choose should reflect some prior knowledge about the problem. Therefore choosing 5% all the time would be somewhat analogous to a bayesian choosing a specific default gaussian as their prior: it defeats the point of choosing a prior, and it's actively harmful if it sabotages what you should actually be using as a prior, because it's not even uninformative, it's a highly opinionated prior instead. As for points 1 to 3, technically I agree, but there's a lot of misdirection involved. The point on effect sizes is true (and indeed something many people get wrong), but it is contrived to make effect sizes more useful than p-values. In which case, the obvious answer is, you should be choosing an example where reporting a p-value is more important than an effect size. One way to look at a p-value is as a ranking, which would be useful for comparing between effect sizes of incomparable units. Is a student with a 19/20 grade from a european school better than an american student with a 4.6 GPA? Reducing the compatibility scores to rankings, can help you compare these two effect sizes immediately. Prior probability, similarly. "If" you interpret things as you did, then yes, p-values suck. But you're not supposed to. What the p-value tells you is "the data and model are 'this much' compatible". It's up to you then to say "and therefore" vs "must have been an atypical sample". In other words, there is still space for a prior here. And in theory, you are free to repurpose the compatibility score of a p-value to introduce this prior directly (though nobody does this in practice). Regarding p=0.05 vs p=0.001 not mattering; of course they do. But only if they're used as compatibility rankings as opposed to decision thresholds. If you compare two models, and one has p=0.05 and the other has p=0.001, this tells you two things: a) they are both very incompatible with the data, b) the latter is a lot more incompatible than the former. The problem is not that people use p-values, the problem that people abuse them, to make decisions that are not necessarily crisply supported by the p-values used to push them. But this could be said of any metric. I have actively seen people propose "deciding in favour of a model if the Bayes Factor is > 3". This is exactly the same faulty logic, and the fact that BF is somehow "bayesian" won't protect subsequent researchers who use this heuristic from entering a new reproducibility crisis.

gjm11 4y ago
I agree that the article is somewhat about the idea that having a Standard Default p-value Threshold is unwise, but I think it's mostly suggesting that in its particular context it's generally better to use a larger p-value. "Defaults matter", the author says (not "Defaults are a trap"). "Do we need that much risk aversion?" (not "Sometimes we need more risk aversion, sometimes less"). "If we were starting over ... I doubt that's what we would pick" (not "If we were starting over, I think we'd have a process that begins by considering what p-value threshold would be appropriate"). At the end, he considers situations where the criterion used is strict and "the company progresses too slowly" (but not ones where it's lax and "the company moves too fast and breaks too many things" or "the company thrashes about unstably") and "Why not just admit you want something akin to p=0.25 in the first place" (not "... something akin to p=0.25 some of the time, and something akin to p=0.001 some of the time?") and "I believe that nobody wants to write down a large p-value threshold, because it feels unscientific" (and nothing about reasons why people might be reluctant to write down an unusually small p-value threshold). I agree, in case it needs to be said, that if you are doing hypothesis testing then you should adapt your choice of thresholds to your (explicit or implicit) prior. That was approximately the second of the three points I made. "You should be choosing an example where reporting a p-value is more important than an effect size." Why should I be doing that? I think that almost always both are important (more precisely, I think that almost always thinking of what you've learned from an experiment into a p-value and a point-estimate effect size is suboptimal, but you want the information both of those things are trying to tell you). And if I've correctly understood what you say about rankings, I completely disagree with it; almost none of the time is the p-value what you should be ranking things on. Again, consider those two medications in my hypothetical example: I do not believe any reasonable person would think that the right way to rank them relative to one another matches the p-values. (For comparing two individual students? Yeah, maybe, kinda. But approximately 0% of cases where people compute p-values are analogous to that.) I agree that picking any single metric and applying brainless black-and-white rules using it is liable to get you in trouble, whatever that metric is. But some metrics are better than others, and a fixed p-value threshold (1) has been a widespread common practice and (2) is for most purposes a really bad idea.
- tpoacher 4y ago

gjm11 4y ago

I agree that the article is somewhat about the idea that having a Standard Default p-value Threshold is unwise, but I think it's mostly suggesting that in its particular context it's generally better to use a larger p-value. "Defaults matter", the author says (not "Defaults are a trap"). "Do we need that much risk aversion?" (not "Sometimes we need more risk aversion, sometimes less"). "If we were starting over ... I doubt that's what we would pick" (not "If we were starting over, I think we'd have a process that begins by considering what p-value threshold would be appropriate"). At the end, he considers situations where the criterion used is strict and "the company progresses too slowly" (but not ones where it's lax and "the company moves too fast and breaks too many things" or "the company thrashes about unstably") and "Why not just admit you want something akin to p=0.25 in the first place" (not "... something akin to p=0.25 some of the time, and something akin to p=0.001 some of the time?") and "I believe that nobody wants to write down a large p-value threshold, because it feels unscientific" (and nothing about reasons why people might be reluctant to write down an unusually small p-value threshold). I agree, in case it needs to be said, that if you are doing hypothesis testing then you should adapt your choice of thresholds to your (explicit or implicit) prior. That was approximately the second of the three points I made. "You should be choosing an example where reporting a p-value is more important than an effect size." Why should I be doing that? I think that almost always both are important (more precisely, I think that almost always thinking of what you've learned from an experiment into a p-value and a point-estimate effect size is suboptimal, but you want the information both of those things are trying to tell you). And if I've correctly understood what you say about rankings, I completely disagree with it; almost none of the time is the p-value what you should be ranking things on. Again, consider those two medications in my hypothetical example: I do not believe any reasonable person would think that the right way to rank them relative to one another matches the p-values. (For comparing two individual students? Yeah, maybe, kinda. But approximately 0% of cases where people compute p-values are analogous to that.) I agree that picking any single metric and applying brainless black-and-white rules using it is liable to get you in trouble, whatever that metric is. But some metrics are better than others, and a fixed p-value threshold (1) has been a widespread common practice and (2) is for most purposes a really bad idea.

tpoacher 4y ago