3 ms·
Let's ignore n for a moment, because it doesn't affect the chance of getting a more extreme result under the null hypothesis. And that's always a good thing to
by mumblemumble 7y ago
Let's ignore n for a moment, because it doesn't affect the chance of getting a more extreme result under the null hypothesis. And that's always a good thing to wonder about when you're looking at multiple comparisons.
They calculated 11 p-values, and got a significant result for two of them under α=0.1. We'd expect that that to happen about half the time if the null hypothesis is true.
(Realistically, they got a significant result on two numbuers that were both derived from the same measurement, which is less compelling than two unrelated ones. But I'm not sure how to model that, so, like any true armchair statistician, I'm going to handle it by ignoring it.)
I also did a hasty power calculation, and estimate that a study of this size could detect an effect as large as the biggest one they reported about half the time.
In summary: If there's no real effect, there's a 50/50 chance of getting a significant p-value out of this experimental design. If there is a real effect, still a 50/50 chance of getting a significant p-value.
Which hypothesis would I pick if I had to guess? Well, I'm sure this is also junk statistics, but the p-values sure look to me like they could have been drawn from a uniform distribution, which is what I would expect them to look like under the null hypothesis. Definitely not what I would expect them to look like under the alternative hypothesis, given that all these tests are trying to measure roughly the same thing.
- pliny 7y ago>They calculated 11 p-values, and got a significant result for two of them under α=0.1. We'd expect that that to happen about half the time if the null hypothesis is true. Not true: > Pairwise comparisons were used to explore significant effects, and were corrected for type 1 errors using Bonferroni adjustment
- mumblemumble 7y agoAha. That's what I get for jumping straight to the regression table. So I retract my attempt to suggest that this was a fishing expedition. That said, I'm still inclined to say that, even given the Bonferroni adjustment, the distribution of results looks more like what I'd expect under the null hypothesis. It's hard to imagine that all these tests they used are uncorrelated.
- gus_massa 7y agoI counted 18 p values (12 + (2+2) + 2), one with * (p<0.05) and three with # (p<0.10). In the sentence about the Bonferroni adjustment, I don't understand what "pairwise" mean. (I understand the word, I just don't understand how it is applied. They have 12 quite correlated test. In the correction, are they using m=2 or m=12? Are they correcting only the test that have two subtest?)
- Spikkzwnp 7y agoAssuming of course that p-values are not problematic [1]: "The reign of the p-value is over: what alternative analyses could we employ to fill the power vacuum?" https://royalsocietypublishing.org/doi/10.1098/rsbl.2019.0174 https://royalsocietypublishing.org/doi/10.1098/rsbl.2019.017... [2]: "A Litany of Problems With p-values" https://www.fharrell.com/post/pval-litany/ https://www.fharrell.com/post/pval-litany/