3 ms·

In Study 2 there we 236 subjects, which AFAIK is a pretty big sample size for these kinds of experiments. There error bars on the graph are still somewhat large

In Study 2 there we 236 subjects, which AFAIK is a pretty big sample size for these kinds of experiments. There error bars on the graph are still somewhat large, so there was a lot of variability, but assuming there is no bias in what data the experimenters chose to include, combining both studies, p < .02*.01 = .002. Furthermore, as the article notes, there was a prior study that also showed cognitive effects of Tylenol. While these effects could be spurious due to publication or selection bias or deliberate manipulation, it's not all that plausible that they happened by chance.

carbocation 13y ago
Just multiplying the P-values from different studies is invalid. You can intuit this by imagining 10000 studies, each of which had a P-value of 0.99. Seeing 10000 negative studies and no positive studies, you'd be convinced that there was no real effect, intuitively. But multiplying 0.99 10000 times yields a P-value of 2.2 x 10^(-44), a result of impressive significance. To combine studies, many people use inverse variance weighted meta-analysis, e.g., https://en.wikipedia.org/wiki/Inverse-variance_weighting https://en.wikipedia.org/wiki/Inverse-variance_weighting
- simonster 13y ago
  You're right. There is a mathematical explanation for this in addition to the intuitive one you provide: If the null hypothesis is true, the p-value of each test is uniformly distributed between 0 and 1, but the product of p-values will not be uniformly distributed between 0 and 1, so it is not a p-value. To combine p-values, what we really want is the CDF of the product of p1 and p2 in the distribution formed by the product of uniform random variables. Mathematica can compute this; the general formula is (p1p2)(1-log(p1p2)), and for the two p-values in question this yields p = 0.0019 (which is close to what I said initially, but only because I made a calculation error; .02*.01 = 0.0002). Thanks for pointing out the mistake in my reasoning! I learned something.

carbocation 13y ago

Just multiplying the P-values from different studies is invalid. You can intuit this by imagining 10000 studies, each of which had a P-value of 0.99. Seeing 10000 negative studies and no positive studies, you'd be convinced that there was no real effect, intuitively. But multiplying 0.99 10000 times yields a P-value of 2.2 x 10^(-44), a result of impressive significance. To combine studies, many people use inverse variance weighted meta-analysis, e.g., https://en.wikipedia.org/wiki/Inverse-variance_weighting https://en.wikipedia.org/wiki/Inverse-variance_weighting

simonster 13y ago
You're right. There is a mathematical explanation for this in addition to the intuitive one you provide: If the null hypothesis is true, the p-value of each test is uniformly distributed between 0 and 1, but the product of p-values will not be uniformly distributed between 0 and 1, so it is not a p-value. To combine p-values, what we really want is the CDF of the product of p1 and p2 in the distribution formed by the product of uniform random variables. Mathematica can compute this; the general formula is (p1p2)(1-log(p1p2)), and for the two p-values in question this yields p = 0.0019 (which is close to what I said initially, but only because I made a calculation error; .02*.01 = 0.0002). Thanks for pointing out the mistake in my reasoning! I learned something.