29 ms·

I drink coffee nearly every day; I am reading this in a coffeeshop. But, p = 0.028? Come on. If you tested for a few dozen things, you would expect to find a

I drink coffee nearly every day; I am reading this in a coffeeshop. But, p = 0.028? Come on. If you tested for a few dozen things, you would expect to find at least one that had a p-value like that. Was this pre-registered to look at only DNA damage, via this metric? Or did they look at 100 different things to find one with p=0.028? Of course, if they pre-registered and their theory was that it would reduce DNA damage as measured in this way, that would be a bit different, but from this article it's hard to say. Now, back to drinking my mocha latte.

longerthoughts 8y ago
It's a little annoying if they're cherry-picking but does it really impact the validity of the results? If anything, I'd think they'd have a greater incentive to skew results if they were specifically testing a theory about DNA damage.
- exolymph 8y ago
  Yes, p-hacking and related techniques do impact the validity of the results. More on that here: https://info.umkc.edu/drbanderson/p-hacking-and-the-problem-of-multiple-comparisons/ https://info.umkc.edu/drbanderson/p-hacking-and-the-problem-...
- darkpuma 8y ago
  > "It's a little annoying if they're cherry-picking but does it really impact the validity of the results?" Yes, that's a well known method of p-hacking.
- samatman 8y ago
  The Relevant Xkcd is actually an excellent guide to why this is a problem: https://xkcd.com/882/ https://xkcd.com/882/
- pergadad 8y ago
  "I want to prove that it's good for something, but I don't care what it's good for" is not really a good research question. Then you'll measure 100 things and only publish the one that makes your intervention look good and possibly ignore all the ones that make it look bad. It's also rather useless if you don't at least have some idea of the mechanism. Coffee has lots of different parts, so what part of it is useful - is it the caffeine? The tannins? ...? And how does it have this effect? Is it even the coffee or just that the coffee group took 5 minutes of rest to drink their brew and that is really the effect?
corsivo 8y ago
> Now, back to drinking my mocha latte. Another study claims that consuming coffee with milk-products inhibits our body's ability to absorb beneficial compounds, so it might not matter.
angel_j 8y ago
The first author has done similar studies with lower p values: https://www.ncbi.nlm.nih.gov/pubmed/24740588 https://www.ncbi.nlm.nih.gov/pubmed/24740588 https://www.ncbi.nlm.nih.gov/pubmed/26632023 https://www.ncbi.nlm.nih.gov/pubmed/26632023
bertil 8y ago
I'm going to assume that the article doesn't mention a Bonferroni correction and ignore it all.

longerthoughts 8y ago

It's a little annoying if they're cherry-picking but does it really impact the validity of the results? If anything, I'd think they'd have a greater incentive to skew results if they were specifically testing a theory about DNA damage.

exolymph 8y ago
Yes, p-hacking and related techniques do impact the validity of the results. More on that here: https://info.umkc.edu/drbanderson/p-hacking-and-the-problem-of-multiple-comparisons/ https://info.umkc.edu/drbanderson/p-hacking-and-the-problem-...
darkpuma 8y ago
> "It's a little annoying if they're cherry-picking but does it really impact the validity of the results?" Yes, that's a well known method of p-hacking.
samatman 8y ago
The Relevant Xkcd is actually an excellent guide to why this is a problem: https://xkcd.com/882/ https://xkcd.com/882/
pergadad 8y ago
"I want to prove that it's good for something, but I don't care what it's good for" is not really a good research question. Then you'll measure 100 things and only publish the one that makes your intervention look good and possibly ignore all the ones that make it look bad. It's also rather useless if you don't at least have some idea of the mechanism. Coffee has lots of different parts, so what part of it is useful - is it the caffeine? The tannins? ...? And how does it have this effect? Is it even the coffee or just that the coffee group took 5 minutes of rest to drink their brew and that is really the effect?

corsivo 8y ago

> Now, back to drinking my mocha latte. Another study claims that consuming coffee with milk-products inhibits our body's ability to absorb beneficial compounds, so it might not matter.

angel_j 8y ago

The first author has done similar studies with lower p values: https://www.ncbi.nlm.nih.gov/pubmed/24740588 https://www.ncbi.nlm.nih.gov/pubmed/24740588 https://www.ncbi.nlm.nih.gov/pubmed/26632023 https://www.ncbi.nlm.nih.gov/pubmed/26632023

bertil 8y ago

I'm going to assume that the article doesn't mention a Bonferroni correction and ignore it all.