4 ms·

This is actually a much better explanation of the problem than the main article. The main article kept saying that p values were not meant to be definitive, bu

This is actually a much better explanation of the problem than the main article. The main article kept saying that p values were not meant to be definitive, but without explaining what was wrong with them. At least, not very clearly. This comment is a much clearer explanation -- imho -- as to what goes wrong in the industry. I also think it is right to focus the attention away from a particular statistical measure.

kerkeslager 7y ago
It seems clear to me that the author just didn't understand the problem with P values. It's part of a larger problem of science journalism being done by journalists without scientific backgrounds. It's not their fault--even if you have natural ability and interest in both communication and discovery, it's hard to get an education in both. I have the opposite problem: my abilities lie more in the statistics/science than the communication--I suspect the only reason that this explanation is being received so well is that I happened across an effective example by pure luck. The probability of me communicating effectively is probably P < .5. ;)
- kqr 7y ago
  Let me check if I get it right: can this be said to be a case of incorrectly aggregating experiments? We're in a sense taking min(p) over all p-values, or any(significant) over all results, when we should use an aggregation method that takes into account the total number of studies aggregated?
  - kerkeslager 7y ago
    Yes. For example, if you dig, you'll find that a lot of the evidence for a correlation between telomere shortening (associated with aging) and processed meat consumption comes from a 2008 study in the American Journal of Clinical Nutrition[1]. They found a P=.006 correlation between processed meat consumption and telomere shortening. But when you look into the study further, they collected data on 47 different food groups. Let's calculate how unlikely a P=.006 event is if you try 47 times to find it: A. If you have a probability of something happening once P_1, then the probability of that happening x times in a row, P_x, is P_x = P_1^x. For example, the probability of flipping a coin once and getting heads is P_1=.5, so the probability of that happening twice in a row is P_2=.5^2=.25, which you can verify by enumerating the possibilities (1 in 4 of the following combinations is 2 heads in a row: HH, HT, TH, TT). The probability of flipping 3 heads in a row is P_3=.5^3=.125 (only 1 of these 8 possibilities is all heads: HHH, HHT, HTH, HTT, THH, THT, TTH, TTT).[2] B. The chances of a P=.006 event NOT happening P' are P'=1-P=.994 (the rule of 1). C. Combining facts A and B, the chances of a P_1=.006 event NOT happening (P'_1) 47 times in a row P'_47' is P'_47=.994^47=.754 (this is rounded to the 3 significant digits). D. Applying the rule of 1 again, the probability of finding a P=.006 result if you try 47 times is P'_47' = 1-P'_47 = .246. So basically, the actual confidence value on that study is P=.246: there's about a 1 in 4 chance they would find a P=.006 result for one of the 47 food groups tested if the null hypothesis is true. The null hypothesis in this case being "diet doesn't affect telomere length". The paper doesn't seem to list their hypothesis, but they say things like "all others were P>.05", which indicates that they were hypothesizing a P<.05 result. Doing the same math again for the probability of a P=.05 result occurring if you try 47 times, P'_47' = 1 - (1 - .05)^47 = 0.91025516807. That's over a 90% chance! In other words, by doing the test on 47 food groups, they nearly guaranteed that they would find a result within their definition of significance. This is very bad science, and a study with this bad of a statistical design never should have been funded in the first place. This study has not been replicated as far as I know, but is often cited in pop science[3][4][5]. [1] https://academic.oup.com/ajcn/article/88/5/1405/4649028 https://academic.oup.com/ajcn/article/88/5/1405/4649028 [2] Note that this rule only applies to independent probabilities; the result of coin flip does not affect the result of the next coin flip, and the result of telomere correlation to one food group does not affect the result of telomere correlation to another food group. This rule won't work when applied to dependent probabilities, where the earlier tries affects later tries. For example if you draw an ace from a deck of cards (P=4/52) and don't place the ace back in the deck, the probability of drawing another ace is lower (P=3/51). [3] https://www.livestrong.com/article/506649-foods-that-boost-telomeres-telomerase/ https://www.livestrong.com/article/506649-foods-that-boost-t... [4] https://resources.teloyears.com/diet-and-telomeres/processed-meat-not-unprocessed-red-meat-inversely-associated-leukocyte-telomere-length-strong-heart-family-study https://resources.teloyears.com/diet-and-telomeres/processed... [5] https://siimland.com/how-to-increase-telomere-length/ https://siimland.com/how-to-increase-telomere-length/

kerkeslager 7y ago

It seems clear to me that the author just didn't understand the problem with P values. It's part of a larger problem of science journalism being done by journalists without scientific backgrounds. It's not their fault--even if you have natural ability and interest in both communication and discovery, it's hard to get an education in both. I have the opposite problem: my abilities lie more in the statistics/science than the communication--I suspect the only reason that this explanation is being received so well is that I happened across an effective example by pure luck. The probability of me communicating effectively is probably P < .5. ;)

kqr 7y ago
Let me check if I get it right: can this be said to be a case of incorrectly aggregating experiments? We're in a sense taking min(p) over all p-values, or any(significant) over all results, when we should use an aggregation method that takes into account the total number of studies aggregated?
- kerkeslager 7y ago
  Yes. For example, if you dig, you'll find that a lot of the evidence for a correlation between telomere shortening (associated with aging) and processed meat consumption comes from a 2008 study in the American Journal of Clinical Nutrition[1]. They found a P=.006 correlation between processed meat consumption and telomere shortening. But when you look into the study further, they collected data on 47 different food groups. Let's calculate how unlikely a P=.006 event is if you try 47 times to find it: A. If you have a probability of something happening once P_1, then the probability of that happening x times in a row, P_x, is P_x = P_1^x. For example, the probability of flipping a coin once and getting heads is P_1=.5, so the probability of that happening twice in a row is P_2=.5^2=.25, which you can verify by enumerating the possibilities (1 in 4 of the following combinations is 2 heads in a row: HH, HT, TH, TT). The probability of flipping 3 heads in a row is P_3=.5^3=.125 (only 1 of these 8 possibilities is all heads: HHH, HHT, HTH, HTT, THH, THT, TTH, TTT).[2] B. The chances of a P=.006 event NOT happening P' are P'=1-P=.994 (the rule of 1). C. Combining facts A and B, the chances of a P_1=.006 event NOT happening (P'_1) 47 times in a row P'_47' is P'_47=.994^47=.754 (this is rounded to the 3 significant digits). D. Applying the rule of 1 again, the probability of finding a P=.006 result if you try 47 times is P'_47' = 1-P'_47 = .246. So basically, the actual confidence value on that study is P=.246: there's about a 1 in 4 chance they would find a P=.006 result for one of the 47 food groups tested if the null hypothesis is true. The null hypothesis in this case being "diet doesn't affect telomere length". The paper doesn't seem to list their hypothesis, but they say things like "all others were P>.05", which indicates that they were hypothesizing a P<.05 result. Doing the same math again for the probability of a P=.05 result occurring if you try 47 times, P'_47' = 1 - (1 - .05)^47 = 0.91025516807. That's over a 90% chance! In other words, by doing the test on 47 food groups, they nearly guaranteed that they would find a result within their definition of significance. This is very bad science, and a study with this bad of a statistical design never should have been funded in the first place. This study has not been replicated as far as I know, but is often cited in pop science[3][4][5]. [1] https://academic.oup.com/ajcn/article/88/5/1405/4649028 https://academic.oup.com/ajcn/article/88/5/1405/4649028 [2] Note that this rule only applies to independent probabilities; the result of coin flip does not affect the result of the next coin flip, and the result of telomere correlation to one food group does not affect the result of telomere correlation to another food group. This rule won't work when applied to dependent probabilities, where the earlier tries affects later tries. For example if you draw an ace from a deck of cards (P=4/52) and don't place the ace back in the deck, the probability of drawing another ace is lower (P=3/51). [3] https://www.livestrong.com/article/506649-foods-that-boost-telomeres-telomerase/ https://www.livestrong.com/article/506649-foods-that-boost-t... [4] https://resources.teloyears.com/diet-and-telomeres/processed-meat-not-unprocessed-red-meat-inversely-associated-leukocyte-telomere-length-strong-heart-family-study https://resources.teloyears.com/diet-and-telomeres/processed... [5] https://siimland.com/how-to-increase-telomere-length/ https://siimland.com/how-to-increase-telomere-length/