4 ms·

I am extremely skeptical about the statistical analysis here. It sounds like they tested nine food types, eight beverages and then even tried out different typ

I am extremely skeptical about the statistical analysis here. It sounds like they tested nine food types, eight beverages and then even tried out different types of comparisons for each food/beverage. And the best p-value they observed was p=0.02: "Among nine food types (cereal, fruits, vegetables, diary, red meat, poultry, fish, sweets and salty snacks) and eight beverages (juices, coffee, tea, mineral water, alcoholic- and sweetened carbonated beverages) only intake of red meat was related to T/S ratio. Individuals with increased consumption of red meat have had higher T/S ratio and the strongest significant differences were observed between consumer groups: “never” and “1–2 daily” (p = 0.02)." Nowhere in their methods do they mention adjusting for multiple testing. If we do that using the standard Bonferroni approach, the p-value would have to be multiplied by a factor of at least 17 x 2 to account for the number of options that they tried. So the adjusted p-value would be 0.02 x 17 x 2 = 0.68 and thus far from statistical significance. In other words, this seems to be a real-life example of this: https://xkcd.com/882/ https://xkcd.com/882/ Maybe I missed something, but if this is really the case then it's embarrassing that this made it through peer review.

jbmorgado 9y ago
> "And the best p-value they observed was p=0.02" A p-value of 0.02 is quite good in physics and absolutely outstanding in medical sciences or biology.
- BeetleB 9y ago
  You didn't address anything the GP said. The more variables you test for, the more likely you'll get a low p-value by pure chance.
highd 9y ago
Great catch. This is a super important factor. The real concern in science is how easy it can be to hide the fact that so many possibilities were tested, precluding this observation. As another example, there are GWAS studies where people take surveys and medical histories of large groups of people and then sequence their DNA to look for correlations across all possibilities. Those studies only report results with p values less than 1e-4 or 1e-5.
kerkeslager 9y ago
I'm curious about multiple testing. Could you explain (our rather, link something that does)?
- veddox 9y ago
  Put simply: if you test a random event often enough, you're going to get lucky eventually. In statistics, the question everybody asks is whether or not a given result is "significant". To answer this question, one calculates a p-value, which gives you the probability that your data are purely random (i.e. not due to any causal factor). By convention, if p < 0.05, one calls this result "significant". There are various tests to calculate this p-value, depending on the type of data you have available. One of the most common types are t-tests, which are used when you wish to compare two samples. (For example, you might want to compare the number of butterfly species on mountain meadows with the number of species in vineyards.) The t-test takes various factors into account, including the mean and the variance of your datasets. This makes it a pretty good tool for comparison, much better than looking merely at, say, the arithmetic mean. The problem is that occasionally, the t-test is going to give you a false positive. That means you end up thinking that two things are causally linked that in reality are completely independent. The probability of such a wrong result rises with the number of tests you perform in a given study. That is why such studies need to adjust for multiple testing. Often this is done with the Bonferroni method, which basically means multiplying your p-value by a/n, where a is your level of significance (0.05) and n is the number of tests you perform. Hope that made sense :-)
  - kerkeslager 9y ago
    The concept I was missing was that testing two different random variables at the same time (do meat or vegetables affect telomere length in this group) has the same probability of yielding a false positive for one of the variables as testing the same variable twice (does meat affect telomere length in this group, does meat affect telomere length in that group) has of yielding a false positive for one of the tests.
- asgraham 9y ago
  A quick recap of the basics (multiple testing appears at the X): Precisely, what a p-value means is: "Assuming the null hypothesis is true, the probability that I would get this test statistic from the null distribution is p," where p is your p-value. When you call a p-value "significant" that means you're rejecting some null hypothesis. That null hypothesis entails some null distribution, and what you're saying is: "It is unlikely the test statistic I've calculated comes from that null distribution." Somehow 0.05 has become the p-value of significance, which means we say a test rejects the null hypothesis if the test statistic we calculated had less than a 1 in 20 chance of being drawn from the null distribution. The t-test is the test the linked paper used. Essentially they compare list of telomere lengths sorted by different habits to see if they're drawn from the same (normal) distribution. For example, the significant comparison was the list of telomere lengths of all subjects who didn't eat any red meat vs. the list of telomere lengths of all subjects who ate "1-2 [red meat] daily". Based on two lists of numbers, you calculate the t-statistic. Under the null hypothesis that the two lists were drawn from the same normal distribution, the probability distribution of the t-statistics is known. If you calculated a t-statistic that falls on the tail of that distribution (meaning, a t-statistic large enough to be very unlikely) then you say: "Aha! The null hypothesis is probably wrong. The two lists come from different distributions. The red meat eaters have significantly longer telomeres." X Now let's use a really specific example to show why that doesn't scale. Suppose you generate two lists of normally-distributed numbers with the same mean and variance What you're really testing is whether they're drawn from the same distribution which, in this case, we know they're not (we drew them from the same distribution ourselves). Now do that 100 times. If you want (and if you've already downloaded scipy) you can follow along and verify what I'm saying: from numpy.random import randn from scipy.stats import ttest_ind N_tests = 100; N_samples = 500; sig_level = 0.05 results = [ttest_ind(randn(N_samples),randn(N_samples)) for i in range(N_tests)] is_significant = [result[1] < sig_level for result in results] print("The proportion of results that are significant is: ", sum(is_significant) / N_tests) If you run this a few times, you'll see you get ~0.05 every time. If you increase the number of tests (the number of lists you compare), you'll be even more likely to get close to 0.05. Now, the natural response to this is: "But that's absurd! You've just compared a bunch of random lists! That has nothing to do with a scientific inquiry looking for meaning and relation within a single dataset!" And that's almost true. And there are absolutely well-designed experiments where there are 100 perfectly reasonable hypotheses to test. What this tells you, is that in that perfectly well-designed experiment you will get about 5 significant results (p < 0.05). Hell, you'll probably even get one "highly significant" result (p < 0.01). By definition. That means in that case it's meaningless to publish your results at the 0.05 significance level. You correct for that with something like Bonferroni correction (which is probably far too conservative) or Tukey's range test [1], both of which say roughly: Given I tried this many tests, how likely was I to get this test statistic under the null hypothesis? One closing observation: This holds across experiments and papers. What I mean is, if every year you run 20 t-tests, at the 0.05 significance level you can expect to get 1 spurious significant result every year. I suspect most researchers run more than 20 t-tests per year. This is why a single "significant" result mined from a dataset should never be the basis of an academic paper. That's why experimental design and replicability are so important. [1] I may have failed to respond to your question for the most part, but here's a link! https://en.wikipedia.org/wiki/Tukey%27s_range_test https://en.wikipedia.org/wiki/Tukey%27s_range_test
captainmuon 9y ago
That would be really sad. Every bachelor student in particle physics learns about the "look-elsewhere effect", but in a way, our field is about nothing, we do it out of intellectual curiosity. But in medical studies, where it is potentially about lifes, you see sometimes so sloppy statistical analysis. Sometimes I get the impression that it is OK, even fashionable, in certain fields to know nothing about statistics. Like in school when it was not cool to be good in math. (Source: anecdotes from fellow medical and non-medical researchers) Thankfully, there are also many good medical studies, and I know there are many researchers in medicine who are much better statisticians then I am. I haven't looked into this concrete study, so I can't comment on it. But I'm a bit worried about the field in general...
lordnacho 9y ago
Well I won't add anything on whether this particular study is flawed, though I recognise the same doubts as you. What I can say is it seems like an awfully tiresome drudge to replicate someone's statistical analysis. Where's the raw data? How did you clean it? What tests did you run? With what parameters? And what library? It's different in R / Numpy, oh no, another week of sorting the haystack. It would be easier if there were a GitHub for each study where you could just pull the exact same analysis that the authors did. Also, I get the feeling most experts see themselves as experts in their particular field. Not surprising, given how people start by studying some subject and then get more and more degrees in it. But actually, the more I read papers from various fields, the more it seems statistics is the critical skill. Domain knowledge is necessary, but it means very little without stats. Now it would be nice if researchers in every field were also experts in statistics, but I happen to have a friend whose post-doc is to assist biomedical researchers with their stats. So he's a biochem domain expert, but also a stats expert, and the other researchers come to his little group to make sure things are statistically well reasoned.

jbmorgado 9y ago

> "And the best p-value they observed was p=0.02" A p-value of 0.02 is quite good in physics and absolutely outstanding in medical sciences or biology.

BeetleB 9y ago
You didn't address anything the GP said. The more variables you test for, the more likely you'll get a low p-value by pure chance.

highd 9y ago

Great catch. This is a super important factor. The real concern in science is how easy it can be to hide the fact that so many possibilities were tested, precluding this observation. As another example, there are GWAS studies where people take surveys and medical histories of large groups of people and then sequence their DNA to look for correlations across all possibilities. Those studies only report results with p values less than 1e-4 or 1e-5.

kerkeslager 9y ago

I'm curious about multiple testing. Could you explain (our rather, link something that does)?

veddox 9y ago
Put simply: if you test a random event often enough, you're going to get lucky eventually. In statistics, the question everybody asks is whether or not a given result is "significant". To answer this question, one calculates a p-value, which gives you the probability that your data are purely random (i.e. not due to any causal factor). By convention, if p < 0.05, one calls this result "significant". There are various tests to calculate this p-value, depending on the type of data you have available. One of the most common types are t-tests, which are used when you wish to compare two samples. (For example, you might want to compare the number of butterfly species on mountain meadows with the number of species in vineyards.) The t-test takes various factors into account, including the mean and the variance of your datasets. This makes it a pretty good tool for comparison, much better than looking merely at, say, the arithmetic mean. The problem is that occasionally, the t-test is going to give you a false positive. That means you end up thinking that two things are causally linked that in reality are completely independent. The probability of such a wrong result rises with the number of tests you perform in a given study. That is why such studies need to adjust for multiple testing. Often this is done with the Bonferroni method, which basically means multiplying your p-value by a/n, where a is your level of significance (0.05) and n is the number of tests you perform. Hope that made sense :-)
- kerkeslager 9y ago
  The concept I was missing was that testing two different random variables at the same time (do meat or vegetables affect telomere length in this group) has the same probability of yielding a false positive for one of the variables as testing the same variable twice (does meat affect telomere length in this group, does meat affect telomere length in that group) has of yielding a false positive for one of the tests.
asgraham 9y ago
A quick recap of the basics (multiple testing appears at the X): Precisely, what a p-value means is: "Assuming the null hypothesis is true, the probability that I would get this test statistic from the null distribution is p," where p is your p-value. When you call a p-value "significant" that means you're rejecting some null hypothesis. That null hypothesis entails some null distribution, and what you're saying is: "It is unlikely the test statistic I've calculated comes from that null distribution." Somehow 0.05 has become the p-value of significance, which means we say a test rejects the null hypothesis if the test statistic we calculated had less than a 1 in 20 chance of being drawn from the null distribution. The t-test is the test the linked paper used. Essentially they compare list of telomere lengths sorted by different habits to see if they're drawn from the same (normal) distribution. For example, the significant comparison was the list of telomere lengths of all subjects who didn't eat any red meat vs. the list of telomere lengths of all subjects who ate "1-2 [red meat] daily". Based on two lists of numbers, you calculate the t-statistic. Under the null hypothesis that the two lists were drawn from the same normal distribution, the probability distribution of the t-statistics is known. If you calculated a t-statistic that falls on the tail of that distribution (meaning, a t-statistic large enough to be very unlikely) then you say: "Aha! The null hypothesis is probably wrong. The two lists come from different distributions. The red meat eaters have significantly longer telomeres." X Now let's use a really specific example to show why that doesn't scale. Suppose you generate two lists of normally-distributed numbers with the same mean and variance What you're really testing is whether they're drawn from the same distribution which, in this case, we know they're not (we drew them from the same distribution ourselves). Now do that 100 times. If you want (and if you've already downloaded scipy) you can follow along and verify what I'm saying: from numpy.random import randn from scipy.stats import ttest_ind N_tests = 100; N_samples = 500; sig_level = 0.05 results = [ttest_ind(randn(N_samples),randn(N_samples)) for i in range(N_tests)] is_significant = [result[1] < sig_level for result in results] print("The proportion of results that are significant is: ", sum(is_significant) / N_tests) If you run this a few times, you'll see you get ~0.05 every time. If you increase the number of tests (the number of lists you compare), you'll be even more likely to get close to 0.05. Now, the natural response to this is: "But that's absurd! You've just compared a bunch of random lists! That has nothing to do with a scientific inquiry looking for meaning and relation within a single dataset!" And that's almost true. And there are absolutely well-designed experiments where there are 100 perfectly reasonable hypotheses to test. What this tells you, is that in that perfectly well-designed experiment you will get about 5 significant results (p < 0.05). Hell, you'll probably even get one "highly significant" result (p < 0.01). By definition. That means in that case it's meaningless to publish your results at the 0.05 significance level. You correct for that with something like Bonferroni correction (which is probably far too conservative) or Tukey's range test [1], both of which say roughly: Given I tried this many tests, how likely was I to get this test statistic under the null hypothesis? One closing observation: This holds across experiments and papers. What I mean is, if every year you run 20 t-tests, at the 0.05 significance level you can expect to get 1 spurious significant result every year. I suspect most researchers run more than 20 t-tests per year. This is why a single "significant" result mined from a dataset should never be the basis of an academic paper. That's why experimental design and replicability are so important. [1] I may have failed to respond to your question for the most part, but here's a link! https://en.wikipedia.org/wiki/Tukey%27s_range_test https://en.wikipedia.org/wiki/Tukey%27s_range_test

captainmuon 9y ago

That would be really sad. Every bachelor student in particle physics learns about the "look-elsewhere effect", but in a way, our field is about nothing, we do it out of intellectual curiosity. But in medical studies, where it is potentially about lifes, you see sometimes so sloppy statistical analysis. Sometimes I get the impression that it is OK, even fashionable, in certain fields to know nothing about statistics. Like in school when it was not cool to be good in math. (Source: anecdotes from fellow medical and non-medical researchers) Thankfully, there are also many good medical studies, and I know there are many researchers in medicine who are much better statisticians then I am. I haven't looked into this concrete study, so I can't comment on it. But I'm a bit worried about the field in general...

lordnacho 9y ago

Well I won't add anything on whether this particular study is flawed, though I recognise the same doubts as you. What I can say is it seems like an awfully tiresome drudge to replicate someone's statistical analysis. Where's the raw data? How did you clean it? What tests did you run? With what parameters? And what library? It's different in R / Numpy, oh no, another week of sorting the haystack. It would be easier if there were a GitHub for each study where you could just pull the exact same analysis that the authors did. Also, I get the feeling most experts see themselves as experts in their particular field. Not surprising, given how people start by studying some subject and then get more and more degrees in it. But actually, the more I read papers from various fields, the more it seems statistics is the critical skill. Domain knowledge is necessary, but it means very little without stats. Now it would be nice if researchers in every field were also experts in statistics, but I happen to have a friend whose post-doc is to assist biomedical researchers with their stats. So he's a biochem domain expert, but also a stats expert, and the other researchers come to his little group to make sure things are statistically well reasoned.