3 ms·

Relevant XKCD: https://xkcd.com/882/ https://xkcd.com/882/ Do 20 experiments with a p<5% criterion, and it’s likely that one will be a false positive. Only pub

by setopt 2y ago

Relevant XKCD: https://xkcd.com/882/ https://xkcd.com/882/ Do 20 experiments with a p<5% criterion, and it’s likely that one will be a false positive. Only publish positive results, and someone will eventually publish a false positive result without fraud.

etrautmann 2y ago
Very few studies report a single statistical test as the sole conclusion. Most papers should assess some outcome in multiple ways using complementary data, multiple analyses etc. not always of course, but there are lots of ways of making sure your conclusions a robust without relying on a single analysis result.
- kerkeslager 2y ago
  That doesn't fix the problem at all. No matter how many statistical tests you run on a sample, you can't get around the fact that the sample may not be representative of the population or the underlying phenomenon. You need different samples. There isn't a statistical trick that gets around this. For example: let's say there's a cancer with 20% survival rate. You test a treatment with 25 experimental and 25 control patients, 40% in the experimental group survive[1]. You can analyze this with a bunch of statistical methods. You can ask different questions about the patients, focusing on well-being rather than simple carcinogenic remission. But ultimately, the thing that happened in this study is that 50% got better and no fiddling with numbers or changing the questions you ask is going to change that underlying phenomenon. You can check for blood markers of cancer: you get 50% have no blood markers. You ask them questions about how they feel: you get 50% feel better. You body scan the area where tumors were: you get 50% no longer have tumors. You have only tested one phenomenon in one sample, and that essentially amounts to 5 people getting better. [1] I know this is not how cancer treatment studies work exactly, this is a simplified hypothetical.
  - etrautmann 2y ago
    That’s not what I’m saying. Obviously you can’t run multiple tests on one set of data. If you have a hypothesis, test it on multiple data sets in multiple ways and find supporting evidence rather than finding a single p=.05 and writing a story around it.
    - kerkeslager 2y ago
      Okay, if that's what you're saying, then I don't understand why it sounds like you're disagreeing with the poster you're responding to.
nathell 2y ago
> Do 20 experiments with a p<5% criterion, and it’s likely that one will be a false positive. That would be true if p were the probability of null hypothesis being true given the data observed, but that’s not what it is.
- lucianbr 2y ago
  I really have no clue what p is, but I also really believe Randall Munroe does. Not that he's above making mistakes, but come on.
  - mtts 2y ago
    p is the chance that you would have gotten this result if the null hypothesis were true.
- stonemetal12 2y ago
  >For example, if 1,000,000 tests are carried out, then 5% of them (that is, 50,000 tests) are expected to lead to p < 0.05 by chance when the null hypothesis is actually true for all these tests. https://eurradiolexp.springeropen.com/articles/10.1186/s41747-020-0145-y#:~:text=Keeping%20the%20significance%20threshold%20at,true%20for%20all%20these%20tests https://eurradiolexp.springeropen.com/articles/10.1186/s4174....
- kerkeslager 2y ago
  Even if you are right, how would this response be helpful? You're not giving the right answer, you're just saying the answer given is wrong. Nobody is coming out of this interaction with corrected knowledge in their head. This is the sort of thing I used to do when I was younger, and looking back, the reason I did it was because I was basing my sense of self-worth in being smarter than other people. Ironically, this made me dumber, because I was less open to the possibility of being wrong and therefore was slower to learn. And, I found doing this made people dislike me.
  - deleted 2y ago
    [deleted]
humansareok1 2y ago
>someone will eventually publish a false positive result without fraud. I think most people correctly intuit that this is actually a type of very pernicious fraud.
- whack 2y ago
  If 20 different people all conduct the same experiment, and the 19 negative results are never published, there is no fraud involved when the 20th person publishes his positive result without realizing it is a 5% statistical anomaly. That person probably has no idea that 19 other people tried and failed at near-identical experiments. This seems like such an obvious problem in the way science is currently done. Are people so focused on their own individual fields that they aren't thinking about and fixing such glaring meta problems?
  - prerok 2y ago
    As others already pointed out, there is no incentive to do so. Consider: "Hey, look, I went on top of tower of Pisa and threw down two identically shaped balls, one iron and one wooden. They dropped at the same time!" The above is the expected result and would only be interesting if the result is different from expectation. Now, if 1000 scientists did this and each published the confirmation of what we knew would happen then who would read that? But, if one scientist said: "I tried and they drop at different times!" that would be different. The 999 scientists would then try to replicate again and then the papers of the 999 would be interesting again.
  - humansareok1 2y ago
    This is completely different from one lab running the same experiment 20 times and publishing the one positive result.
some_random 2y ago
It's a great xkcd but it's really wrong on one count, it's not some outside media/popular force compelling scientists to investigate something. Most of the time, researchers are looking to prove something they "know" to be true. They truly believe that jelly beans cause acne, they just need to prove it. When they get a negative result, they simply don't believe it. Something must have gone wrong, obviously, because jelly beans obviously cause acne, so maybe it's a color thing? Ah hah it's the green ones, now that we have our results we can construct other metrics to support this correct data! Eventually if we (the public) are lucky someone else in the field will disagree and run the trial again, which is how you get the alt text.
- aeternum 2y ago
  Yes, I find that when reading a paper I think to myself "Do the authors really want this to be true?" and if the answer is yes as it often is, I boost my own acceptance criteria to p<.03 Particle physics still uses five sigma as the significance threshold.

by setopt 2y ago

etrautmann 2y ago

Very few studies report a single statistical test as the sole conclusion. Most papers should assess some outcome in multiple ways using complementary data, multiple analyses etc. not always of course, but there are lots of ways of making sure your conclusions a robust without relying on a single analysis result.

kerkeslager 2y ago
That doesn't fix the problem at all. No matter how many statistical tests you run on a sample, you can't get around the fact that the sample may not be representative of the population or the underlying phenomenon. You need different samples. There isn't a statistical trick that gets around this. For example: let's say there's a cancer with 20% survival rate. You test a treatment with 25 experimental and 25 control patients, 40% in the experimental group survive[1]. You can analyze this with a bunch of statistical methods. You can ask different questions about the patients, focusing on well-being rather than simple carcinogenic remission. But ultimately, the thing that happened in this study is that 50% got better and no fiddling with numbers or changing the questions you ask is going to change that underlying phenomenon. You can check for blood markers of cancer: you get 50% have no blood markers. You ask them questions about how they feel: you get 50% feel better. You body scan the area where tumors were: you get 50% no longer have tumors. You have only tested one phenomenon in one sample, and that essentially amounts to 5 people getting better. [1] I know this is not how cancer treatment studies work exactly, this is a simplified hypothetical.
- etrautmann 2y ago
  That’s not what I’m saying. Obviously you can’t run multiple tests on one set of data. If you have a hypothesis, test it on multiple data sets in multiple ways and find supporting evidence rather than finding a single p=.05 and writing a story around it.
  - kerkeslager 2y ago
    Okay, if that's what you're saying, then I don't understand why it sounds like you're disagreeing with the poster you're responding to.

nathell 2y ago

> Do 20 experiments with a p<5% criterion, and it’s likely that one will be a false positive. That would be true if p were the probability of null hypothesis being true given the data observed, but that’s not what it is.

lucianbr 2y ago
I really have no clue what p is, but I also really believe Randall Munroe does. Not that he's above making mistakes, but come on.
- mtts 2y ago
  p is the chance that you would have gotten this result if the null hypothesis were true.
stonemetal12 2y ago
>For example, if 1,000,000 tests are carried out, then 5% of them (that is, 50,000 tests) are expected to lead to p < 0.05 by chance when the null hypothesis is actually true for all these tests. https://eurradiolexp.springeropen.com/articles/10.1186/s41747-020-0145-y#:~:text=Keeping%20the%20significance%20threshold%20at,true%20for%20all%20these%20tests https://eurradiolexp.springeropen.com/articles/10.1186/s4174....
kerkeslager 2y ago
Even if you are right, how would this response be helpful? You're not giving the right answer, you're just saying the answer given is wrong. Nobody is coming out of this interaction with corrected knowledge in their head. This is the sort of thing I used to do when I was younger, and looking back, the reason I did it was because I was basing my sense of self-worth in being smarter than other people. Ironically, this made me dumber, because I was less open to the possibility of being wrong and therefore was slower to learn. And, I found doing this made people dislike me.
- deleted 2y ago
  [deleted]

humansareok1 2y ago

>someone will eventually publish a false positive result without fraud. I think most people correctly intuit that this is actually a type of very pernicious fraud.

whack 2y ago
If 20 different people all conduct the same experiment, and the 19 negative results are never published, there is no fraud involved when the 20th person publishes his positive result without realizing it is a 5% statistical anomaly. That person probably has no idea that 19 other people tried and failed at near-identical experiments. This seems like such an obvious problem in the way science is currently done. Are people so focused on their own individual fields that they aren't thinking about and fixing such glaring meta problems?
- prerok 2y ago
  As others already pointed out, there is no incentive to do so. Consider: "Hey, look, I went on top of tower of Pisa and threw down two identically shaped balls, one iron and one wooden. They dropped at the same time!" The above is the expected result and would only be interesting if the result is different from expectation. Now, if 1000 scientists did this and each published the confirmation of what we knew would happen then who would read that? But, if one scientist said: "I tried and they drop at different times!" that would be different. The 999 scientists would then try to replicate again and then the papers of the 999 would be interesting again.
- humansareok1 2y ago
  This is completely different from one lab running the same experiment 20 times and publishing the one positive result.

some_random 2y ago

It's a great xkcd but it's really wrong on one count, it's not some outside media/popular force compelling scientists to investigate something. Most of the time, researchers are looking to prove something they "know" to be true. They truly believe that jelly beans cause acne, they just need to prove it. When they get a negative result, they simply don't believe it. Something must have gone wrong, obviously, because jelly beans obviously cause acne, so maybe it's a color thing? Ah hah it's the green ones, now that we have our results we can construct other metrics to support this correct data! Eventually if we (the public) are lucky someone else in the field will disagree and run the trial again, which is how you get the alt text.

aeternum 2y ago
Yes, I find that when reading a paper I think to myself "Do the authors really want this to be true?" and if the answer is yes as it often is, I boost my own acceptance criteria to p<.03 Particle physics still uses five sigma as the significance threshold.