3 ms·

> p=0.05 means that one result in 20 is going to be the result of chance. You made the same mistake most people make here: you turned the arrow of the implicat

> p=0.05 means that one result in 20 is going to be the result of chance. You made the same mistake most people make here: you turned the arrow of the implication. It is not "successful experiment implies chance (probability 5%)" but "chance implies successful experiment (probability 5%)". What does that mean in practice? Imagine a hypothetical scientist that is fundamentally confused about something important, so all hypotheses they generate are false. Yet, using p=0.05, 5% of those hypotheses will be "confirmed experimentally". In that case, it is not 5% of the "experimentally confirmed" hypotheses that are wrong -- it is full 100%. Even without any cherry-picking. The problem is not that p=0.05 is too high. The problem is, it doesn't actually mean what most people believe it means.

lisper 3y ago
I think we're actually in violent agreement here, but I just wasn't precise enough. Let me try again: p=0.05 means that one POSITIVE result in 20 is going to be the result of chance and not causality In other words: if I have some kind of intervention or treatment, and that intervention or treatment produces some result in a test group relative to a control group with p=0.05, then the odds of getting that result simply by chance and not because the treatment or intervention actually had an effect are 5%. The practical effect of this is that there are two different ways of getting a p=0.05 result: 1. Find a treatment or intervention that actually works or 2. Test ~20 different (useless) interventions. Or test one useless intervention ~20 times. A single p=0.05 result in isolation is useless because there is no way to know which of the two methods produced it. This is why replication is so important. The odds of getting a p=0.05 result by chance is 5%. But the odds of getting TWO of them in sequential trials is 0.25%, and the odds of a positive result being the result of pure chance decrease exponentially with each subsequent replication.
- thaumasiotes 3y ago
  > Let me try again: > p=0.05 means that one POSITIVE result in 20 is going to be the result of chance and not causality No, you still didn't get it. In the example above, a full 100% of positive results, 20 out of every 20, are the result of chance and not causality. Your followup discussion is better, but your statement at the top doesn't work. (Note also that there is an interaction between p-threshold and sample size which guarantees that, if you're investigating an effect that your sample size is not large enough to detect, any statistically significant result that you get will be several times stronger than the actual effect. They're also quite likely to have the wrong sign.)
  - lisper 3y ago
    > No, you still didn't get it. In the example above, a full 100% of positive results, 20 out of every 20, are the result of chance and not causality. Yep, you're right. I do think I understand this, but rendering it into words is turning out to be surprisingly challenging. Let me try this one more time: p=0.05 means that there is a 5% chance that any one particular positive result is due to chance. If you test a false hypothesis repeatedly, or test multiple false hypotheses, then 5% of the time you will get false positives (at p=0.05). However... > Imagine a hypothetical scientist that is fundamentally confused about something important, so all hypotheses they generate are false. Yet, using p=0.05, 5% of those hypotheses will be "confirmed experimentally". In that case, it is not 5% of the "experimentally confirmed" hypotheses that are wrong -- it is full 100%. This is not wrong, but it's a little misleading because you are presuming that all of the hypotheses being tested are false. If we're testing a hypothesis it's generally because we don't know whether or not it's true; we're trying to find out. That's why it's important to think of a positive result not as "confirmed experimentally" but rather as "not ruled out by this particular experimental result". It is only after failing to rule something out by multiple experiments that we can start to call it "confirmed". And nothing is ever 100% confirmed -- at best it is "not ruled out by the evidence so far".
    - hgsgm 3y ago
      You can't simply ignore the base rate, even if you don't know it. In a purely random world, 5% of experiments are false positives, at p=0.05. None are true positives. In a well ordered world with brilliant hypotheses, there are no false positives. If more than 5% of experiments show positive results at p=0.05, some of them are probably true, so you can try to replicate them with lower p. p=0.05 is a filter for "worth trying to replicate" (but even that is modulated by cost of replication vs value of result). The crisis in science is largely that people confuse "publishable" with "probably true". Anything "probably better then random guessing" is publishable to help other researchers, but that doesn't mean it's probably true.
      - lisper 3y ago
        > p=0.05 is a filter for "worth trying to replicate" Yes, I think that is an excellent way to put it. > The crisis in science is largely that people confuse "publishable" with "probably true". I would put it slightly differently: people conflate "published in a top-tier peer-reviewed journal" with "true beyond reasonable dispute". They also conflate "not published in a top-tier peer-reviewed journal" with "almost certainly false." But I think we're in substantial agreement here.
majormajor 3y ago
> What does that mean in practice? Imagine a hypothetical scientist that is fundamentally confused about something important, so all hypotheses they generate are false. Yet, using p=0.05, 5% of those hypotheses will be "confirmed experimentally". In that case, it is not 5% of the "experimentally confirmed" hypotheses that are wrong -- it is full 100%. Even without any cherry-picking. Well, that's example is also introducing dependence, which is a tricky thing of course whenever we talk about chance and stats. But there's also another issue - a statement like "5% of positive published results are by chance since we have a p<=0.05 standard" treats every set of results as if p=0.05, whereas some of them are considerably lower anyway. Though the point of bad actors cherry-picking to screw up the data also comes into play here. (And of course, fully independent things in life are much harder to find than one might think at first.)
- stkdump 3y ago
  I agree that the point about the 'confused scientist' is important, even if that itself is not stated clearly enough. Here is my own reading: Imagine that a scientist is making experiments of the form: Does observable variable A correlate with observable variable B? Now imagine that there are billions of observable variables and almost all of them are not correlated. And imagine that there is no better way to come up with plausible correlations to test than randomly picking variables. Then it will take a very long time and a very large number of experiments to find a pair that is truly correlated. It will be inevitable that most positive results are bogus.

lisper 3y ago

I think we're actually in violent agreement here, but I just wasn't precise enough. Let me try again: p=0.05 means that one POSITIVE result in 20 is going to be the result of chance and not causality In other words: if I have some kind of intervention or treatment, and that intervention or treatment produces some result in a test group relative to a control group with p=0.05, then the odds of getting that result simply by chance and not because the treatment or intervention actually had an effect are 5%. The practical effect of this is that there are two different ways of getting a p=0.05 result: 1. Find a treatment or intervention that actually works or 2. Test ~20 different (useless) interventions. Or test one useless intervention ~20 times. A single p=0.05 result in isolation is useless because there is no way to know which of the two methods produced it. This is why replication is so important. The odds of getting a p=0.05 result by chance is 5%. But the odds of getting TWO of them in sequential trials is 0.25%, and the odds of a positive result being the result of pure chance decrease exponentially with each subsequent replication.

thaumasiotes 3y ago
> Let me try again: > p=0.05 means that one POSITIVE result in 20 is going to be the result of chance and not causality No, you still didn't get it. In the example above, a full 100% of positive results, 20 out of every 20, are the result of chance and not causality. Your followup discussion is better, but your statement at the top doesn't work. (Note also that there is an interaction between p-threshold and sample size which guarantees that, if you're investigating an effect that your sample size is not large enough to detect, any statistically significant result that you get will be several times stronger than the actual effect. They're also quite likely to have the wrong sign.)
- lisper 3y ago
  > No, you still didn't get it. In the example above, a full 100% of positive results, 20 out of every 20, are the result of chance and not causality. Yep, you're right. I do think I understand this, but rendering it into words is turning out to be surprisingly challenging. Let me try this one more time: p=0.05 means that there is a 5% chance that any one particular positive result is due to chance. If you test a false hypothesis repeatedly, or test multiple false hypotheses, then 5% of the time you will get false positives (at p=0.05). However... > Imagine a hypothetical scientist that is fundamentally confused about something important, so all hypotheses they generate are false. Yet, using p=0.05, 5% of those hypotheses will be "confirmed experimentally". In that case, it is not 5% of the "experimentally confirmed" hypotheses that are wrong -- it is full 100%. This is not wrong, but it's a little misleading because you are presuming that all of the hypotheses being tested are false. If we're testing a hypothesis it's generally because we don't know whether or not it's true; we're trying to find out. That's why it's important to think of a positive result not as "confirmed experimentally" but rather as "not ruled out by this particular experimental result". It is only after failing to rule something out by multiple experiments that we can start to call it "confirmed". And nothing is ever 100% confirmed -- at best it is "not ruled out by the evidence so far".
  - hgsgm 3y ago
    You can't simply ignore the base rate, even if you don't know it. In a purely random world, 5% of experiments are false positives, at p=0.05. None are true positives. In a well ordered world with brilliant hypotheses, there are no false positives. If more than 5% of experiments show positive results at p=0.05, some of them are probably true, so you can try to replicate them with lower p. p=0.05 is a filter for "worth trying to replicate" (but even that is modulated by cost of replication vs value of result). The crisis in science is largely that people confuse "publishable" with "probably true". Anything "probably better then random guessing" is publishable to help other researchers, but that doesn't mean it's probably true.
    - lisper 3y ago
      > p=0.05 is a filter for "worth trying to replicate" Yes, I think that is an excellent way to put it. > The crisis in science is largely that people confuse "publishable" with "probably true". I would put it slightly differently: people conflate "published in a top-tier peer-reviewed journal" with "true beyond reasonable dispute". They also conflate "not published in a top-tier peer-reviewed journal" with "almost certainly false." But I think we're in substantial agreement here.

majormajor 3y ago

> What does that mean in practice? Imagine a hypothetical scientist that is fundamentally confused about something important, so all hypotheses they generate are false. Yet, using p=0.05, 5% of those hypotheses will be "confirmed experimentally". In that case, it is not 5% of the "experimentally confirmed" hypotheses that are wrong -- it is full 100%. Even without any cherry-picking. Well, that's example is also introducing dependence, which is a tricky thing of course whenever we talk about chance and stats. But there's also another issue - a statement like "5% of positive published results are by chance since we have a p<=0.05 standard" treats every set of results as if p=0.05, whereas some of them are considerably lower anyway. Though the point of bad actors cherry-picking to screw up the data also comes into play here. (And of course, fully independent things in life are much harder to find than one might think at first.)

stkdump 3y ago
I agree that the point about the 'confused scientist' is important, even if that itself is not stated clearly enough. Here is my own reading: Imagine that a scientist is making experiments of the form: Does observable variable A correlate with observable variable B? Now imagine that there are billions of observable variables and almost all of them are not correlated. And imagine that there is no better way to come up with plausible correlations to test than randomly picking variables. Then it will take a very long time and a very large number of experiments to find a pair that is truly correlated. It will be inevitable that most positive results are bogus.