5 ms·

More to the point is how "replication" is defined. This quote covers it: "Note that the 36% figure comes from a definition of replication that mimics the defini

by geomark 11y ago

More to the point is how "replication" is defined. This quote covers it: "Note that the 36% figure comes from a definition of replication that mimics the definition used by regulatory agencies: results are considered replicated if a p-value < 0.05 was reached in both the original study and the replicated one." If you read the post where that quote comes from [1] they make a number of points about how a better and more rigorous definition of replicated is needed, because p-value alone doesn't tell the whole story. [1] http://simplystatistics.org/2015/10/20/we-need-a-statistically-rigorous-and-scientifically-meaningful-definition-of-replication/ http://simplystatistics.org/2015/10/20/we-need-a-statistical...

DennisP 11y ago
So say one study has a p-value of .04, meaning there's a .04 probability that there's no effect and the results occurred by chance. A replication comes along and gets a p-value of .06, so it gets counted as a failed replication. And yet, the probability that both results happened by chance is only .0024.
- nonbel 11y ago
  >"there's a .04 probability that there's no effect and the results occurred by chance." No, the p value is the probability of observing a result at least as extreme as your own given the null hypothesis is true. There are two errors here 1) Transposing the conditional: P(A|B) != P(B|A) http://rationalwiki.org/wiki/Confusion_of_the_inverse http://rationalwiki.org/wiki/Confusion_of_the_inverse 2) Deviations from the null hypothesis can occur even the absence of a treatment effect, ie one of your model assumptions is wrong, baseline differences, etc. Anyway, that is why instead of statistical significance researchers need to estimate the size of the effect. If you estimate the effect is in the range -1 to +2 then you publish that result. Others also estimate the range and see if these are consistent with each other.
  - DennisP 11y ago
    > probability of observing a result at least as extreme as your own given the null hypothesis is true. My wording was poor but that's what I meant.
    - nonbel 11y ago
      They are two completely different things. Check out the link. Would you accidentally say "It's cloudy therefore there is a high probability of rain" when you meant "It's raining so there is a high probability of clouds"?
nonbel 11y ago
From your link: >"So should intersecting confidence intervals be our definition of replication? This too has a flaw since it favors imprecise studies with very large confidence intervals. If effect size is ignored, we may waste our time trying to replicate studies reporting practically meaningless findings." I don't see why the definition of a replication should have anything to do with practical use or precision of the estimate. These are other important, but different, issues. Intersecting estimates of the plausible range is fine as a definition. The observations are consistent with each other. The best way to make these estimates is yet another tangential issue.

by geomark 11y ago

DennisP 11y ago

So say one study has a p-value of .04, meaning there's a .04 probability that there's no effect and the results occurred by chance. A replication comes along and gets a p-value of .06, so it gets counted as a failed replication. And yet, the probability that both results happened by chance is only .0024.

nonbel 11y ago
>"there's a .04 probability that there's no effect and the results occurred by chance." No, the p value is the probability of observing a result at least as extreme as your own given the null hypothesis is true. There are two errors here 1) Transposing the conditional: P(A|B) != P(B|A) http://rationalwiki.org/wiki/Confusion_of_the_inverse http://rationalwiki.org/wiki/Confusion_of_the_inverse 2) Deviations from the null hypothesis can occur even the absence of a treatment effect, ie one of your model assumptions is wrong, baseline differences, etc. Anyway, that is why instead of statistical significance researchers need to estimate the size of the effect. If you estimate the effect is in the range -1 to +2 then you publish that result. Others also estimate the range and see if these are consistent with each other.
- DennisP 11y ago
  > probability of observing a result at least as extreme as your own given the null hypothesis is true. My wording was poor but that's what I meant.
  - nonbel 11y ago
    They are two completely different things. Check out the link. Would you accidentally say "It's cloudy therefore there is a high probability of rain" when you meant "It's raining so there is a high probability of clouds"?

nonbel 11y ago

From your link: >"So should intersecting confidence intervals be our definition of replication? This too has a flaw since it favors imprecise studies with very large confidence intervals. If effect size is ignored, we may waste our time trying to replicate studies reporting practically meaningless findings." I don't see why the definition of a replication should have anything to do with practical use or precision of the estimate. These are other important, but different, issues. Intersecting estimates of the plausible range is fine as a definition. The observations are consistent with each other. The best way to make these estimates is yet another tangential issue.