3 ms·

The claims of the original study, the one that found problems with reproducibility, are being overstated. From the news report at the time: Strictly on the bas

The claims of the original study, the one that found problems with reproducibility, are being overstated. From the news report at the time: Strictly on the basis of significance — a statistical measure of how likely it is that a result did not occur by chance — 35 of the studies held up, and 62 did not. (Three were excluded because their significance was not clear.) The overall “effect size,” a measure of the strength of a finding, dropped by about half across all of the studies. Yet very few of the redone studies contradicted the original ones; their results were simply weaker. Also: The only factor that did [affect the likelihood of successful reproduction] was the strength of the original effect — that is, the most robust findings tended to remain easily detectable, if not necessarily as strong. Here's the HN discussion of the study: https://news.ycombinator.com/item?id=10131387 https://news.ycombinator.com/item?id=10131387

nonbel 11y ago
>"Strictly on the basis of significance — a statistical measure of how likely it is that a result did not occur by chance" Nope!
- AstralStorm 11y ago
  Indeed, what it might have meant is "strictly on the basis of p-values" or confidence intervals. Still, that is very risky, p-values attained using different tests cannot be compared. Not to the point, p-values have no meaning if you do not have a control group, because you do not have access to a population that might be under the null hypothesis. P-values are sometimes computed against a non-null hypotheses, e.g. null test is giving people $5 not nothing, while the main gives $100. This is not valid. Or against the same subgroup, which might not really be representative.
  - nonbel 11y ago
    I don't think your point is the same as mine. The p-value only tells you how likely it is to see a deviation at least as extreme as the data you plugged into the equation, if the model you are testing were true. That's it.

nonbel 11y ago

>"Strictly on the basis of significance — a statistical measure of how likely it is that a result did not occur by chance" Nope!

AstralStorm 11y ago
Indeed, what it might have meant is "strictly on the basis of p-values" or confidence intervals. Still, that is very risky, p-values attained using different tests cannot be compared. Not to the point, p-values have no meaning if you do not have a control group, because you do not have access to a population that might be under the null hypothesis. P-values are sometimes computed against a non-null hypotheses, e.g. null test is giving people $5 not nothing, while the main gives $100. This is not valid. Or against the same subgroup, which might not really be representative.
- nonbel 11y ago
  I don't think your point is the same as mine. The p-value only tells you how likely it is to see a deviation at least as extreme as the data you plugged into the equation, if the model you are testing were true. That's it.