4 ms·

In some fields like psychology, power is more likely to already be 10% or 20% for the majority of studies, and in fact P-hacking and low standards for evidence

In some fields like psychology, power is more likely to already be 10% or 20% for the majority of studies, and in fact P-hacking and low standards for evidence would be far less harmful if power were higher, because low power leads to inflated effect size estimates. Additionally, power calculations are always just a guess and easy to fudge, so it's pretty much a given that current statistical power would not be maintained with more stringent critical values. See http://andrewgelman.com/2014/11/17/power-06-looks-like-get-used/ http://andrewgelman.com/2014/11/17/power-06-looks-like-get-u... So this proposal is really the opposite of pragmatic. Pragmatic would be requiring effect size estimates and confidence intervals in all published papers. It is surprising how many papers will talk about highly significant effects without actually discussing how large the estimated effect is thought to be, which gives authors a lot of leeway when exaggerating the importance of their findings.

reilly3000 9y ago
The cost of increasing sample size is significant; this is a trade-off that allows smaller projects to still conduct valuable research.
- Klockan 9y ago
  Studies with low significance are no better than anecdotes.
  - stdbrouw 9y ago
    This is not true at all. Small studies are simply less accurate. Would you prefer to conduct further research on a cure for cancer that's been shown to lead to recovery in 0-50% of cases (the confidence interval includes 0%, so not significant) or in .5-1% of cases (significant)?
    - Will_Parker 9y ago
      How does it make sense to suppose it could lead to recovery in 50% of cases, if a drug that does nothing has a 5% chance of producing the same result in your study?
      - stdbrouw 9y ago
        Because it has a 95% chance of producing a different (better) result. Statistics is all about the quantification of uncertainty.
    - imh 9y ago
      As a matter of fact, I can do even better than 0-50% of cases! Consuming widgets has been shown to lead to recovery in 0-100% of cases! It isn't as simple as you suggest.
      - stdbrouw 9y ago
        Yep, and that makes widgets a more interesting target for follow-up studies than a treatment that has been shown to result in no clinically significant improvement. It really is that simple.
    - Klockan 9y ago
      Surely we could create a system to fund extensions of studies with promising initial results? I just think that it is dangerous to call 95% significant.
  - csours 9y ago
    https://xkcd.com/552/ https://xkcd.com/552/ Alt-Text: "Correlation doesn't imply causation, but it does waggle its eyebrows suggestively and gesture furtively while mouthing 'look over there'." I think you could replace correlation with "small studies" or "anecdotes". These are all things that suggest there may be some effect, and there may be merit in further study.
    - Klockan 9y ago
      I agree that anecdotes are useful, I disagree with publishing them in scientific journals and telling the public that we found significant results.
  - xelxebar 9y ago
    Well, Bayes theorem should let us, in theory, combine several small studies into the equivalent of one larger one. I'm practice it's harder than simple number crunching due to methodological differences between studies, but it seems a shame to call a study essentially worthless. I like to be a little optimistic and think, "Data is data and data is good. The challenge is in interpreting it."
    - _dps 9y ago
      > "Data is data and data is good" This is a reasonable intuition, but is hard to square with the increased risk of publication bias for small studies (i.e. we should question whether the typical small study published is really "data" in the sense we'd like). Since small studies are more sensitive to noise and "fishing" for an effect, and since non-results are rarely published, what happens with meta-analysis of many published small studies is that you end up primarily looking at outliers rather than typical results. Since most research questions have a directionality to what counts as "interesting", the outliers also tend to be clustered on one side rather than evenly split. The two papers I recommend anyone read if they care about these problems are: Why Most Published Research Findings are False http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124 http://journals.plos.org/plosmedicine/article?id=10.1371/jou... The garden of forking paths: Why multiple comparisons can be a problem, even when there is no “fishing expedition” or “p-hacking” and the research hypothesis was posited ahead of time http://www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf http://www.stat.columbia.edu/~gelman/research/unpublished/p_...
    - Klockan 9y ago
      There is plenty of evidence that people who recreate results often tries until they get the same result as the original paper. For example a scientist made a mistake when counting the number of chromosomes and reported that humans had 48, a number that was replicated for over 30 years. In this case it would obviously have been better if the original paper was never published. https://www.ncbi.nlm.nih.gov/pubmed/16463025 https://www.ncbi.nlm.nih.gov/pubmed/16463025
  - czep 9y ago
    That's true 95% of the time.
    - Klockan 9y ago
      Do you believe that 95% of studies proving that humans possess paranormal abilities are true?
maxerickson 9y ago
So ultimately the issue is that push button statistics don't work?
- xaa 9y ago
  Ultimately the issue is that we somehow got fixated on the p-value, which (roughly and imprecisely speaking) quantifies the probability that there is any effect, even a small one, rather than using effect size estimates which estimate both the magnitude of the observed effect and the uncertainty in that estimate. Using p-values as our primary metric means an overemphasis on finding small effects (which are usually not clinically relevant anyway) and unduly low focus on things with big effects. If an effect is real, but very small, that too may well cause replicability problems because it suggests the effect may not be very robust to small changes in experimental conditions, whereas a big effect would be more likely to be robust. If you think about the really important scientific findings -- the ones that made a big impact and are indisputably true -- statistics usually aren't necessary to prove them, because the effect size is so large it is simply obvious. I'm not against using statistics anyway, of course, but the point is that we should be looking mainly for effects with big effect sizes if we are after important findings, IMO. It is only a major bonus that big effect sizes are most likely to be replicable.

reilly3000 9y ago

The cost of increasing sample size is significant; this is a trade-off that allows smaller projects to still conduct valuable research.

Klockan 9y ago
Studies with low significance are no better than anecdotes.
- stdbrouw 9y ago
  This is not true at all. Small studies are simply less accurate. Would you prefer to conduct further research on a cure for cancer that's been shown to lead to recovery in 0-50% of cases (the confidence interval includes 0%, so not significant) or in .5-1% of cases (significant)?
  - Will_Parker 9y ago
    How does it make sense to suppose it could lead to recovery in 50% of cases, if a drug that does nothing has a 5% chance of producing the same result in your study?
    - stdbrouw 9y ago
      Because it has a 95% chance of producing a different (better) result. Statistics is all about the quantification of uncertainty.
  - imh 9y ago
    As a matter of fact, I can do even better than 0-50% of cases! Consuming widgets has been shown to lead to recovery in 0-100% of cases! It isn't as simple as you suggest.
    - stdbrouw 9y ago
      Yep, and that makes widgets a more interesting target for follow-up studies than a treatment that has been shown to result in no clinically significant improvement. It really is that simple.
  - Klockan 9y ago
    Surely we could create a system to fund extensions of studies with promising initial results? I just think that it is dangerous to call 95% significant.
- csours 9y ago
  https://xkcd.com/552/ https://xkcd.com/552/ Alt-Text: "Correlation doesn't imply causation, but it does waggle its eyebrows suggestively and gesture furtively while mouthing 'look over there'." I think you could replace correlation with "small studies" or "anecdotes". These are all things that suggest there may be some effect, and there may be merit in further study.
  - Klockan 9y ago
    I agree that anecdotes are useful, I disagree with publishing them in scientific journals and telling the public that we found significant results.
- xelxebar 9y ago
  Well, Bayes theorem should let us, in theory, combine several small studies into the equivalent of one larger one. I'm practice it's harder than simple number crunching due to methodological differences between studies, but it seems a shame to call a study essentially worthless. I like to be a little optimistic and think, "Data is data and data is good. The challenge is in interpreting it."
  - _dps 9y ago
    > "Data is data and data is good" This is a reasonable intuition, but is hard to square with the increased risk of publication bias for small studies (i.e. we should question whether the typical small study published is really "data" in the sense we'd like). Since small studies are more sensitive to noise and "fishing" for an effect, and since non-results are rarely published, what happens with meta-analysis of many published small studies is that you end up primarily looking at outliers rather than typical results. Since most research questions have a directionality to what counts as "interesting", the outliers also tend to be clustered on one side rather than evenly split. The two papers I recommend anyone read if they care about these problems are: Why Most Published Research Findings are False http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124 http://journals.plos.org/plosmedicine/article?id=10.1371/jou... The garden of forking paths: Why multiple comparisons can be a problem, even when there is no “fishing expedition” or “p-hacking” and the research hypothesis was posited ahead of time http://www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf http://www.stat.columbia.edu/~gelman/research/unpublished/p_...
  - Klockan 9y ago
    There is plenty of evidence that people who recreate results often tries until they get the same result as the original paper. For example a scientist made a mistake when counting the number of chromosomes and reported that humans had 48, a number that was replicated for over 30 years. In this case it would obviously have been better if the original paper was never published. https://www.ncbi.nlm.nih.gov/pubmed/16463025 https://www.ncbi.nlm.nih.gov/pubmed/16463025
- czep 9y ago
  That's true 95% of the time.
  - Klockan 9y ago
    Do you believe that 95% of studies proving that humans possess paranormal abilities are true?

maxerickson 9y ago

So ultimately the issue is that push button statistics don't work?

xaa 9y ago
Ultimately the issue is that we somehow got fixated on the p-value, which (roughly and imprecisely speaking) quantifies the probability that there is any effect, even a small one, rather than using effect size estimates which estimate both the magnitude of the observed effect and the uncertainty in that estimate. Using p-values as our primary metric means an overemphasis on finding small effects (which are usually not clinically relevant anyway) and unduly low focus on things with big effects. If an effect is real, but very small, that too may well cause replicability problems because it suggests the effect may not be very robust to small changes in experimental conditions, whereas a big effect would be more likely to be robust. If you think about the really important scientific findings -- the ones that made a big impact and are indisputably true -- statistics usually aren't necessary to prove them, because the effect size is so large it is simply obvious. I'm not against using statistics anyway, of course, but the point is that we should be looking mainly for effects with big effect sizes if we are after important findings, IMO. It is only a major bonus that big effect sizes are most likely to be replicable.