5 ms·

Agree. It doesn't stop p-hacking, it just makes it harder. Definitely treating the symptom instead of the disease. We ultimately need institutional and cultu

Agree. It doesn't stop p-hacking, it just makes it harder. Definitely treating the symptom instead of the disease. We ultimately need institutional and cultural change, but it's not obvious how to do that in the short term, so making it harder to claim significance might be a step in the right direction. On the other hand, you might expect that new discoveries by nature have less data since data is likely more expensive for brand new research, and by extension a lower likelihood of meeting these sorts of stringent statistical requirements. Decreasing the p-value threshold may be counter-productive if we dismiss legitimate new discoveries due to essentially economic constraints with data gathering, which would have the impact of making it less likely to get funding to pursue the problem in more depth, thereby slowing the advance of discoveries.

jjoonathan 9y ago
> It doesn't stop p-hacking, it just makes it harder. I could see the reverse happening, where higher p-value standards lead to normalization of deviance in the form of worse p-hacking.
- felippee 9y ago
  I agree with that. In my prediction legitimate, honest research with p~0.05 will become unpublishable, while p-hacked bullshit will prevail in glory.
  - PeterisP 9y ago
    If you do a honest preliminary study and get a p~0.05 result, then repeating the study with 70% larger sample size should get you p~0.005; but if you've p-hacked, it won't.
    - felippee 9y ago
      But sometimes getting 70% larger sample will cost you so much money, that you'd need another grant, one you'd get after you'd publish your p=0.05 result, which given the new rules you'd not be able to publish... Statistics is just another method and almost every method can be hacked or abused. Science is not about putting checkmarks in tables but about reading and understanding ideas and reproducing the results. Tweaking some numeric values is not going to help the review process which is fundamentally broken these days.
      - zeckalpha 9y ago
        This is addressed in the paper. The additional costs of larger sample sizes outweigh the costs of studies based on false positives.

jjoonathan 9y ago

> It doesn't stop p-hacking, it just makes it harder. I could see the reverse happening, where higher p-value standards lead to normalization of deviance in the form of worse p-hacking.

felippee 9y ago
I agree with that. In my prediction legitimate, honest research with p~0.05 will become unpublishable, while p-hacked bullshit will prevail in glory.
- PeterisP 9y ago
  If you do a honest preliminary study and get a p~0.05 result, then repeating the study with 70% larger sample size should get you p~0.005; but if you've p-hacked, it won't.
  - felippee 9y ago
    But sometimes getting 70% larger sample will cost you so much money, that you'd need another grant, one you'd get after you'd publish your p=0.05 result, which given the new rules you'd not be able to publish... Statistics is just another method and almost every method can be hacked or abused. Science is not about putting checkmarks in tables but about reading and understanding ideas and reproducing the results. Tweaking some numeric values is not going to help the review process which is fundamentally broken these days.
    - zeckalpha 9y ago
      This is addressed in the paper. The additional costs of larger sample sizes outweigh the costs of studies based on false positives.