4 ms·

A lot of words to say something that essentially boils down to, "Make sure your findings replicate."

yellowstuff 4y ago
Most people care about their findings replicating. The hard part is knowing what you need to do to improve your chances.
hackerlight 4y ago
More like "make sure your test power is what you think it is". There will still be results that fail to replicate by virtue of the rejection of the null hypothesis by chance, but that should only happen 1 in 20 times at an alpha of 0.05. With all the bad practices that alter test power, such as p-hacking and the file drawer effect, that 1 in 20 blows up to 1 in 2.
- hackandthink 4y ago
  Gelman makes a distinction between p-hacking and choosing the test based on your data. p-hacking is confirming your preferred hypothesis - just try again. the latter let's you write a paper. (test power is different: if there's really an effect, would you detect it)
  - hackerlight 4y ago
    Oops, I meant to say test size, not test power. A test size of 0.05 is only really 0.05 if the entire process you used was free from p-hacking. And as a reader of that paper that process includes the file drawer effect. All these things have an impact on distorting the test size away from what it is advertised to be.
jldugger 4y ago
In principle the robustness approach (redoing the analysis with some different parameters) tries to show good faith. This matters because observational studies have no meaningful replication, no second collection of data. In practice it just means practitioners gravitate towards methods more likely to show false positives. Like, this discontinuity at zero approach seems geared to show large effects from noise. When the linear regression across the dataset is a flat line, splitting at the discontinuity pretty much mathematically requires _any_ slope have an opposite slope on the other side.
civilized 4y ago
As if that was a thing you could just do? "Oh, forgot to make sure my findings replicated. I'll be sure to check that box next time, easy enough!"

Most people care about their findings replicating. The hard part is knowing what you need to do to improve your chances.

hackerlight 4y ago

More like "make sure your test power is what you think it is". There will still be results that fail to replicate by virtue of the rejection of the null hypothesis by chance, but that should only happen 1 in 20 times at an alpha of 0.05. With all the bad practices that alter test power, such as p-hacking and the file drawer effect, that 1 in 20 blows up to 1 in 2.

hackandthink 4y ago
Gelman makes a distinction between p-hacking and choosing the test based on your data. p-hacking is confirming your preferred hypothesis - just try again. the latter let's you write a paper. (test power is different: if there's really an effect, would you detect it)
- hackerlight 4y ago
  Oops, I meant to say test size, not test power. A test size of 0.05 is only really 0.05 if the entire process you used was free from p-hacking. And as a reader of that paper that process includes the file drawer effect. All these things have an impact on distorting the test size away from what it is advertised to be.

jldugger 4y ago

In principle the robustness approach (redoing the analysis with some different parameters) tries to show good faith. This matters because observational studies have no meaningful replication, no second collection of data. In practice it just means practitioners gravitate towards methods more likely to show false positives. Like, this discontinuity at zero approach seems geared to show large effects from noise. When the linear regression across the dataset is a flat line, splitting at the discontinuity pretty much mathematically requires _any_ slope have an opposite slope on the other side.

civilized 4y ago

As if that was a thing you could just do? "Oh, forgot to make sure my findings replicated. I'll be sure to check that box next time, easy enough!"