3 ms·
> They can and do slice data in various ways If you’re looking for statistical validity , this is not the way to go about it; if you perform 20 analyses, there
by danielheath 2y ago
> They can and do slice data in various ways
If you’re looking for statistical validity , this is not the way to go about it; if you perform 20 analyses, there’s too high a chance one of them will spuriously show p>.95 (see p-hacking).
- yuliyp 2y agoThey're not trying to write a paper with the identified. They're trying to identify when regressions creep in so they can be fixed. Half the paper talks about all the ways they filter signals into things which are likely actionable. Sure they'll still get some wrong, but as long as the precision is high enough for engineers to treat the regression reports seriously and the recall is high enough that costs stay down, that's all that matters.
- danielheath 2y agoI understand that - was more pointing out that "check every possible statistical test" tanks your S/N ratio by firing way too many false positives.