4 ms·
Statistical significance is about whether an effect can reliably be said to have been measured at all; it's not about whether or not the effect itself would be
by danparsonson 5mo ago
Statistical significance is about whether an effect can reliably be said to have been measured at all; it's not about whether or not the effect itself would be significant in the sense of moving some other needle.
The ~5% improvement reported here might just be an artefact of the data collection or random variation, rather than a consistent repeatable change.
- tgv 5mo agoI know what significance means, and I also know that getting it from a p-value is nonsensical. > The ~5% improvement reported here might just be an artefact of the data collection or random variation, rather than a consistent repeatable change. You're questioning method or data representativeness, not significance. 250 samples is just about enough to for a 5% difference in NHST (stddev is around .4, so 1.64 sigma is .4/15.8*1.64=0.04 for single sided testing).
- not2b 5mo agoYes, it looks just barely significant. Results that are on the edge like that often aren't reproducible.