4 ms·

'Not statistically significant' means you can and should say nothing about it — it just as likely happened by chance as any other theory and, as such, you shoul

by sambeau 10y ago

'Not statistically significant' means you can and should say nothing about it — it just as likely happened by chance as any other theory and, as such, you should leave it that: it happened by chance. In terms of analysis - it might as well not have happened at all and is only worth mentioning if you had expected something significant to happen.

Bartweiss 10y ago
> it just as likely happened by chance as any other theory and, as such, you should leave it that: it happened by chance. That's wrong. That's very wrong, and it reinforces the confusion already surrounding 'significance'. p=0.05 means that there was a 5% chance of obtaining results at least as extreme as measured if the null hypothesis was true (i.e. there was no difference to observe). p=.06 is a 6% chance, and p=.04 is a 4% chance. None of those are "as likely as not", which would be 50%, or p=0.5. Nor is p=.05 an inherently meaningful cutoff - it's an arbitrary threshold on a continuum of possibilities, and indeed 'significance' is often set at p=.01 instead. Overemphasis of a cutoff leads to p-hacking, and treating significance as binary promotes bad studies (p=.04 is not as meaningful as p=.00001). This article is too credulous to an unproven result, but saying that all results >.05 are "as likely by chance" or not worth discussing is intensely wrong. The original blog post responded quite legitimately by indicating an unexpected-but-unproven result, and suggesting further study.
fragsworth 10y ago
> 'Not statistically significant' means you can and should say nothing about it No, absolutely not. Mentioning that they did it, and could not come to a conclusion because there weren't enough participants, is far better and more information than pretending it never happened. It can indicate that we probably should investigate further and maybe do studies with enough participants.

by sambeau 10y ago

Bartweiss 10y ago

> it just as likely happened by chance as any other theory and, as such, you should leave it that: it happened by chance. That's wrong. That's very wrong, and it reinforces the confusion already surrounding 'significance'. p=0.05 means that there was a 5% chance of obtaining results at least as extreme as measured if the null hypothesis was true (i.e. there was no difference to observe). p=.06 is a 6% chance, and p=.04 is a 4% chance. None of those are "as likely as not", which would be 50%, or p=0.5. Nor is p=.05 an inherently meaningful cutoff - it's an arbitrary threshold on a continuum of possibilities, and indeed 'significance' is often set at p=.01 instead. Overemphasis of a cutoff leads to p-hacking, and treating significance as binary promotes bad studies (p=.04 is not as meaningful as p=.00001). This article is too credulous to an unproven result, but saying that all results >.05 are "as likely by chance" or not worth discussing is intensely wrong. The original blog post responded quite legitimately by indicating an unexpected-but-unproven result, and suggesting further study.

fragsworth 10y ago

> 'Not statistically significant' means you can and should say nothing about it No, absolutely not. Mentioning that they did it, and could not come to a conclusion because there weren't enough participants, is far better and more information than pretending it never happened. It can indicate that we probably should investigate further and maybe do studies with enough participants.