3 ms·

I thought "statistically insignificant" meant we couldn't conclude anything. So I was surprised. [1] says: > In statistical hypothesis testing,[1][2] a result

by jraph 2y ago

I thought "statistically insignificant" meant we couldn't conclude anything. So I was surprised. [1] says: > In statistical hypothesis testing,[1][2] a result has statistical significance when a result at least as "extreme" would be very infrequent if the null hypothesis were true So I understand this journal publishes results for which a hypothesis was tested, found to give insignificant results, which would rule out the hypothesis assuming the research was correctly conducted, without biases in the methodology, with a big enough sample, etc. Which would be worthy to know but no journal usually takes this research because it doesn't make the headlines (which yes, I've always found was a shame). Do I get this right? [1] https://en.wikipedia.org/wiki/Statistical_significance https://en.wikipedia.org/wiki/Statistical_significance

bumby 2y ago
Yes, statistical insignificance doesn't "prove" the null hypothesis, it just fails to reject it. It's a subtle, but sometime misunderstood distinction. It's a measure of how big the effect size is and how often you'd expect to see it just by chance rather than due to the variables you're measuring. If it's a really extreme difference, we expect it to happen less often just by chance alone than if it's a really miniscule difference. >Which would be worthy to know but no journal usually takes this research because it doesn't make the headlines. That's usually correct, which gives rise to all kinds of issues like the article talks about. It can result in a lot of wasted time (when you're conducting what you "think" is a new experiment, but it's been done many times but unpublished because it doesn't provide statistically significant results). It provides little incentive for replication, which can lead to stronger conclusions about the results than may be warranted, etc.
- foldr 2y ago
  The flip side of this is that there is almost always a very small effect, even if you are testing a crazy hypothesis (there are very weak correlations between all sorts of things). So you can often get a ‘significant’ result just by using a huge sample, even though the effect size is too small to matter practically.
  - deleted 2y ago
    [deleted]

by jraph 2y ago

bumby 2y ago

Yes, statistical insignificance doesn't "prove" the null hypothesis, it just fails to reject it. It's a subtle, but sometime misunderstood distinction. It's a measure of how big the effect size is and how often you'd expect to see it just by chance rather than due to the variables you're measuring. If it's a really extreme difference, we expect it to happen less often just by chance alone than if it's a really miniscule difference. >Which would be worthy to know but no journal usually takes this research because it doesn't make the headlines. That's usually correct, which gives rise to all kinds of issues like the article talks about. It can result in a lot of wasted time (when you're conducting what you "think" is a new experiment, but it's been done many times but unpublished because it doesn't provide statistically significant results). It provides little incentive for replication, which can lead to stronger conclusions about the results than may be warranted, etc.

foldr 2y ago
The flip side of this is that there is almost always a very small effect, even if you are testing a crazy hypothesis (there are very weak correlations between all sorts of things). So you can often get a ‘significant’ result just by using a huge sample, even though the effect size is too small to matter practically.
- deleted 2y ago
  [deleted]