3 ms·

Is there a reason why so many studies (this included) fail to report the effect size? The way "significant" is used in literature, readers (even the educated on

by fny 3y ago

Is there a reason why so many studies (this included) fail to report the effect size? The way "significant" is used in literature, readers (even the educated ones) interpret the research as suggesting some groundbreaking result. Here, for example, only a 1-ish point change occurred on a six point scale from 3 to 2. I say 1-ish because the confidence intervals are rather wide and even intersect somewhat (see discussion.) So anger is not eliminated rather you'll be somewhat less angry.

aftoprokrustes 3y ago
In my experience (10 years in academia), people who understand statistics enough do look at effect size rather than significance levels. But unfortunately, even in one of the top european universities, a lot of researchers (who are ~70% PhD candidates with only basic stats training) tend to misinterpret significance as meaning "large effect". There is another cause, more pernicious, which is that in the absence of a big effect, "a significant effect of X on Y was found" sells better than "no large effect was found", because there is a bias towards reporting positive effects in the litterature. In my field, it is actually usual for people to take big surveys, iterate dozens of statistical models with any kind of interactions, and only report one where some interesting parameters end up being significant, and derive policy recommendations based on this. Which is a recipe for non reproducible results.
SerpentJoe 3y ago
There are at least two reasonable readings of "anger is eliminated", with one being true if anger is reduced at all, and both being true if anger is reduced to zero.
Lerc 3y ago
I think the reason is rather simple. Funding. I feel this is one of the two basic tricks that many many papers do to preemptively justify their existence. The other being the shoehorning of the hotbutton issue of the day and imagining some way the study addresses that issue. It was kind of weird watching the transition from when every second paper could obliquely assist in the fight against AIDS to now when they seem to assist in the fight against climate change. I feel like it is a minor dishonesty committed to combat the unreasonable requirement for every scientific discovery to be immediately applicable.

by fny 3y ago

aftoprokrustes 3y ago

In my experience (10 years in academia), people who understand statistics enough do look at effect size rather than significance levels. But unfortunately, even in one of the top european universities, a lot of researchers (who are ~70% PhD candidates with only basic stats training) tend to misinterpret significance as meaning "large effect". There is another cause, more pernicious, which is that in the absence of a big effect, "a significant effect of X on Y was found" sells better than "no large effect was found", because there is a bias towards reporting positive effects in the litterature. In my field, it is actually usual for people to take big surveys, iterate dozens of statistical models with any kind of interactions, and only report one where some interesting parameters end up being significant, and derive policy recommendations based on this. Which is a recipe for non reproducible results.

SerpentJoe 3y ago

There are at least two reasonable readings of "anger is eliminated", with one being true if anger is reduced at all, and both being true if anger is reduced to zero.

Lerc 3y ago

I think the reason is rather simple. Funding. I feel this is one of the two basic tricks that many many papers do to preemptively justify their existence. The other being the shoehorning of the hotbutton issue of the day and imagining some way the study addresses that issue. It was kind of weird watching the transition from when every second paper could obliquely assist in the fight against AIDS to now when they seem to assist in the fight against climate change. I feel like it is a minor dishonesty committed to combat the unreasonable requirement for every scientific discovery to be immediately applicable.