3 ms·

>unblinded study with no placebo control arm. “If you have nine people,” says Horvath, “and you get a statistically significant result, it means there’s a stron

>unblinded study with no placebo control arm. “If you have nine people,” says Horvath, “and you get a statistically significant result, it means there’s a strong effect.” Wow, no. Just... No. It doesn't. That is so wrong that, from a researcher with a doctorate in biostatistics, it's very difficult not to infer deliberate misinformation on their part. To give some benefit of the doubt though, maybe it was just a very poor choice of words. Statistical significance with 9 samples is suggestive at best. It just barely qualifies as a pilot study. They even say that significance means there's a large effect, which again... No. Significance in small sample indicates practically nothing by itself. What was the actual effect size? Though even at 0.8, with 9 samples I would be be very cautious in my optimistic.

SamoyedFurFluff 5y ago
I wonder if, read charitably, the intent is to say with a strong enough effect a small statistic size doesn’t necessarily mean useless. For example, if I removed the hearts from 9 living people and they all died, I can conclude people need their hearts to live.
- ineedasername 5y ago
  Yes, I try to allow some benefit of the doubt in my comment, but from a doctorate is statistics it's a bit harder to excuse. He doesn't have to make an argument from logical implication, there are perfectly good measures of effect size he could use that would allow him to say something like "not only was this statistically significant but it demonstrated a large effect size as well." My hope is he said something like "We not only saw a P value of < 0.04 but a Cohen's D of 0.7." and then the reporter was like, "yeah I'm gonna need you to dumb that down for our readers"
naasking 5y ago
> That is so wrong that, from a researcher with a doctorate in biostatistics, it's very difficult not to infer deliberate misinformation on their part. We don't really know the context for that statement. The journalist could have easily misunderstood something and used quoted him in a totally wrong context.
- ineedasername 5y ago
  I can give him a little benefit of the doubt, but if he said those specific words and it wasn't a misquote then it was an extremely poor choice of words. Those words, in that order, are so fundamentally wrong that I cannot imagine them being correct even in additional surrounding context unless the next few words that were cutoff were "...as measured by [insert preferred methodology for effect size]" that would still be poor phrasing, implying that significance led to effect size.

SamoyedFurFluff 5y ago

I wonder if, read charitably, the intent is to say with a strong enough effect a small statistic size doesn’t necessarily mean useless. For example, if I removed the hearts from 9 living people and they all died, I can conclude people need their hearts to live.

ineedasername 5y ago
Yes, I try to allow some benefit of the doubt in my comment, but from a doctorate is statistics it's a bit harder to excuse. He doesn't have to make an argument from logical implication, there are perfectly good measures of effect size he could use that would allow him to say something like "not only was this statistically significant but it demonstrated a large effect size as well." My hope is he said something like "We not only saw a P value of < 0.04 but a Cohen's D of 0.7." and then the reporter was like, "yeah I'm gonna need you to dumb that down for our readers"

naasking 5y ago

> That is so wrong that, from a researcher with a doctorate in biostatistics, it's very difficult not to infer deliberate misinformation on their part. We don't really know the context for that statement. The journalist could have easily misunderstood something and used quoted him in a totally wrong context.

ineedasername 5y ago
I can give him a little benefit of the doubt, but if he said those specific words and it wasn't a misquote then it was an extremely poor choice of words. Those words, in that order, are so fundamentally wrong that I cannot imagine them being correct even in additional surrounding context unless the next few words that were cutoff were "...as measured by [insert preferred methodology for effect size]" that would still be poor phrasing, implying that significance led to effect size.