4 ms·

It annoys me when news stories don't link to their sources. Here it is: http://onlinelibrary.wiley.com/doi/10.1111/j.1365-2605.2009.01019.x/abstract http://onli

by stenl 11y ago

It annoys me when news stories don't link to their sources. Here it is: http://onlinelibrary.wiley.com/doi/10.1111/j.1365-2605.2009.01019.x/abstract http://onlinelibrary.wiley.com/doi/10.1111/j.1365-2605.2009.... The study is terrible. All the reported effects include 0 well within the 95% confidence interval (e.g. ranges like -2.3 to +3.2), yet mysteriously the P values end up just the right side of "significant". This reeks of confirmation bias, or worse.

refurb 11y ago
I'm glad someone looked up the actual paper. Not surprising that the findings were weak.
stdbrouw 11y ago
Well, no, there are a couple of intervals that don't include 0 and those are the ones they report. Other than that, this does look like a pretty poor study. They do that classic (but misguided) thing where they say that their study was conducted on a small sample and so the fact that they found anything at all is super-duper-extra telling. In reality, it just makes it more likely that any effect that is reported will be exaggerated – as was brilliantly pointed out by Andrew Gelman and John Carlin in http://www.stat.columbia.edu/~gelman/research/published/retropower_final.pdf http://www.stat.columbia.edu/~gelman/research/published/retr....
- carbocation 11y ago
  Accounting for multiple testing by Bonferroni, none of the results is significant (at least, none reported in the abstract).
rcthompson 11y ago
And since they're doing multiple tests, 0.05 isn't even a stringent enough threshold: https://xkcd.com/882/ https://xkcd.com/882/
- chimeracoder 11y ago
  Actually, it's worse than that - theoretically, they should be doing an F-test, but those are essentially useless[0] except as exercises for unfortunate undergraduates. This gets into the flaws with hypothesis testing (and p-values) as a tool altogether, though. And while I would love to see the natural sciences eschew these rather rudimentary heuristics in favor of the more sophisticated modeling tools that we already have, I'm not holding my breath on that happening anytime soon. [0] http://andrewgelman.com/2009/05/18/noooooooooooooo/ http://andrewgelman.com/2009/05/18/noooooooooooooo/
formulaT 11y ago
I couldn't see the article, only the abstract. But what they report in the abstract is some regression coefficients for a "masculinity score" and associated p-values. A very simple analysis, and not much that can go wrong there. My only concern is with cherry picking the chemicals. It wasn't clear if they were only reporting specific chemicals that had significant relationships, or if all the chemicals that would be expected to have a significant relationship did.

by stenl 11y ago

refurb 11y ago

I'm glad someone looked up the actual paper. Not surprising that the findings were weak.

stdbrouw 11y ago

Well, no, there are a couple of intervals that don't include 0 and those are the ones they report. Other than that, this does look like a pretty poor study. They do that classic (but misguided) thing where they say that their study was conducted on a small sample and so the fact that they found anything at all is super-duper-extra telling. In reality, it just makes it more likely that any effect that is reported will be exaggerated – as was brilliantly pointed out by Andrew Gelman and John Carlin in http://www.stat.columbia.edu/~gelman/research/published/retropower_final.pdf http://www.stat.columbia.edu/~gelman/research/published/retr....

carbocation 11y ago
Accounting for multiple testing by Bonferroni, none of the results is significant (at least, none reported in the abstract).

rcthompson 11y ago

And since they're doing multiple tests, 0.05 isn't even a stringent enough threshold: https://xkcd.com/882/ https://xkcd.com/882/

chimeracoder 11y ago
Actually, it's worse than that - theoretically, they should be doing an F-test, but those are essentially useless[0] except as exercises for unfortunate undergraduates. This gets into the flaws with hypothesis testing (and p-values) as a tool altogether, though. And while I would love to see the natural sciences eschew these rather rudimentary heuristics in favor of the more sophisticated modeling tools that we already have, I'm not holding my breath on that happening anytime soon. [0] http://andrewgelman.com/2009/05/18/noooooooooooooo/ http://andrewgelman.com/2009/05/18/noooooooooooooo/

formulaT 11y ago

I couldn't see the article, only the abstract. But what they report in the abstract is some regression coefficients for a "masculinity score" and associated p-values. A very simple analysis, and not much that can go wrong there. My only concern is with cherry picking the chemicals. It wasn't clear if they were only reporting specific chemicals that had significant relationships, or if all the chemicals that would be expected to have a significant relationship did.