5 ms·

Just curious, isn't this kind of basic legitimacy verification supposed to be done by the peer reviewing process? Are scientific journals really publishing pape

by mbrock 10y ago

Just curious, isn't this kind of basic legitimacy verification supposed to be done by the peer reviewing process? Are scientific journals really publishing papers so obviously illegitimate that laymen with a bit of statistical knowledge can spot obvious errors or problems?

veddox 10y ago
Unfortunately, yes. Of course a lot of it depends on the journal, but there are enough out there that will publish just about anything. Even the more serious journals sometimes let "bad apples" slip through. In the end, peer reviewers are only humans too, make mistakes, have a lot of other work to do, and don't even get paid for their review work. Recently we had an extensive discussion on HN about problems facing science as an institution (based on this article: http://www.vox.com/2016/7/14/12016710/science-challeges-research-funding-peer-review-process http://www.vox.com/2016/7/14/12016710/science-challeges-rese...). Peer review was the fourth point on their list of problems.
- ahazred8ta 10y ago
  Yup, that's https://news.ycombinator.com/item?id=12114551 https://news.ycombinator.com/item?id=12114551 --- "Big Problems Facing Science" (vox.com)
dagw 10y ago
Things like a small sample size and confounding variable, doesn't in any way invalidate a paper. Especially in fields where collecting data is difficult (only a small number of people have a certain obscure disease for example, or it's practically impossible to isolate one variable from another) those are simply facts of life that you have to deal with. Sometimes you all you can do is say I only managed to find 4 cases of X, but they all had Y. Was not able to control for Z, still I wasn't expecting that and found it kind of interesting. As long as you aren't faking data (like you actually found 12 cases of X but threw out the ones that didn't fit your conclusion), then that is absolutely worth publishing. It's always worth getting results out there even if they don't/can't live up to statistical certainty one might hope for. Every paper doesn't have to stand on its own and come to some undeniable ground breaking conclusion. Over time you'll hopefully collect enough published data to be able to do some more statistically powerful and useful meta-studies down the road.
- veddox 10y ago
  Absolutely. Especially in the life sciences it can be very hard to get statistically solid data. None of which means there isn't a lot of statistics crap floating around, though.
pgt 10y ago
Hi @mbrock. Take, for example, the acetaminophen study[^1] currently on the front page claiming an association between prenatal use of the drug and behavioural and emotional problems in children. @fifteenforty works in this area and immediately pointed out[^2] the potential cause not mentioned in the abstract: why were these women taking painkillers in the first place and could that be the cause of the developmental problems? Further down, in the Results & Conclusions, "association" implies "causal" when they mention "via an intrauterine mechanism?" [^1]: https://news.ycombinator.com/item?id=12293675 https://news.ycombinator.com/item?id=12293675 [^2]: @fifteenforty comment, https://news.ycombinator.com/item?id=12295107 https://news.ycombinator.com/item?id=12295107 Link to original study: http://archpedi.jamanetwork.com/article.aspx?articleid=2543281 http://archpedi.jamanetwork.com/article.aspx?articleid=25432... LA Times coverage: http://www.latimes.com/science/sciencenow/la-sci-sn-acetaminophen-pregnancy-behavior-problems-20160815-snap-story.html http://www.latimes.com/science/sciencenow/la-sci-sn-acetamin...

by mbrock 10y ago

veddox 10y ago

Unfortunately, yes. Of course a lot of it depends on the journal, but there are enough out there that will publish just about anything. Even the more serious journals sometimes let "bad apples" slip through. In the end, peer reviewers are only humans too, make mistakes, have a lot of other work to do, and don't even get paid for their review work. Recently we had an extensive discussion on HN about problems facing science as an institution (based on this article: http://www.vox.com/2016/7/14/12016710/science-challeges-research-funding-peer-review-process http://www.vox.com/2016/7/14/12016710/science-challeges-rese...). Peer review was the fourth point on their list of problems.

ahazred8ta 10y ago
Yup, that's https://news.ycombinator.com/item?id=12114551 https://news.ycombinator.com/item?id=12114551 --- "Big Problems Facing Science" (vox.com)

dagw 10y ago

Things like a small sample size and confounding variable, doesn't in any way invalidate a paper. Especially in fields where collecting data is difficult (only a small number of people have a certain obscure disease for example, or it's practically impossible to isolate one variable from another) those are simply facts of life that you have to deal with. Sometimes you all you can do is say I only managed to find 4 cases of X, but they all had Y. Was not able to control for Z, still I wasn't expecting that and found it kind of interesting. As long as you aren't faking data (like you actually found 12 cases of X but threw out the ones that didn't fit your conclusion), then that is absolutely worth publishing. It's always worth getting results out there even if they don't/can't live up to statistical certainty one might hope for. Every paper doesn't have to stand on its own and come to some undeniable ground breaking conclusion. Over time you'll hopefully collect enough published data to be able to do some more statistically powerful and useful meta-studies down the road.

veddox 10y ago
Absolutely. Especially in the life sciences it can be very hard to get statistically solid data. None of which means there isn't a lot of statistics crap floating around, though.

pgt 10y ago

Hi @mbrock. Take, for example, the acetaminophen study[^1] currently on the front page claiming an association between prenatal use of the drug and behavioural and emotional problems in children. @fifteenforty works in this area and immediately pointed out[^2] the potential cause not mentioned in the abstract: why were these women taking painkillers in the first place and could that be the cause of the developmental problems? Further down, in the Results & Conclusions, "association" implies "causal" when they mention "via an intrauterine mechanism?" [^1]: https://news.ycombinator.com/item?id=12293675 https://news.ycombinator.com/item?id=12293675 [^2]: @fifteenforty comment, https://news.ycombinator.com/item?id=12295107 https://news.ycombinator.com/item?id=12295107 Link to original study: http://archpedi.jamanetwork.com/article.aspx?articleid=2543281 http://archpedi.jamanetwork.com/article.aspx?articleid=25432... LA Times coverage: http://www.latimes.com/science/sciencenow/la-sci-sn-acetaminophen-pregnancy-behavior-problems-20160815-snap-story.html http://www.latimes.com/science/sciencenow/la-sci-sn-acetamin...