4 ms·

> The problem in the research community is that scientists have little incentive to duplicate earlier work just to check if it’s correct. Many journals have exp

by sjackso 12y ago

> The problem in the research community is that scientists have little incentive to duplicate earlier work just to check if it’s correct. Many journals have explicit policies forbidding the publication of work that attempts to replicate previous experiments. This drive for unending novelty in the sciences is a shame on many levels. The good and useful work of duplicating and verifying results goes undone, and scientists are driven ever more forcefully towards designing studies only on the basis of what will attract grant agencies. Duplicating major results carefully would be useful to the scientific record. Trying things that will probably fail, and then publishing negative results, would be useful too. But, for most researchers, doing this useful work appears to be career suicide.

rgejman 12y ago
There are two problems, only one of which is addressed by duplication. 1. The original study may be plain wrong. Author 1 claims to describe phenomena X, author 2 can't find any evidence for X using the exact same methodology. 2. The original study is wrong because the methodology is bad, interpretation is incorrect, reagents are not specific, techniques are not clean, etc. There's a strong argument to be made that it is more important to do orthogonal work which indirectly verifies the original results, as opposed to simply trying to reproduce the original work. This solves both #1 and #2. The paper cited in the article is a great example. Gluing hair onto a plastic leg and putting it in a miniature wind tunnel is probably an exercise in bad methodology—but they might have still gotten the right result for their experiment. And that's useful because now we know that there may be something different about either plastic legs vs. real legs or miniature wind tunnels vs. big wind tunnels. This is a trite example, but it should be clear how this might be more important in other studies.
epistasis 12y ago
I've never encountered a field in science where fundamental results aren't sufficiently replicated by other labs. That nobody spent $500-$2000 to estimate leg hair drag probably speaks more about people not wanting to be caught measuring leg hair drag than about the entire field's tendency to replicate findings. It's also really weird for people to consider $500/hour tunnel time to be expensive for a sport with as much money as cycling. Really, I thought it was weird, and probably inappropriate, to mix in so much of an outsider's amateur and unsopported opinion about science into an otherwise interesting story about leg hair drag. If there's a serious story somewhere about results not getting replicated when they should in many fields of science, I'd like to hear about it. There's often stuff on the fringe, but any serious result will be replicated soon. Take, for example, the acid-bath stem cell retraction from not long ago. Where else are important results not replicated? The Higgs? It's a baffling perspective.
- thaumaturgy 12y ago
  > If there's a serious story somewhere about results not getting replicated when they should in many fields of science, I'd like to hear about it. Actually, there is! It's been referred to as "the decline effect", and it's fascinating. The New Yorker published an article about it a few years ago (http://www.newyorker.com/magazine/2010/12/13/the-truth-wears-off?currentPage=all http://www.newyorker.com/magazine/2010/12/13/the-truth-wears...). It has resurfaced in popularity a couple of times. It was written by disgraced journalist Jonah Lehrer, but later independent investigation failed to find any fault with the article he wrote (http://www.lastwordonnothing.com/2012/11/05/jonah-lehrer-nature-of-truth/ http://www.lastwordonnothing.com/2012/11/05/jonah-lehrer-nat... -- worth reading because it includes statements from scientists on Lehrer's article). So far, my favorite explanation is the direct, obvious one, which is that we're sometimes -- perhaps more often than we'd expect -- falling victim to statistics. Given the very large number of experiments and studies being conducted everywhere, and the selection for positive results, statistical anomalies are being accidentally selected for and then not being extensively enough re-evaluated much later. The article discusses that as a possible explanation, but the journalist does his job, igniting reader interest, by suggesting the effect somehow "defies the laws of statistics". I favor the simpler explanation, which is that our meat brains simply underestimate the vast number of ways in which statistics would like to bugger us. Some of the things that both you and the parent commenter mention are discussed in the article. For example: > Jennions, similarly, argues that the decline effect is largely a product of publication bias, or the tendency of scientists and scientific journals to prefer positive data over null results, which is what happens when no effect is found. and > Richard Palmer, a biologist at the University of Alberta, who has studied the problems surrounding fluctuating asymmetry, suspects that an equally significant issue is the selective reporting of results—the data that scientists choose to document in the first place. (aside: I'm not one of HN's vocal "science is junk" members. I love science, or, at least, I like to admire its butt (http://explosm.net/comics/3557/ http://explosm.net/comics/3557/). I don't believe there's something fundamentally wrong with the scientific method. But, I do think more emphasis needs to be placed on repeatability of experimental results over longer periods of time, and I hope that this will become more of a trend as the world economy continues to grow.)
  - 12y ago
Xcelerate 12y ago
I agree. You can't even publish negative results, although the data would be incredibly useful to the scientific community. The whole point of science is consistent replication to aid prediction. Some rich person needs to start a program incentivizing this kind of work.
- deleted 12y ago
  [deleted]
- ansgri 12y ago
  It seems more likely that some rich person already have duplicated some "key" discoveries and is happily reaping the benefits of knowing what does work, without making much noise of it.
baddox 12y ago
It seems like a drive for unending novelty would cause people to want to gather evidence that contradicts well-established studies. That's by definition novel, and would probably result in good press.
collyw 12y ago
I would like to think that there is a self correcting element to this. If someone produces genuinely high impact novelty science, others will want to build on that - and will end up replicating it anyway. If it isn't possible to replicate or isn't important, then it will be forgotten about. I agree about publishing negative results. The experiments that don't work, but might have had a high impact will likely be repeated, as no one knows it hasn't been tried.
- lutusp 12y ago
  > I agree about publishing negative results. If only science publishers would agree. But they're in a money-making business and novel results, "breakthrough" results, even if unverified and soon to be disproven, sell journals much more efficiently than null results or falsifications of earlier work.
Balgair 12y ago
https://en.wikipedia.org/wiki/John_P._A._Ioannidis https://en.wikipedia.org/wiki/John_P._A._Ioannidis "Ioannidis's 2005 paper "Why Most Published Research Findings Are False"[5] has been the most downloaded technical paper from the journal PLoS Medicine.[8] A profile of his work in this area appears in the November 2010 issue of The Atlantic.[9] The Atlantic article notes Ioannidis analyzed "49 of the most highly regarded research findings in medicine over the previous 13 years". In the paper Ioannidis compared the 45 studies that claimed to have uncovered effective interventions with data from subsequent studies with larger sample sizes: 7 (16%) of the studies were contradicted, 7 (16%) the effects were smaller than in the initial study and 31 (68%) of the studies remained either unchallenged or the findings could not be replicated.[5] Statisticians Goodman and Greenland agreed that "many medical research findings are less definitive than readers suspect" but disputed his headline claims as unsupportable by the methods used.[10][11] Ioannidis responded to this critique[12] and other researchers have generally supported the general thrust of his findings.[13][14] Ioannidis' work is focused on improving research design standards."

by sjackso 12y ago

rgejman 12y ago

There are two problems, only one of which is addressed by duplication. 1. The original study may be plain wrong. Author 1 claims to describe phenomena X, author 2 can't find any evidence for X using the exact same methodology. 2. The original study is wrong because the methodology is bad, interpretation is incorrect, reagents are not specific, techniques are not clean, etc. There's a strong argument to be made that it is more important to do orthogonal work which indirectly verifies the original results, as opposed to simply trying to reproduce the original work. This solves both #1 and #2. The paper cited in the article is a great example. Gluing hair onto a plastic leg and putting it in a miniature wind tunnel is probably an exercise in bad methodology—but they might have still gotten the right result for their experiment. And that's useful because now we know that there may be something different about either plastic legs vs. real legs or miniature wind tunnels vs. big wind tunnels. This is a trite example, but it should be clear how this might be more important in other studies.

epistasis 12y ago

I've never encountered a field in science where fundamental results aren't sufficiently replicated by other labs. That nobody spent $500-$2000 to estimate leg hair drag probably speaks more about people not wanting to be caught measuring leg hair drag than about the entire field's tendency to replicate findings. It's also really weird for people to consider $500/hour tunnel time to be expensive for a sport with as much money as cycling. Really, I thought it was weird, and probably inappropriate, to mix in so much of an outsider's amateur and unsopported opinion about science into an otherwise interesting story about leg hair drag. If there's a serious story somewhere about results not getting replicated when they should in many fields of science, I'd like to hear about it. There's often stuff on the fringe, but any serious result will be replicated soon. Take, for example, the acid-bath stem cell retraction from not long ago. Where else are important results not replicated? The Higgs? It's a baffling perspective.

thaumaturgy 12y ago
> If there's a serious story somewhere about results not getting replicated when they should in many fields of science, I'd like to hear about it. Actually, there is! It's been referred to as "the decline effect", and it's fascinating. The New Yorker published an article about it a few years ago (http://www.newyorker.com/magazine/2010/12/13/the-truth-wears-off?currentPage=all http://www.newyorker.com/magazine/2010/12/13/the-truth-wears...). It has resurfaced in popularity a couple of times. It was written by disgraced journalist Jonah Lehrer, but later independent investigation failed to find any fault with the article he wrote (http://www.lastwordonnothing.com/2012/11/05/jonah-lehrer-nature-of-truth/ http://www.lastwordonnothing.com/2012/11/05/jonah-lehrer-nat... -- worth reading because it includes statements from scientists on Lehrer's article). So far, my favorite explanation is the direct, obvious one, which is that we're sometimes -- perhaps more often than we'd expect -- falling victim to statistics. Given the very large number of experiments and studies being conducted everywhere, and the selection for positive results, statistical anomalies are being accidentally selected for and then not being extensively enough re-evaluated much later. The article discusses that as a possible explanation, but the journalist does his job, igniting reader interest, by suggesting the effect somehow "defies the laws of statistics". I favor the simpler explanation, which is that our meat brains simply underestimate the vast number of ways in which statistics would like to bugger us. Some of the things that both you and the parent commenter mention are discussed in the article. For example: > Jennions, similarly, argues that the decline effect is largely a product of publication bias, or the tendency of scientists and scientific journals to prefer positive data over null results, which is what happens when no effect is found. and > Richard Palmer, a biologist at the University of Alberta, who has studied the problems surrounding fluctuating asymmetry, suspects that an equally significant issue is the selective reporting of results—the data that scientists choose to document in the first place. (aside: I'm not one of HN's vocal "science is junk" members. I love science, or, at least, I like to admire its butt (http://explosm.net/comics/3557/ http://explosm.net/comics/3557/). I don't believe there's something fundamentally wrong with the scientific method. But, I do think more emphasis needs to be placed on repeatability of experimental results over longer periods of time, and I hope that this will become more of a trend as the world economy continues to grow.)
- 12y ago

Xcelerate 12y ago

I agree. You can't even publish negative results, although the data would be incredibly useful to the scientific community. The whole point of science is consistent replication to aid prediction. Some rich person needs to start a program incentivizing this kind of work.

deleted 12y ago
[deleted]
ansgri 12y ago
It seems more likely that some rich person already have duplicated some "key" discoveries and is happily reaping the benefits of knowing what does work, without making much noise of it.

baddox 12y ago

It seems like a drive for unending novelty would cause people to want to gather evidence that contradicts well-established studies. That's by definition novel, and would probably result in good press.

collyw 12y ago

I would like to think that there is a self correcting element to this. If someone produces genuinely high impact novelty science, others will want to build on that - and will end up replicating it anyway. If it isn't possible to replicate or isn't important, then it will be forgotten about. I agree about publishing negative results. The experiments that don't work, but might have had a high impact will likely be repeated, as no one knows it hasn't been tried.

lutusp 12y ago
> I agree about publishing negative results. If only science publishers would agree. But they're in a money-making business and novel results, "breakthrough" results, even if unverified and soon to be disproven, sell journals much more efficiently than null results or falsifications of earlier work.

Balgair 12y ago

https://en.wikipedia.org/wiki/John_P._A._Ioannidis https://en.wikipedia.org/wiki/John_P._A._Ioannidis "Ioannidis's 2005 paper "Why Most Published Research Findings Are False"[5] has been the most downloaded technical paper from the journal PLoS Medicine.[8] A profile of his work in this area appears in the November 2010 issue of The Atlantic.[9] The Atlantic article notes Ioannidis analyzed "49 of the most highly regarded research findings in medicine over the previous 13 years". In the paper Ioannidis compared the 45 studies that claimed to have uncovered effective interventions with data from subsequent studies with larger sample sizes: 7 (16%) of the studies were contradicted, 7 (16%) the effects were smaller than in the initial study and 31 (68%) of the studies remained either unchallenged or the findings could not be replicated.[5] Statisticians Goodman and Greenland agreed that "many medical research findings are less definitive than readers suspect" but disputed his headline claims as unsupportable by the methods used.[10][11] Ioannidis responded to this critique[12] and other researchers have generally supported the general thrust of his findings.[13][14] Ioannidis' work is focused on improving research design standards."