6 ms·

I see a lot of people commenting here that there's no incentive to repeat previous research because it's not useful for getting grants, etc. This kind of true b

by mmierz 10y ago

I see a lot of people commenting here that there's no incentive to repeat previous research because it's not useful for getting grants, etc. This kind of true but I think it misses something important. At least in life sciences (can't comment on other fields), it's not that scientists don't repeat each other's results. After all, if you're going to invest a significant fraction of your tiny lab budget on a research project, you need to make sure that the basic premise is sound, so it's not uncommon that the first step is to confirm the previous published result before continuing. And if the replication fails, it's obviously not a wise idea to proceed with a project that relies on the prior result. But that work never makes it into a paper. If the replication succeeds, great! Proceed with the project. But it's time-consuming and expensive to make the reproduction publication worthy, so it will probably get buried in a data supplement if it's published at all. If the replication fails, it's even more time-consuming and expensive to convincingly demonstrate the negative result. Moreover, the work is being done by an ambitious student or postdoc who is staring down a horrible job market and needs novel results and interesting publications in order to have a future in science. Why would someone like that spend a year attacking the work of an established scientist over an uninteresting and possibly wrong negative result, and getting a crappy paper and an enemy out of it in the end, instead of planning for their own future? If enough people fail to replicate a result, it becomes "common knowledge" in the field that the result is wrong, and it kind of fades away. But it's not really in anyone's interest to write an explicit rebuttal, so it never happens.

untilHellbanned 10y ago
Yes. Life scientist here. This is 100% on point. It seems that most people who comment that we need 100% published repetition aren't aware of how the process currently works as you describe it. Repetition happens regularly if not in the majority of published research. It's just that it often doesn't get written up in a way that one can easily see the connection with the previous work. Citations are supposed to help this problem but because citations aren't just links and are hidden behind paywalls (a whole different can of worms), its not easy to see what's been replicated vs. not.
refurb 10y ago
As a former life scientist, I completely agree. If something truly revolutionary is published and it's relevant to your own work, it's very common to repeat the experiment yourself. If it works, great, continue on with your own work and reference the original paper when you publish new results. If it doesn't work, the outcome can range from: (1) mentally note and move onto an alternative if they are available to (2) spend a ton of time trying to "fix" it, including contacting the original author. The only time you see reproducibility experiments published is if the original paper made some serious errors, including outright fraud. An there is no way you'd be able to test the reproducibility of every paper because some papers aren't that important. The important ones get the attention and if you see multiple scientists referencing a work (and building off of it), you can have confidence it actually works.
- vibrio 10y ago
  As a former life scientist, I completely agree also. Things do get repeated. If the findings aren't robust, scientists cut bait and move on. This knowledge is shared informally, for certian. This "Formalzation" of the experimental repetition would be difficult since no one wants grant money or their careers spent generating "yeah, s/he's right about that' outcomes.
odbol_ 10y ago
This is why I'm always skeptical of people zealously claiming "all GMOs are perfectly safe! It's been proven!" Yeah, proven by expensive studies that were funded by the company making the GMO. Who is going to pay for another study to try to disprove it? There's a reason we sprayed DET all over our vegetables for years before it was banned: there was no scientific studies proving that it was harmful, even though it clearly was harmful in hindsight. Science is not instant, and there's no way someone can claim that some brand-new GMO is "perfectly safe", without any long-term studies on its effects over 10, 20, 30 years of exposure. That's just not possible. And yet you try to explain it to these science zealots and they just brush you off as being "anti-science".
- Obi_Juan_Kenobi 10y ago
  Do you mean DDT? Anyway, beyond the 'philosophy of science' issue of whether you can prove something, there is good affirmative evidence that existing GMOs are safe for numerous reasons. First, there's no mechanistic reason to think they would be dangerous. T-DNAs are not somehow magically toxic to humans; everything you eat is riddled with millennia of t-dnas, old viruses, transposon blooms, etc. etc. The technologies themselves should be safe as well. BT is well understood and embraced by the organic community as an applied natural pesticide, so you would need to find evidence that the localization somehow makes it toxic. Glyphosate resistance is also unlikely to have any effect a priori because it affects a metabolic pathway completely absent in animals. Argue all you like about how nothing can be 'perfectly safe', sure, but there's no reason to think that GMOs are dangerous, and people have looked quite hard. ______ Finally, just look at the Seralini pile-of-bullshit for evidence that there's plenty of incentive to publish anything critical of GMOs. No one is sitting on career-making evidence. https://en.wikipedia.org/wiki/S%C3%A9ralini_affair https://en.wikipedia.org/wiki/S%C3%A9ralini_affair
  - odbol_ 10y ago
    > First, there's no mechanistic reason to think they would be dangerous. That's like saying "there's no reason to think peanuts would be dangerous" since humans eat them all the time. And yet, they are deadly to some humans. No one knows why. But they do know that food allergies are far more prevalent in the U.S. than in other countries. And now, suddenly, people in Africa and China are starting to exhibit food allergies that the U.S. has had for a while. So what have we started shipping over to them that's causing these allergies? Who will fund that study?
eanzenberg 10y ago
>>If the replication fails, it's even more time-consuming and expensive to convincingly demonstrate the negative result. Said scientist will gain immense fame and recognition if they show established ideas are wrong. (Source: academia Physicist)
- Obi_Juan_Kenobi 10y ago
  I think your's and the parent comment are both right, just addressing different extremes. Sometimes you find evidence that doesn't quite match up with previous work, but if an explanatory framework isn't forthcoming, it can be difficult to say who's results are right, and what they mean. On the other hand, sometimes you come across something blatantly wrong, so you do your follow up experiments to confirm and you're all set. It's only the latter case that will quickly make it into a manuscript. I will say, I've seen some big corrections (like this-allele-was-actually-a-completely-different-mutant bad) that just get buried in the results section of a subsequent publication, and no retraction was ever submitted for the original paper. That was definitely a failure in the field, and likely the result of the status of the authors involved.
- tdaltonc 10y ago
  You only get famous if you produce a new order that replaces the old one. Throwing stones at the old order doesn't actually get you very much unless you provide something better. Especially in psych/bio, a negative finding can almost always be blamed on the experimentalist -- even more so if they are young. It is no great feat to conduct a study so poorly that fail to detect a signal others have reported.
fabian2k 10y ago
It's a bit redundant now, but it's a point that deserves repetition. Replication happens by the scientists that want to build on the original work and advance it. Not because they simply want to replicate it, but simply to be sure that you understand the method and all the practical details that are necessary. If there is only a single paper about the result, most scientists will regard that with a certain suspicion. That doesn't mean we expect every new thing to be fake, but there are just too many ways these experiments can go wrong. One problem is certainly that the negative reproduction of experiments are more often communicated informally, and not published. If you're active in the field you will likely hear from the scientists that tried to reproduce the results without any success. But most of these are not published. Any experiment worth replicating, is usually also worth expanding and building on, and that is where the actual verification is taking place. Science works in this regard, but it doesn't work as cleanly and efficiently as one could hope for.
texthompson 10y ago
Why is it not in anyone's interest to write an explicit rebuttal?

by mmierz 10y ago

untilHellbanned 10y ago

Yes. Life scientist here. This is 100% on point. It seems that most people who comment that we need 100% published repetition aren't aware of how the process currently works as you describe it. Repetition happens regularly if not in the majority of published research. It's just that it often doesn't get written up in a way that one can easily see the connection with the previous work. Citations are supposed to help this problem but because citations aren't just links and are hidden behind paywalls (a whole different can of worms), its not easy to see what's been replicated vs. not.

refurb 10y ago

As a former life scientist, I completely agree. If something truly revolutionary is published and it's relevant to your own work, it's very common to repeat the experiment yourself. If it works, great, continue on with your own work and reference the original paper when you publish new results. If it doesn't work, the outcome can range from: (1) mentally note and move onto an alternative if they are available to (2) spend a ton of time trying to "fix" it, including contacting the original author. The only time you see reproducibility experiments published is if the original paper made some serious errors, including outright fraud. An there is no way you'd be able to test the reproducibility of every paper because some papers aren't that important. The important ones get the attention and if you see multiple scientists referencing a work (and building off of it), you can have confidence it actually works.

vibrio 10y ago
As a former life scientist, I completely agree also. Things do get repeated. If the findings aren't robust, scientists cut bait and move on. This knowledge is shared informally, for certian. This "Formalzation" of the experimental repetition would be difficult since no one wants grant money or their careers spent generating "yeah, s/he's right about that' outcomes.

odbol_ 10y ago

This is why I'm always skeptical of people zealously claiming "all GMOs are perfectly safe! It's been proven!" Yeah, proven by expensive studies that were funded by the company making the GMO. Who is going to pay for another study to try to disprove it? There's a reason we sprayed DET all over our vegetables for years before it was banned: there was no scientific studies proving that it was harmful, even though it clearly was harmful in hindsight. Science is not instant, and there's no way someone can claim that some brand-new GMO is "perfectly safe", without any long-term studies on its effects over 10, 20, 30 years of exposure. That's just not possible. And yet you try to explain it to these science zealots and they just brush you off as being "anti-science".

Obi_Juan_Kenobi 10y ago
Do you mean DDT? Anyway, beyond the 'philosophy of science' issue of whether you can prove something, there is good affirmative evidence that existing GMOs are safe for numerous reasons. First, there's no mechanistic reason to think they would be dangerous. T-DNAs are not somehow magically toxic to humans; everything you eat is riddled with millennia of t-dnas, old viruses, transposon blooms, etc. etc. The technologies themselves should be safe as well. BT is well understood and embraced by the organic community as an applied natural pesticide, so you would need to find evidence that the localization somehow makes it toxic. Glyphosate resistance is also unlikely to have any effect a priori because it affects a metabolic pathway completely absent in animals. Argue all you like about how nothing can be 'perfectly safe', sure, but there's no reason to think that GMOs are dangerous, and people have looked quite hard. ______ Finally, just look at the Seralini pile-of-bullshit for evidence that there's plenty of incentive to publish anything critical of GMOs. No one is sitting on career-making evidence. https://en.wikipedia.org/wiki/S%C3%A9ralini_affair https://en.wikipedia.org/wiki/S%C3%A9ralini_affair
- odbol_ 10y ago
  > First, there's no mechanistic reason to think they would be dangerous. That's like saying "there's no reason to think peanuts would be dangerous" since humans eat them all the time. And yet, they are deadly to some humans. No one knows why. But they do know that food allergies are far more prevalent in the U.S. than in other countries. And now, suddenly, people in Africa and China are starting to exhibit food allergies that the U.S. has had for a while. So what have we started shipping over to them that's causing these allergies? Who will fund that study?

eanzenberg 10y ago

>>If the replication fails, it's even more time-consuming and expensive to convincingly demonstrate the negative result. Said scientist will gain immense fame and recognition if they show established ideas are wrong. (Source: academia Physicist)

Obi_Juan_Kenobi 10y ago
I think your's and the parent comment are both right, just addressing different extremes. Sometimes you find evidence that doesn't quite match up with previous work, but if an explanatory framework isn't forthcoming, it can be difficult to say who's results are right, and what they mean. On the other hand, sometimes you come across something blatantly wrong, so you do your follow up experiments to confirm and you're all set. It's only the latter case that will quickly make it into a manuscript. I will say, I've seen some big corrections (like this-allele-was-actually-a-completely-different-mutant bad) that just get buried in the results section of a subsequent publication, and no retraction was ever submitted for the original paper. That was definitely a failure in the field, and likely the result of the status of the authors involved.
tdaltonc 10y ago
You only get famous if you produce a new order that replaces the old one. Throwing stones at the old order doesn't actually get you very much unless you provide something better. Especially in psych/bio, a negative finding can almost always be blamed on the experimentalist -- even more so if they are young. It is no great feat to conduct a study so poorly that fail to detect a signal others have reported.

fabian2k 10y ago

It's a bit redundant now, but it's a point that deserves repetition. Replication happens by the scientists that want to build on the original work and advance it. Not because they simply want to replicate it, but simply to be sure that you understand the method and all the practical details that are necessary. If there is only a single paper about the result, most scientists will regard that with a certain suspicion. That doesn't mean we expect every new thing to be fake, but there are just too many ways these experiments can go wrong. One problem is certainly that the negative reproduction of experiments are more often communicated informally, and not published. If you're active in the field you will likely hear from the scientists that tried to reproduce the results without any success. But most of these are not published. Any experiment worth replicating, is usually also worth expanding and building on, and that is where the actual verification is taking place. Science works in this regard, but it doesn't work as cleanly and efficiently as one could hope for.

texthompson 10y ago

Why is it not in anyone's interest to write an explicit rebuttal?