5 ms·

You should know that there are a million of tiny ways for an experiment to "fail", requiring one do repeat it. Reagents could be bad, a machine could have broke

by mbreese 11y ago

You should know that there are a million of tiny ways for an experiment to "fail", requiring one do repeat it. Reagents could be bad, a machine could have broken mid-cycle, a positive (or negative) control could have been wrong... Basically meaning, "it didn't work". In this case, any "data" that would have gotten would be incredibly suspect and you'd need to repeat the experiment. In the vast majority of labs, this is nothing nefarious, it's the way science is done. If something fails, you try to figure out why it failed, try to fix the issue, and then repeat the experiment. Only once everything works correctly can you get valid data to then try and interpret. And if you get an interesting result, you still need to repeat the experiment 2-3 times to be sure. Repeating failed experiments isn't an issue - and has nothing to do with alchemy. It's just basic troubleshooting.

TisButMe 11y ago
I absolutely agree that sometimes, you need to redo an experiment for good reasons. In most cases I've seen, people do not know why they redo the experiment, though. They know it hasn't produced the data they expected, so they redo it. Maybe it was because a reagent was bad, or a co-worker left the incubator open overnight, or maybe it was because the model is stupid. Who knows? That's my point, actually. Biologists are playing with systems they do not understand, changing parameters somewhat randomly without any control over them, and they then try to interpret whatever comes out, but ONLY it fits what they wanted. If it doesn't, then "Oh, the PCR machine is at it again!", and they throw the results away.
- mbreese 11y ago
  It seems like you had a really bad experience in lab. I'm sorry for that. But it's a mistake to paint the entire field in a negative light because of this. Not all labs are bad, and some produce really outstanding work. Sometimes the issue is the PCR machine. Sometimes it's the water (my favorite troubleshooting experience from grad school). And figuring out where the issues are (is the the protocol? the reagents? or is this real signal?) can be difficult. Playing with systems we don't understand is kinda the point.
  - chris_wot 11y ago
    But then you risk thinking you understand the system, but you actually don't. How can you be sure the equipment failure didn't make you see something that wasn't there?
    - mbreese 11y ago
      That's why you repeat the experiment to confirm your results. A result due to a sporadic equipment failure likely won't be reproducible. If you can't replicate your own results, how could you expect anyone else to be able to?
  - TisButMe 11y ago
    I've been lucky enough to work in outstanding labs, with people published in Nature, and other journals of that quality. I've worked in 2 countries, and for 4 different labs. I've also talked with people from all over the world, who have worked everywhere, from Harvard to the Pasteur Institute to Cambridge University. The stories are all the same. I hoped I would find some place where people were trying to do things the right way, but what I found is that currently, you don't need to to be published in top journals, so why bother? It's really refreshing to hear you talk about trying to troubleshoot why an experiment didn't work the way you expected, I hear mostly of people retrying blindly until it "succeeds". What did you do with what you learned with the water causing the failure? Did you publish this, so that someone (or you!) could try to figure out why water was a problem, or at least so that no one would have the same issue? This is the other point: when people do bother about finding about why things fail, I've never seen any of them try to follow up on that, and figure out not only what made it fail, but why it made it fail. "Yeah, the annealing temp was not the right one". Ok, but why? Of course playing with systems we don't understand is the point, but we have to be very careful about them. We should be varying 1 parameter at a time. This is mostly impossible in biology, but right now we're not even trying to do anything about it.
    - nycticorax 11y ago
      People don't investigate things like that, because, frankly, no one cares. Nor should they. If your computer is acting funny, and you find out a spider has made a nest inside it, and it works fine once you clear out the spider nest, would you then decide to determine exactly why that spider nest, in that place, causes the exact problems you observed? No, of course not. Because that's not an interesting question. Computers are complicated machines, and they can break in lots of different ways, most of which are not very interesting in their details. Similarly, suppose you study cultured cells (which are notoriously finicky), and you want to compare what the cells do in the presence of drug X vs control. But at first, you find that all of your control cells die. And eventually you find out that if you use brand X of bottled water vs tap water, the control cells thrive. Are you seriously proposing that you should then drop all work on drug X, and get to work determining exactly what is up with the tap water in your town? I mean, maybe that would be a fruitful research avenue, if you're worried that the tap water isn't safe for human consumption, or you think that there's something interesting about exactly how the tap water is killing your control cells. But most of the time, investigating the tap water would be an expensive distraction from the question you actually want to answer. And most scientists, I think, would (reasonably) decide to get on with investigating the effects of drug X on their cells, and not worry too much about precisely why the tap water killed the control cells. And I don't think there's anything wrong with that. Life is short, and you have to choose your questions carefully.
jballanc 11y ago
The issue is there is "failed" and then there is "failed". Yes, many times you have to repeat an experiment because of bad reagents, broken machines, little tweaks are needed to some obscure parameter, or someone left the lab door open... However, if you experiment is well controlled, then the controls will reveal this sort of "failure". When I was still running gels, many times when first running a new setup we'd run only controls. If your experiment fails because your controls failed, then that's just science. But I've also seen the other kind of "failure". The kind where the controls came out perfectly, but the effect or the association or the expression profile that the researcher was hoping for didn't show up. When these sorts of failures are ignored or discarded, then we do science a huge disservice. I am encouraged, though, that there recently seems to be a movement toward, if not outright publishing such negative results, then at least archiving them and sharing them with others in the field. After all, without Michelson and Morley's "failure" we might not have special relativity.
- akshatpradhan 11y ago
  >But I've also seen the other kind of "failure". The kind where the controls came out perfectly, but the effect or the association or the expression profile that the researcher was hoping for didn't show up. When these sorts of failures are ignored or discarded, then we do science a huge disservice. This.
- mrhhug 11y ago
  >But I've also seen the other kind of "failure". The kind where the controls came out perfectly, but the effect or the association or the expression profile that the researcher was hoping for didn't show up. When these sorts of failures are ignored or discarded, then we do science a huge disservice. Why does this happen? Clearly this is what the article insinuates. Is publish or perish that strong? Every honest experiment with honest results benefits society. Not every prediction and result combination results in a prize in your lifetime, but that in no way should influence someone's value as a scientist. That science may be used later for something we had not intended (could i offer you the hope of posthumus recognition?). Finding a way it does not work may save someone else some time. This benefits the scientific community. Not everyone gets to walk on another planet, some people have to build the ship.

by mbreese 11y ago

TisButMe 11y ago

I absolutely agree that sometimes, you need to redo an experiment for good reasons. In most cases I've seen, people do not know why they redo the experiment, though. They know it hasn't produced the data they expected, so they redo it. Maybe it was because a reagent was bad, or a co-worker left the incubator open overnight, or maybe it was because the model is stupid. Who knows? That's my point, actually. Biologists are playing with systems they do not understand, changing parameters somewhat randomly without any control over them, and they then try to interpret whatever comes out, but ONLY it fits what they wanted. If it doesn't, then "Oh, the PCR machine is at it again!", and they throw the results away.

mbreese 11y ago
It seems like you had a really bad experience in lab. I'm sorry for that. But it's a mistake to paint the entire field in a negative light because of this. Not all labs are bad, and some produce really outstanding work. Sometimes the issue is the PCR machine. Sometimes it's the water (my favorite troubleshooting experience from grad school). And figuring out where the issues are (is the the protocol? the reagents? or is this real signal?) can be difficult. Playing with systems we don't understand is kinda the point.
- chris_wot 11y ago
  But then you risk thinking you understand the system, but you actually don't. How can you be sure the equipment failure didn't make you see something that wasn't there?
  - mbreese 11y ago
    That's why you repeat the experiment to confirm your results. A result due to a sporadic equipment failure likely won't be reproducible. If you can't replicate your own results, how could you expect anyone else to be able to?
- TisButMe 11y ago
  I've been lucky enough to work in outstanding labs, with people published in Nature, and other journals of that quality. I've worked in 2 countries, and for 4 different labs. I've also talked with people from all over the world, who have worked everywhere, from Harvard to the Pasteur Institute to Cambridge University. The stories are all the same. I hoped I would find some place where people were trying to do things the right way, but what I found is that currently, you don't need to to be published in top journals, so why bother? It's really refreshing to hear you talk about trying to troubleshoot why an experiment didn't work the way you expected, I hear mostly of people retrying blindly until it "succeeds". What did you do with what you learned with the water causing the failure? Did you publish this, so that someone (or you!) could try to figure out why water was a problem, or at least so that no one would have the same issue? This is the other point: when people do bother about finding about why things fail, I've never seen any of them try to follow up on that, and figure out not only what made it fail, but why it made it fail. "Yeah, the annealing temp was not the right one". Ok, but why? Of course playing with systems we don't understand is the point, but we have to be very careful about them. We should be varying 1 parameter at a time. This is mostly impossible in biology, but right now we're not even trying to do anything about it.
  - nycticorax 11y ago
    People don't investigate things like that, because, frankly, no one cares. Nor should they. If your computer is acting funny, and you find out a spider has made a nest inside it, and it works fine once you clear out the spider nest, would you then decide to determine exactly why that spider nest, in that place, causes the exact problems you observed? No, of course not. Because that's not an interesting question. Computers are complicated machines, and they can break in lots of different ways, most of which are not very interesting in their details. Similarly, suppose you study cultured cells (which are notoriously finicky), and you want to compare what the cells do in the presence of drug X vs control. But at first, you find that all of your control cells die. And eventually you find out that if you use brand X of bottled water vs tap water, the control cells thrive. Are you seriously proposing that you should then drop all work on drug X, and get to work determining exactly what is up with the tap water in your town? I mean, maybe that would be a fruitful research avenue, if you're worried that the tap water isn't safe for human consumption, or you think that there's something interesting about exactly how the tap water is killing your control cells. But most of the time, investigating the tap water would be an expensive distraction from the question you actually want to answer. And most scientists, I think, would (reasonably) decide to get on with investigating the effects of drug X on their cells, and not worry too much about precisely why the tap water killed the control cells. And I don't think there's anything wrong with that. Life is short, and you have to choose your questions carefully.

jballanc 11y ago

The issue is there is "failed" and then there is "failed". Yes, many times you have to repeat an experiment because of bad reagents, broken machines, little tweaks are needed to some obscure parameter, or someone left the lab door open... However, if you experiment is well controlled, then the controls will reveal this sort of "failure". When I was still running gels, many times when first running a new setup we'd run only controls. If your experiment fails because your controls failed, then that's just science. But I've also seen the other kind of "failure". The kind where the controls came out perfectly, but the effect or the association or the expression profile that the researcher was hoping for didn't show up. When these sorts of failures are ignored or discarded, then we do science a huge disservice. I am encouraged, though, that there recently seems to be a movement toward, if not outright publishing such negative results, then at least archiving them and sharing them with others in the field. After all, without Michelson and Morley's "failure" we might not have special relativity.

akshatpradhan 11y ago
>But I've also seen the other kind of "failure". The kind where the controls came out perfectly, but the effect or the association or the expression profile that the researcher was hoping for didn't show up. When these sorts of failures are ignored or discarded, then we do science a huge disservice. This.
mrhhug 11y ago
>But I've also seen the other kind of "failure". The kind where the controls came out perfectly, but the effect or the association or the expression profile that the researcher was hoping for didn't show up. When these sorts of failures are ignored or discarded, then we do science a huge disservice. Why does this happen? Clearly this is what the article insinuates. Is publish or perish that strong? Every honest experiment with honest results benefits society. Not every prediction and result combination results in a prize in your lifetime, but that in no way should influence someone's value as a scientist. That science may be used later for something we had not intended (could i offer you the hope of posthumus recognition?). Finding a way it does not work may save someone else some time. This benefits the scientific community. Not everyone gets to walk on another planet, some people have to build the ship.