6 ms·

I absolutely agree that sometimes, you need to redo an experiment for good reasons. In most cases I've seen, people do not know why they redo the experiment, t

I absolutely agree that sometimes, you need to redo an experiment for good reasons. In most cases I've seen, people do not know why they redo the experiment, though. They know it hasn't produced the data they expected, so they redo it. Maybe it was because a reagent was bad, or a co-worker left the incubator open overnight, or maybe it was because the model is stupid. Who knows? That's my point, actually. Biologists are playing with systems they do not understand, changing parameters somewhat randomly without any control over them, and they then try to interpret whatever comes out, but ONLY it fits what they wanted. If it doesn't, then "Oh, the PCR machine is at it again!", and they throw the results away.

mbreese 11y ago
It seems like you had a really bad experience in lab. I'm sorry for that. But it's a mistake to paint the entire field in a negative light because of this. Not all labs are bad, and some produce really outstanding work. Sometimes the issue is the PCR machine. Sometimes it's the water (my favorite troubleshooting experience from grad school). And figuring out where the issues are (is the the protocol? the reagents? or is this real signal?) can be difficult. Playing with systems we don't understand is kinda the point.
- chris_wot 11y ago
  But then you risk thinking you understand the system, but you actually don't. How can you be sure the equipment failure didn't make you see something that wasn't there?
  - mbreese 11y ago
    That's why you repeat the experiment to confirm your results. A result due to a sporadic equipment failure likely won't be reproducible. If you can't replicate your own results, how could you expect anyone else to be able to?
- TisButMe 11y ago
  I've been lucky enough to work in outstanding labs, with people published in Nature, and other journals of that quality. I've worked in 2 countries, and for 4 different labs. I've also talked with people from all over the world, who have worked everywhere, from Harvard to the Pasteur Institute to Cambridge University. The stories are all the same. I hoped I would find some place where people were trying to do things the right way, but what I found is that currently, you don't need to to be published in top journals, so why bother? It's really refreshing to hear you talk about trying to troubleshoot why an experiment didn't work the way you expected, I hear mostly of people retrying blindly until it "succeeds". What did you do with what you learned with the water causing the failure? Did you publish this, so that someone (or you!) could try to figure out why water was a problem, or at least so that no one would have the same issue? This is the other point: when people do bother about finding about why things fail, I've never seen any of them try to follow up on that, and figure out not only what made it fail, but why it made it fail. "Yeah, the annealing temp was not the right one". Ok, but why? Of course playing with systems we don't understand is the point, but we have to be very careful about them. We should be varying 1 parameter at a time. This is mostly impossible in biology, but right now we're not even trying to do anything about it.
  - nycticorax 11y ago
    People don't investigate things like that, because, frankly, no one cares. Nor should they. If your computer is acting funny, and you find out a spider has made a nest inside it, and it works fine once you clear out the spider nest, would you then decide to determine exactly why that spider nest, in that place, causes the exact problems you observed? No, of course not. Because that's not an interesting question. Computers are complicated machines, and they can break in lots of different ways, most of which are not very interesting in their details. Similarly, suppose you study cultured cells (which are notoriously finicky), and you want to compare what the cells do in the presence of drug X vs control. But at first, you find that all of your control cells die. And eventually you find out that if you use brand X of bottled water vs tap water, the control cells thrive. Are you seriously proposing that you should then drop all work on drug X, and get to work determining exactly what is up with the tap water in your town? I mean, maybe that would be a fruitful research avenue, if you're worried that the tap water isn't safe for human consumption, or you think that there's something interesting about exactly how the tap water is killing your control cells. But most of the time, investigating the tap water would be an expensive distraction from the question you actually want to answer. And most scientists, I think, would (reasonably) decide to get on with investigating the effects of drug X on their cells, and not worry too much about precisely why the tap water killed the control cells. And I don't think there's anything wrong with that. Life is short, and you have to choose your questions carefully.
- XorNot 11y ago
  "We need to get a new carbon filter for that MilliQ machine" "Wait there's a carbon filter in the milliQ machine?" "Yeah..." "...oh, that explains all those results I got" ^ Actual conversation I had in a breakroom. Stuff breaks all the time. The temperature of the lab changes. Connections get loose. 95% of the time, when the experiment doesn't work, it's because you screwed up on something that's either difficult to control or impossible to control or which you didn't even realize probably wasn't the thing you expected it to be.

mbreese 11y ago

It seems like you had a really bad experience in lab. I'm sorry for that. But it's a mistake to paint the entire field in a negative light because of this. Not all labs are bad, and some produce really outstanding work. Sometimes the issue is the PCR machine. Sometimes it's the water (my favorite troubleshooting experience from grad school). And figuring out where the issues are (is the the protocol? the reagents? or is this real signal?) can be difficult. Playing with systems we don't understand is kinda the point.

chris_wot 11y ago
But then you risk thinking you understand the system, but you actually don't. How can you be sure the equipment failure didn't make you see something that wasn't there?
- mbreese 11y ago
  That's why you repeat the experiment to confirm your results. A result due to a sporadic equipment failure likely won't be reproducible. If you can't replicate your own results, how could you expect anyone else to be able to?
TisButMe 11y ago
I've been lucky enough to work in outstanding labs, with people published in Nature, and other journals of that quality. I've worked in 2 countries, and for 4 different labs. I've also talked with people from all over the world, who have worked everywhere, from Harvard to the Pasteur Institute to Cambridge University. The stories are all the same. I hoped I would find some place where people were trying to do things the right way, but what I found is that currently, you don't need to to be published in top journals, so why bother? It's really refreshing to hear you talk about trying to troubleshoot why an experiment didn't work the way you expected, I hear mostly of people retrying blindly until it "succeeds". What did you do with what you learned with the water causing the failure? Did you publish this, so that someone (or you!) could try to figure out why water was a problem, or at least so that no one would have the same issue? This is the other point: when people do bother about finding about why things fail, I've never seen any of them try to follow up on that, and figure out not only what made it fail, but why it made it fail. "Yeah, the annealing temp was not the right one". Ok, but why? Of course playing with systems we don't understand is the point, but we have to be very careful about them. We should be varying 1 parameter at a time. This is mostly impossible in biology, but right now we're not even trying to do anything about it.
- nycticorax 11y ago
  People don't investigate things like that, because, frankly, no one cares. Nor should they. If your computer is acting funny, and you find out a spider has made a nest inside it, and it works fine once you clear out the spider nest, would you then decide to determine exactly why that spider nest, in that place, causes the exact problems you observed? No, of course not. Because that's not an interesting question. Computers are complicated machines, and they can break in lots of different ways, most of which are not very interesting in their details. Similarly, suppose you study cultured cells (which are notoriously finicky), and you want to compare what the cells do in the presence of drug X vs control. But at first, you find that all of your control cells die. And eventually you find out that if you use brand X of bottled water vs tap water, the control cells thrive. Are you seriously proposing that you should then drop all work on drug X, and get to work determining exactly what is up with the tap water in your town? I mean, maybe that would be a fruitful research avenue, if you're worried that the tap water isn't safe for human consumption, or you think that there's something interesting about exactly how the tap water is killing your control cells. But most of the time, investigating the tap water would be an expensive distraction from the question you actually want to answer. And most scientists, I think, would (reasonably) decide to get on with investigating the effects of drug X on their cells, and not worry too much about precisely why the tap water killed the control cells. And I don't think there's anything wrong with that. Life is short, and you have to choose your questions carefully.
XorNot 11y ago
"We need to get a new carbon filter for that MilliQ machine" "Wait there's a carbon filter in the milliQ machine?" "Yeah..." "...oh, that explains all those results I got" ^ Actual conversation I had in a breakroom. Stuff breaks all the time. The temperature of the lab changes. Connections get loose. 95% of the time, when the experiment doesn't work, it's because you screwed up on something that's either difficult to control or impossible to control or which you didn't even realize probably wasn't the thing you expected it to be.