5 ms·

I can't state how strongly I disagree with the conclusion that papers should be providing excruciating detail about protocol just because "pharmaceutical compan

• toufka 11y ago
  To follow up, the papers can sometimes be deliberately vague with respect to actual volumes/masses/materials, as theoretically the concentration, & activity and other state-functions sufficiently describe the system. The exact DNA sequence I use might be important, but if I just say I used gene, 'hGENE1', that should be sufficient for the published science. Sure, my hGene1 might have a mutation, or yours might, but we'd never learn that if we just kept passing around the same plasmid, calling it hGene1 (I'm looking at you, cell-lines...). That you used brand-X tubes should generally not be important to the science (conceptually, though sometimes they are physically), and so should be left out of the paper's protocol precisely so that another lab does not use that tube during a replication study. It's not bad science that using brand-X tube yields different results than brand-Y tube, though it can be frustrating to learn that is the crux of the difference you've been seeing. It is precisely that kind of variable that must be ferreted out, and can often only be ferreted out when labs attack the same problem from a different angle.
  • mrestko 11y ago
    In terms of listing certain brands, you're completely correct that this should not cause there to be different results. But I disagree that these details should be left out of the paper. Any competent scientist attempting to replicate a result can make assumptions about which equipment (like tubing) should not reasonably be expected to change the results. It should be up to the replicating scientist to make the equipment substitutions that he deems reasonable. An example might be listing that a certain brand of pippetor tips or centrifuge were used during the experiment. It is highly unlikely that a lab is going to go out of their way to acquire the exact brand of pipettor tips or centrifuge in order to replicate the experiment. But should they have difficulty reproducing the results, having the additional specific data in the original write up allows for the replicating lab to being troubleshooting on their own. They can begin looking at small level details which may have affected their results.
• return0 11y ago
  The specifics of the animals used for example may be the cause of severe errors and cause wildly different results.
  • nmrm2 11y ago
    My criticism was only of the justification given. And honestly, the attitude in scientific publishing has never been "science should be written so that people in industry can reproduce the results at minimal effort/expense and without acquiring scientists who know the area and techniques well." (Also, your post would have worked just as well without the first sentence.)
    • deleted 11y ago
      [deleted]
    • return0 11y ago
      yeah, sorry about that. but biological systems are very complex so it's more important to know whether the claimed argument is verified. Finding a negative result under a different protocol would be very hard to analyze and pinpoint the source of the discrepancy. Also, empirical science is very much the art of precise things. That's the difference between 20th century physics and ancient philosophy.
      • nmrm2 11y ago
        Agreed. I think results should be reproducible. I guess my point is that the amount of time spent writing up and communicating protocol should be governed by an analysis of the scientific benefits of doing so, rather than commercial benefits. I'm not saying that protocols can't be improved at appropriate cost. Rather, I'm saying that the cost of creating protocols that random person at pharma company can use correctly the first time without asking for help is a going to result in an enormous amount of wasted time getting to a level of reproducibility that isn't necessary for doing good science (where good != profitable). And at the frontiers of science where we're using a novel protocol for the first time, I think it's OK to publish without extensive documentation. If the protocol ends up being useful and important enough that other people want to use it, then that's the time to start investing time in high-quality documentation. Which is one of the scientific benefits of collaborative reproduction. (FWIW I think michaelhoffman did a better job of capturing the essence of my complaint in a few sentences than I've done over multiple posts.)
• michaelhoffman 11y ago
  Yes, it's the truth of the biological knowledge that is most important. And we get at that best not by replicating a particular experimental design in two labs, but by examining it in the context of a diversity of different experiments with different assumptions. The Reproducibility Project is run by people in the business of replicating experimental design for profit. They argue that a proportion of public research money should be given to businesses like theirs for this process. This will take a lot of resources that would otherwise be spent examining biological knowledge from different angles rather than the same angle multiple times. In the end, we will know less.
• kazinator 11y ago
  It also casts a doubt on the independence of the verification. It will only catch outright cases of falsification, but innocent situations where the procedure is erroneous or the conclusions are wrongly drawn will slip through. If the reproducers don't quite know what they are doing if left to their own devices, they will invariably be influenced by the original researchers into being blind to exactly the same mistakes and making the same wrong inferences. Another software analogy: it's like QA people needing detailed hand-holding from the original developer in testing some program, instead of independently looking for ways to break it. That is then no better than the original functional testing by the developer.
  • jsprogrammer 11y ago
    There are at least two distinct concepts here. 1) Exactly (as much as possible, given apparent environmental constraints) reproducing the procedure and results claimed and 2) attacking the procedure or results as ineffective or misleading (to what is actually being claimed). I'm not sure either is quite analogous to QA testing in the way you claimed though. Yes, QA testing should be reproducible and yes, QA testing should search for novel tests, but the scientific process is still a bit different and there are multiple distinct and important phases and concepts at play.
• ColinDabritz 11y ago
  That may be true, but one of the key ideals in science is that the findings are reproduceable. If I can't reproduce it, it's helpful for the science to be fairly specific, in case the findings depended on details that were thought to be unimportant, but we later find out were critical. The most interesting science comes in unexpected results. I feel it's like a piece of closed source software releasing "the source code" and yet no one can make it compile. You don't have to have a clean codebase, it can be a bit tricky to build, but it has to be able to be compiled before anyone can have any confidence in your claims. If you don't provide that, it's much less useful. The separate issues of for profit labs and what support they should expect in this context are tough issues, but we shouldn't undermine the science to thwart them.
  • nmrm2 11y ago
    Thanks for giving an excellent example. NuPRL is exactly like this, but no one in their right mind would say the group that put out NuPRL hasn't done good science for 20+ years.
    • ColinDabritz 11y ago
      To clarify, the NuPRL project is like this in what way? They discuss cs topics and algorithms but no source code exists? or it exists but they don't provide it? Or (unlikely) they provide source code no one can compile? I don't want to mix up the side issue of "should code-based research always provide source code" with the metaphors being discussed. I definitely think useful CS research can happen without source code needing to exist, which I hope is the NuPRL project's case. I do feel code based research should provide source code that was created as part of the research when it is material to their claims, including 'we ran a simulation' and similar findings. If you already have that level of detail, you should include it. If you make claims based on data, you need to provide that data. If you make claims based on source code, you need to provide that source code.
      • nmrm2 11y ago
        > the NuPRL project is like this in what way? Difficult to obtain and get running (I'm told by everyone I've asked). So definitely not the first thing; mostly those last two things. And yeah, I think that high-quality source code is definitely a pretty impressive feat when researchers pull in off. And I also agree source code should be provided. But beyond providing a VM, I think it's really dangerous to sort of fixate on building software that's easy to setup/use. There's a pretty significant time cost there in some cases, and it's worth asking whether that's what we want to be spending our Science dollars on.
• raldi 11y ago
  You make a lot of assertions, but the closest you come to explaining the reason for any of those assertions is, "That's the way it's always been done." Can you elaborate on why this new approach isn't even worth trying?
  • nmrm2 11y ago
    Lots of the replies have covered the whys: * Because this is in the context of a for profit company arguing that they should get a slice of the Science funding pie so that pharmaceutical companies have an easier time at reproducing results. It's unclear that this is a better use of funding than investing in new discoveries (unless you are profoundly suspicious of scientists and think they are all fabricating results), but it's fairly obvious why this aligns with the profit motives of pharmaceutical companies. I hope it's obvious why that's troubling to me and others. * Because there is value in reproducing results without doing everything exactly identically (see other posts). * Because there are other mechanisms for obtaining high-quality descriptions of protocols and it doesn't necessarily always make sense to invest lots of time in this early in the scientific process. * Because public morality panics rarely result in good policy making.
• Panoramix 11y ago
  Your analogies are completely off the mark. It is not the same at all. The cornerstone of science is that other people can reproduce your results. Period. There is no use publishing otherwise. Withholding key information, which is so ubiquitous now, is a great disservice to the scientific community.
  • nmrm2 11y ago
    The question is not whether reproducibility is good, it's how much labs should invest upfront in producing descriptions of protocols. My argument is that they should probably invest more than they do now, but not enough that pharmacutical companies are able to reproduce a given experiment without talking to the lab. Science is a collaborative process. There's nothing wrong with collaboration being part of the reproducibility process, as long as the person doing the reproduction maintains their objectivity.
  • coliveira 11y ago
    People often misunderstand the idea of reproducibility in science. The idea is not that scientists need to given a complete, easy to follow set of procedures that will let anyone reproduce an observation. In some areas this is close to impossible. For example, in high energy physics there is only one equipment in the world that can reproduce (with luck) some key experiment. The idea is that, with enough effort and funding somebody else could possibly reach the same conclusions. If you don't agree with some published results, there is an easy way to do it: create your own experiment and publish the results. If the result conflicts, then a new step is to determine why and in which conditions there is a conflict. Science evolves through the debate of ideas and observations, not because somebody is sharing a cookie-cut recipe.
• mcguire 11y ago
  Are the researches at the pharmaceutical companies really "literally anyone"? How do you "focus on the veracity" of the claim if you cannot reproduce it?
• tjradcliffe 11y ago
  If I claim, "An implementation of this algorithm (described in pseudocode) achieves O(N) peformance on that task" and no one can reproduce it based on the pseudocode, I've published a lie. This is a phenomenon I've seen many times in my work as a consultant translating academic discoveries to commercial code. This work invovles digging through both published papers and actual code to find that some grad student has implemented a clever heuristic that is not described in the paper because it is "inessential" that is what the claimed performance actually relies on. The article says, "It's unrealistic to think contract labs or university core facilities can get the same results as a highly specialized team of academic researchers, they say." I can understand this sentiment: science is hard, and at the cutting edge it's really hard. I've certainly got results that I think it would be hard for others to achieve and I've succeed with experiments where others have failed simply because I have good hands. I once took over an experiment that had been worked on by a heavy smoker with shaky hands. He had developed a complicated apparatus for streching ultra-thin plastic foils over a frame. I just used my fingers. If I had published the way I did it and someone like him had tried to reproduce it, they would have failed. However... the kind of reproducibilty being talked about here is not (primarily) technique, it is results. If a result can't be reproduced because the cell line was so very special or whatever, then it likely isn't robust enough to be very interesting in practical biology. It certainly isn't of much interest in drug discovery. So while the science may be OK, results that are incredibly hard to reproduce are of very low value for biotech, and that knowledge is worth something in and of itself.