13 ms·

Cancer reproducibility effort faces backlash

156 points by tlb 11y ago

jessriedel 11y ago
It seems like a sensible check is for this collaboration to include original studies, like the one mentioned in the article lede, that have already been replicated elsewhere. (Ideally they would keep blind the relevant members of the collaboration to this fact.) Then when you say "we failed to replicate X% of the studies" you also say "of the subgroup that had already be replicated, we failed to replicate Y%". If Y isn't much smaller than X, you know the replication collaboration is probably botching this.
nmrm2 11y ago
I can't state how strongly I disagree with the conclusion that papers should be providing excruciating detail about protocol just because "pharmaceutical companies can't reproduce key cancer papers [without the help of the original scientists]". Science has rarely been done like this. It would be like Google complaining that they can't copy psuedocode verbatim out of a paper and have a highly performant algorithm. Or Microsoft complaining that a static analysis defined in a paper wasn't accompanied by a production-ready implementation. Producing protocols that literally anyone could replicate without expending effort is not the business of Science. Replication should focus on the veracity of the underlying truth claim, not the economics of reproducing the results.
- toufka 11y ago
  To follow up, the papers can sometimes be deliberately vague with respect to actual volumes/masses/materials, as theoretically the concentration, & activity and other state-functions sufficiently describe the system. The exact DNA sequence I use might be important, but if I just say I used gene, 'hGENE1', that should be sufficient for the published science. Sure, my hGene1 might have a mutation, or yours might, but we'd never learn that if we just kept passing around the same plasmid, calling it hGene1 (I'm looking at you, cell-lines...). That you used brand-X tubes should generally not be important to the science (conceptually, though sometimes they are physically), and so should be left out of the paper's protocol precisely so that another lab does not use that tube during a replication study. It's not bad science that using brand-X tube yields different results than brand-Y tube, though it can be frustrating to learn that is the crux of the difference you've been seeing. It is precisely that kind of variable that must be ferreted out, and can often only be ferreted out when labs attack the same problem from a different angle.
  - mrestko 11y ago
    In terms of listing certain brands, you're completely correct that this should not cause there to be different results. But I disagree that these details should be left out of the paper. Any competent scientist attempting to replicate a result can make assumptions about which equipment (like tubing) should not reasonably be expected to change the results. It should be up to the replicating scientist to make the equipment substitutions that he deems reasonable. An example might be listing that a certain brand of pippetor tips or centrifuge were used during the experiment. It is highly unlikely that a lab is going to go out of their way to acquire the exact brand of pipettor tips or centrifuge in order to replicate the experiment. But should they have difficulty reproducing the results, having the additional specific data in the original write up allows for the replicating lab to being troubleshooting on their own. They can begin looking at small level details which may have affected their results.
- return0 11y ago
  The specifics of the animals used for example may be the cause of severe errors and cause wildly different results.
  - nmrm2 11y ago
    My criticism was only of the justification given. And honestly, the attitude in scientific publishing has never been "science should be written so that people in industry can reproduce the results at minimal effort/expense and without acquiring scientists who know the area and techniques well." (Also, your post would have worked just as well without the first sentence.)
    - deleted 11y ago
      [deleted]
    - return0 11y ago
      yeah, sorry about that. but biological systems are very complex so it's more important to know whether the claimed argument is verified. Finding a negative result under a different protocol would be very hard to analyze and pinpoint the source of the discrepancy. Also, empirical science is very much the art of precise things. That's the difference between 20th century physics and ancient philosophy.
      - nmrm2 11y ago
        Agreed. I think results should be reproducible. I guess my point is that the amount of time spent writing up and communicating protocol should be governed by an analysis of the scientific benefits of doing so, rather than commercial benefits. I'm not saying that protocols can't be improved at appropriate cost. Rather, I'm saying that the cost of creating protocols that random person at pharma company can use correctly the first time without asking for help is a going to result in an enormous amount of wasted time getting to a level of reproducibility that isn't necessary for doing good science (where good != profitable). And at the frontiers of science where we're using a novel protocol for the first time, I think it's OK to publish without extensive documentation. If the protocol ends up being useful and important enough that other people want to use it, then that's the time to start investing time in high-quality documentation. Which is one of the scientific benefits of collaborative reproduction. (FWIW I think michaelhoffman did a better job of capturing the essence of my complaint in a few sentences than I've done over multiple posts.)
- michaelhoffman 11y ago
  Yes, it's the truth of the biological knowledge that is most important. And we get at that best not by replicating a particular experimental design in two labs, but by examining it in the context of a diversity of different experiments with different assumptions. The Reproducibility Project is run by people in the business of replicating experimental design for profit. They argue that a proportion of public research money should be given to businesses like theirs for this process. This will take a lot of resources that would otherwise be spent examining biological knowledge from different angles rather than the same angle multiple times. In the end, we will know less.
- kazinator 11y ago
  It also casts a doubt on the independence of the verification. It will only catch outright cases of falsification, but innocent situations where the procedure is erroneous or the conclusions are wrongly drawn will slip through. If the reproducers don't quite know what they are doing if left to their own devices, they will invariably be influenced by the original researchers into being blind to exactly the same mistakes and making the same wrong inferences. Another software analogy: it's like QA people needing detailed hand-holding from the original developer in testing some program, instead of independently looking for ways to break it. That is then no better than the original functional testing by the developer.
  - jsprogrammer 11y ago
    There are at least two distinct concepts here. 1) Exactly (as much as possible, given apparent environmental constraints) reproducing the procedure and results claimed and 2) attacking the procedure or results as ineffective or misleading (to what is actually being claimed). I'm not sure either is quite analogous to QA testing in the way you claimed though. Yes, QA testing should be reproducible and yes, QA testing should search for novel tests, but the scientific process is still a bit different and there are multiple distinct and important phases and concepts at play.
- ColinDabritz 11y ago
  That may be true, but one of the key ideals in science is that the findings are reproduceable. If I can't reproduce it, it's helpful for the science to be fairly specific, in case the findings depended on details that were thought to be unimportant, but we later find out were critical. The most interesting science comes in unexpected results. I feel it's like a piece of closed source software releasing "the source code" and yet no one can make it compile. You don't have to have a clean codebase, it can be a bit tricky to build, but it has to be able to be compiled before anyone can have any confidence in your claims. If you don't provide that, it's much less useful. The separate issues of for profit labs and what support they should expect in this context are tough issues, but we shouldn't undermine the science to thwart them.
  - nmrm2 11y ago
    Thanks for giving an excellent example. NuPRL is exactly like this, but no one in their right mind would say the group that put out NuPRL hasn't done good science for 20+ years.
    - ColinDabritz 11y ago
      To clarify, the NuPRL project is like this in what way? They discuss cs topics and algorithms but no source code exists? or it exists but they don't provide it? Or (unlikely) they provide source code no one can compile? I don't want to mix up the side issue of "should code-based research always provide source code" with the metaphors being discussed. I definitely think useful CS research can happen without source code needing to exist, which I hope is the NuPRL project's case. I do feel code based research should provide source code that was created as part of the research when it is material to their claims, including 'we ran a simulation' and similar findings. If you already have that level of detail, you should include it. If you make claims based on data, you need to provide that data. If you make claims based on source code, you need to provide that source code.
      - nmrm2 11y ago
        > the NuPRL project is like this in what way? Difficult to obtain and get running (I'm told by everyone I've asked). So definitely not the first thing; mostly those last two things. And yeah, I think that high-quality source code is definitely a pretty impressive feat when researchers pull in off. And I also agree source code should be provided. But beyond providing a VM, I think it's really dangerous to sort of fixate on building software that's easy to setup/use. There's a pretty significant time cost there in some cases, and it's worth asking whether that's what we want to be spending our Science dollars on.
- raldi 11y ago
  You make a lot of assertions, but the closest you come to explaining the reason for any of those assertions is, "That's the way it's always been done." Can you elaborate on why this new approach isn't even worth trying?
  - nmrm2 11y ago
    Lots of the replies have covered the whys: * Because this is in the context of a for profit company arguing that they should get a slice of the Science funding pie so that pharmaceutical companies have an easier time at reproducing results. It's unclear that this is a better use of funding than investing in new discoveries (unless you are profoundly suspicious of scientists and think they are all fabricating results), but it's fairly obvious why this aligns with the profit motives of pharmaceutical companies. I hope it's obvious why that's troubling to me and others. * Because there is value in reproducing results without doing everything exactly identically (see other posts). * Because there are other mechanisms for obtaining high-quality descriptions of protocols and it doesn't necessarily always make sense to invest lots of time in this early in the scientific process. * Because public morality panics rarely result in good policy making.
- Panoramix 11y ago
  Your analogies are completely off the mark. It is not the same at all. The cornerstone of science is that other people can reproduce your results. Period. There is no use publishing otherwise. Withholding key information, which is so ubiquitous now, is a great disservice to the scientific community.
  - nmrm2 11y ago
    The question is not whether reproducibility is good, it's how much labs should invest upfront in producing descriptions of protocols. My argument is that they should probably invest more than they do now, but not enough that pharmacutical companies are able to reproduce a given experiment without talking to the lab. Science is a collaborative process. There's nothing wrong with collaboration being part of the reproducibility process, as long as the person doing the reproduction maintains their objectivity.
  - coliveira 11y ago
    People often misunderstand the idea of reproducibility in science. The idea is not that scientists need to given a complete, easy to follow set of procedures that will let anyone reproduce an observation. In some areas this is close to impossible. For example, in high energy physics there is only one equipment in the world that can reproduce (with luck) some key experiment. The idea is that, with enough effort and funding somebody else could possibly reach the same conclusions. If you don't agree with some published results, there is an easy way to do it: create your own experiment and publish the results. If the result conflicts, then a new step is to determine why and in which conditions there is a conflict. Science evolves through the debate of ideas and observations, not because somebody is sharing a cookie-cut recipe.
- mcguire 11y ago
  Are the researches at the pharmaceutical companies really "literally anyone"? How do you "focus on the veracity" of the claim if you cannot reproduce it?
- tjradcliffe 11y ago
  If I claim, "An implementation of this algorithm (described in pseudocode) achieves O(N) peformance on that task" and no one can reproduce it based on the pseudocode, I've published a lie. This is a phenomenon I've seen many times in my work as a consultant translating academic discoveries to commercial code. This work invovles digging through both published papers and actual code to find that some grad student has implemented a clever heuristic that is not described in the paper because it is "inessential" that is what the claimed performance actually relies on. The article says, "It's unrealistic to think contract labs or university core facilities can get the same results as a highly specialized team of academic researchers, they say." I can understand this sentiment: science is hard, and at the cutting edge it's really hard. I've certainly got results that I think it would be hard for others to achieve and I've succeed with experiments where others have failed simply because I have good hands. I once took over an experiment that had been worked on by a heavy smoker with shaky hands. He had developed a complicated apparatus for streching ultra-thin plastic foils over a frame. I just used my fingers. If I had published the way I did it and someone like him had tried to reproduce it, they would have failed. However... the kind of reproducibilty being talked about here is not (primarily) technique, it is results. If a result can't be reproduced because the cell line was so very special or whatever, then it likely isn't robust enough to be very interesting in practical biology. It certainly isn't of much interest in drug discovery. So while the science may be OK, results that are incredibly hard to reproduce are of very low value for biotech, and that knowledge is worth something in and of itself.
astazangasta 11y ago
Looking at this through the lens of drug discovery is the wrong way to do this. The problem is with our drug discovery strategy, generally, not with the reproducibility of our research. STK33, for example, is definitely implicated in cancer through a wide variety of mechanisms. It is often mutated in tumors, and multiple studies have picked it up as having a role in driving migration, metastasis, etc. This doesn't mean we can make good drugs to it. Making drugs is hard - they need to be available in the tissue in the right concentrations, often difficult to achieve with a weird-shaped, sticky molecule. They need to have specificity for the tumor, they need to have specificity for the gene target(s) of interest. They need to be effective at modulating the target. More importantly, though, the drug is modulating a target (gene) that is involved in a biological system that involves complex systems of feedback control, produces adaptive responses, and otherwise behaves in unexpected ways in response to modulation. In my experience this is usually underappreciated by most drug discovery strategies, which merely seek to "inhibit the target" as if its involvement in the tumor process means we can simply treat it as an "on-off" switch for cancer. This assumption is asinine, and of course will (and does) lead to frequent failure. STK33 is not an on-off switch, and attempting to treat it that way will likely result in a drug that does nothing.
- toufka 11y ago
  This is absolutely correct. Pharma companies are running into a wall and are flailing to figure out what to do. It's quite clear that from first principles bathing the entire body in trillions of little molecules hoping that they only and completely shut down a single kind of protein, at the right time, in the right place, of the right cell, and do nothing more, is insane. There is some logic behind the ability for a small molecule to help against invading diseases (the antibiotics of the 20th century), but the same strategy will philosophically not work for entire classes of cancers or other innate biological problems. Though we all somehow just assumed that it would. The pharmacy of the future will be entirely curative to non-invading diseases, will repair the DNA that's been mutated, will express or inhibit the proteins that need to be expressed or inhibited. And small molecules will be the payload of these fancy protein-based nano-machines. But this hunt to bring down the cost of those small-molecule targets at the cost of the reputation of the science itself might be foolhardy when the cost is near-infinite in the first place because they're looking from the wrong perspective. tldr; Pharmaceutical companies' hammer that worked so well against the nails of bacterial infection is in no way suited to the plumbing of cancer. And now they're 'investigating the plumbers' to figure out why their fancy new 50-billion-dollar "water-hammers" don't work so well to unclog pipes.
lettergram 11y ago
Interesting... Over the past year I was tickling the idea of making an organization/company/website which automatically would give tax dollars to research. The idea being, you can maximize the amount you right off in taxes and donate to research you desire (i.e. more NASA, less children killed around the globe). The idea stemmed from the the idea that people want access to research (public) AND reproducible. The funds would go directly to research groups, and as an incentive, reproducibility would have bounties based on what people were willing to donate. Because virtually every research group with public research is supported by a non-profit, no one losses additional money, but more funds go towards public interest research GROUPS not organizations with bureaucracy. Somewhat off topic, but this seems another reason for me to start the project.
anonbanker 11y ago
the group clarified that they did not want to replicate the 30 or so experiments in the Cell paper, but just four described in a single key figure. And those experiments would be performed not by another academic lab working in the same area, but by an unnamed contract research organization. sounds like someone wants to quietly weaponize this.
gwern 11y ago
> This past January, the cancer reproducibility project published its protocol for replicating the experiments, and the waiting began for Young to see whether his work will hold up in their hands. He says that if the project does match his results, it will be unsurprising—the paper's findings have already been reproduced. If it doesn't, a lack of expertise in the replicating lab may be responsible. Either way, the project seems a waste of time, Young says. “I am a huge fan of reproducibility. But this mechanism is not the way to test it.” One swallow does not make a spring. With a belief like 'one replication is enough', I'm not sure Young actually appreciates how large sampling error is under usual significance levels or how high heterogeneity between labs is.
- joe_the_user 11y ago
  As far as I can tell from the article, Young's stance is that the contract lab doing the reproduction doesn't seem competent enough to correctly reproduce the experiment. I don't know enough to know if that's true or not. But it seems at least possible that in the rush to reproduce much, the project is cutting costs by using less-skilled contract labs.
  - jsprogrammer 11y ago
    Presumably the replicating lab will record and document their approach and results so that it can be compared with the original experiment.
    - joe_the_user 11y ago
      Indeed, we can hope this. But as the op mentions, lab techniques vary and not everything can be documented. It's reasonable for a scientist to ask for a certain level in a lab attempting reproduction. I don't know if these scientists are demanding too much and have something to hide or if there are problem with the lab involved. However, the article actually mentions numerous scientists expressing misgivings.
      - jsprogrammer 11y ago
TisButMe 11y ago
This is the same behaviour I've seen time and time again in biology labs. People there are re-doing the same experiment over and over until it gives them the result they want, and then they publish that. It's the only field where I've heard people saying "Oh, yeah, my experiment failed, I have to do it again". What does it even mean that an experiment failed? It did exactly what it was supposed to: it gave you data. It didn't fit your expectations? Good, now you have a tool to refine your expectations. But instead, we see PhD students and post-doc working 70h hours week on experiments with seemingly random results until the randomness goes their way. A lot of them have no clue about statistical treatment of data, making a proper model to try and test assumptions against reality. Since they deal with insanely complicated system, with hidden variables all over the place, a proper statistical analysis would be the minimum expected to be able to extract any information from the data, but no matter, once you have a good looking figure, you're done. In cellular/molecular biology, nobody cares about what a p-value is, so as long as Excel tells you it's <0.05, you're golden. The scientific process has been forgotten in biology. Right now it's basically what alchemy was to chemistry. I very happy to see efforts like this one. Sure, they might show that a lot of "key" papers are very wrong, but that's not the crux of it. If there is a reason for biologists to make sure that their results are real, they might try to put a little more effort into checking their work. And when they figure out how much of it is bullshit, they might even try to slow down a little on the publications and go back to the basics for a little while. I'm sorry about this rant, but I've been driven away from a career in virology by those same issues, despite my love for the discipline, so I'm a bit bitter.
- JoshTriplett 11y ago
  https://xkcd.com/882/ https://xkcd.com/882/ Hopefully the people trying to reproduce results don't follow the same strategy. Also see https://en.wikipedia.org/wiki/Data_dredging https://en.wikipedia.org/wiki/Data_dredging .
  - TisButMe 11y ago
    Yeah, I've seen this xkcd. It's on point as usual, but it doesn't show the frightening thing: it's the scientists who are going "Whoa" and believing in the results they've produced...
ryanobjc 11y ago
As a complete outsider reading the attitudes behind the original scientists, it seems to me that they resent the oversight and hate to do extra work. In defending their practices they fall back on "expert work" and essentially are arguing that what they are doing is too complex for anyone else to do and they should be left alone to continue to do it. And from their point of view, it seems all very reasonable. But from the rest of humanity who is being asked to materially support them, and waits for their conclusions to make the world a better place, it seems ... frankly... lazy and selfish. 30 emails, wow! 2 weeks of a graduate student's time -- these are the people who are the least paid right? Below minimum wage even? The demands on their time seem so low, yet the complaints are so high, that one can't help but wonder if the concern really is that their results are too 'magical' and irreproducible and they just fear other people learning about it. I've seen this behavior in professional settings, and ultimately it comes down to a lack of confidence in oneself, the tools and technology and the quality of work being done. Careers are at stake, but is the alternative to just give people a free pass?
- nmrm2 11y ago
  Sorry, but how is this not just anti-intellectualism? > 30 emails, wow! It can take up to half a day to reply to an email containing even one fairly technical question. Obviously the time invested depends on the content of those 30 emails, but it's not at all difficult to imagine this eating a week worth of a PI's time. > 2 weeks of a graduate student's time -- these are the people who are the least paid right? Below minimum wage even? The typical attitude is that this is a justification for not making them spend their time managing these sorts of tasks. You don't get to say "we get to pay you crap because your job is exciting and rewarding" and then load that same person down with a bunch of grunt work (on top of all the normal grunt work). That's how you bleed students and kill the lab's productivity. Also, grad students are not well paid but they are typically no cheaper than post docs; PIs pay tuition. > The demands on their time seem so low, yet the complaints are so high So let's imagine a world where people calling for massive improvements to reproducability get their way. Let's say it is a month worth of time for the lab. For each paper. Multiple papers a year. That's a pretty massive time investment. If you're a top lab, that could become a full-time position. And believe me, you're not going to be able to fill that position with a grad student. That person will have to be well-paid, because their job is going to suck. So it's reasonable that scientists are peeved when they invest all this time and don't perceive their collaborators as acting in good faith, or feel like their collaborators are trying to cut corners to pinch pennies.
azernik 11y ago
> It's unrealistic to think contract labs or university core facilities can get the same results as a highly specialized team of academic researchers, they say. Often a graduate student has spent years perfecting a technique using novel protocols, Young says. Then they need to spend the time documenting those protocols. My dad worked in biological research, and his attitude has always been: if you don't write it down, you might as well not have done the work at all. ESPECIALLY in research.
coliveira 11y ago
People often misunderstand the idea of reproducibility in science. The idea is not that scientists need to given a complete, easy to follow set of procedures that will let anyone reproduce an observation. In some areas this is close to impossible. For example, in high energy physics there is only one equipment in the world that can reproduce (with luck) some key experiment. The idea is that, with enough effort and funding somebody else could possibly reach the same conclusions. If you don't agree with some published results, there is an easy way to do it: create your own experiment and publish the results. If the result conflicts, then a new step is to determine why and in which conditions there is a conflict. Science evolves through the debate of ideas and observations, not because somebody is sharing a cookie-cut recipe.
chris_wot 11y ago
So hold on a moment. These researchers are doing experiments so badly that they can't find the actual procedures they used to get their results? And now they are tracking down old postdocs and lab technicians just to pick their Brian's as to what they actually did?!? How the heck did this stuff get through peer review? Surely I'm missing something critical?
- infamouscow 11y ago
  Academic peer review has nothing to do with quality control and everything to do with maintaining the status quo. Don't take my word for it, read Retraction Watch: http://retractionwatch.com/ http://retractionwatch.com/
x0054 11y ago
I tend to agree that the biology papers often lack proper documentation of procedures and methodologies. This is a wonderful effort to reproduce some of the key experiments. That being said, I think it's also very important to look at the quality and qualifications of the labs doing the reproductions. I don't have any direct link to cancer research, so I can't speak with authority on the subject, but I have been involved in the past with a company working in the Preimplantation Genetic Diagnosis field. The basics of their procedure is to create one or more human embryos via IVF, incubate the embryos for up to 6 days, than either freeze them, or transplant them into the prospective mother. On day 3 or 5 of incubations the embryo is biopsied, and the genetic material is tested to make sure there are no aneuploidy defects. We were also able to test for some other types of genetic abnormalities. This is for people who are having problem becoming pregnant. In any case, some time in the mid 2000s there were 3 papers published in Europe claiming that performing biopsy on Day 3 is extremely detrimental to the embryo, and their conclusion was that PGD with Day 3 should not be performed. The experiments were conducted by people who were unskilled in micro manipulation. They did follow proper protocols, and I am sure they did their best to replicate proper procedures. But micro manipulation is as much skill as it is knowledge. For instance, I can write a detailed procedure on how to shoot a compound bow, and you can follow that procedure exactly. But, without practice, you are not going to hit the bullseye on the first try. Because we were in the business of providing services to doctors, not publishing papers, we constantly tracked our embryo mortality rates, birth rates, and accuracy of testing. The better our results were, the more business we would get. And we couldn't fake the results, because clinics ordering the test would be the once recording all of those statistics for us. Any way, long story short, none of our data agreed with the papers claiming that Day 3 biopsy was detrimental to the embryo. In fact, quite the contrary, many of our statistics suggested that Day 3 biopsy and Day 4 or Day 5 transfer would result in better implantation rates. But, the papers were published, and referenced, and then it became "common knowledge" that Day 3 biopsy is bad, and the medical industry moved on to Day 5 biopsy and embryo cryopreservation, and so has the company I worked with. To the company I worked for it's all the same, money is money. Day 3 or Day 5 biopsy, they make money all the same. But the patients are not more limited. From the stats we have seen, it doesn't look like Day 4 or 5 biopsy is worse for the embryo, but being frozen isn't a walk in the park. With Day 5 biopsy you have to freeze the embryo, in order to allow time for the test results to come back. Any way, it's my 2 cents. Reproducibility is important, but I think it's just as important to change the incentives of those who publish papers. If you goal is to be published, then of course your research will suffer. It's the publish or parish mentality in academia that is the problem, I think.
cwyers 11y ago
> Jeff Settleman, who left academia for industry 5 years ago and is now at Calico Life Sciences in South San Francisco, California, agrees. “You can't give me and Julia Child the same recipe and expect an equally good meal,” he says. Settleman has two papers being replicated. Uh, Julia Child WROTE FREAKING COOKBOOKS. The entire point of Julia Child was that she tried to develop recipes in such a way that another cook could produce an equally good meal. Now, yes, if I went into a boiler room at Goldman Sachs and picked 10 guys at random, I doubt that most would be able to duplicate the recipe. If I picked 10 professional sous chefs at random and none of them were able to make a dish as good as Julia Child's from her recipe, I would start to have my doubts about the recipe. By the same token, I don't expect rank amateurs to be able to duplicate state of the art cancer research. But if labs run by pharma companies and academic institutions are having the failure rate at reproducing research that the article claims, I think it's more than reasonable to start questioning the paper that documented that research, if not the research itself.
- untilHellbanned 11y ago
  Analogy doesn't hold up. The complexity of cancer research is vastly greater than cooking. But anyway, chefs claim that there is special, non-reproducible elements about their environment and creations, e.g., oven, ingredients, etc. all the time. > none of them were able to make a dish as good Defining "good" is the problem for both cancer research and cooking. You can have two experts saying this is "good" or this is "bad" and not really be able to prove who is right. Fine, maybe for cancer research you can ultimately prove who is right but its not realistic in most cases given the resource constraints of even top-flight labs.
  - cwyers 11y ago
    Well, it's not my analogy. But I think it's a good one, it just undermines the point of the person who made it. If I go into Momofuko, I'm looking for a good meal. A good meal is one that tastes good, in this case. If I'm looking at a Julia Child cookbook, I'm looking for a good recipe. One of the criteria for a good recipe may be that it tastes good, or that it produces a healthy meal -- there's several different criteria you can use here. But one criteria for a good recipe is reproducibility -- in order for a recipe to be good it must contain enough information and be accurate enough for me to create the dish that the recipe is for. A recipe for a tasty meal that does not contain the right ingredients or enough detail in the steps to prepare it is a bad recipe. By the same token, an experiment that cannot be repeated is a bad experiment. It may not be false. But its explanatory value is limited -- if a reaction can only take place in water that's treated a certain way or has/lacks certain minerals, then a paper that doesn't tell me that is leaving out important information. Regardless of whether or not you define the point of cancer research in purely scientific terms -- that is, to learn more about cancer -- or in more pragmatic terms -- that is, to allow us to create better cancer treatments -- omitted information about the circumstances surrounding the test that has a significant effect on the test result gives us less information and is less likely to lead to better cancer treatments.
platform 11y ago
I agree with the overall sentiment, the scientific community working on cancer cure(s) is failing us, the patients and their families. And they are failing us because of some fundamental gaps in how the research, and subsequent review/dissemination/presentation of finding is done. I suspect there are multiple failures in the process. The standards of scientific proof and repeatability, used by mathematicians, physicists and chemists are not followed The net result is the following disappointing statistic: "... In 1971, President Nixon and Congress declared war on cancer. Since then, the federal government has spent well over $105 billion on the effort (Kolata 2009b). ... Gina Kolata pointed out in The New York Times that the cancer death rate, adjusted for the size and age of the population, has decreased by only 5 percent since 1950 (Kolata 2009a)." [1] And this was just the US federal government investment. Not counting the private donations, and private company research. Today the annual fed investment is 5bln annually [3] I do not mean to sound totally discouraged, as clearly the screenings have helped many to detect cancers before they metastasized. And I would say the science results are showing that that part of the research is working well. However,for the cancers that can be rarely be detected before the spread (eg pancreatic cancer and others) -- the investment our country and other societies have put in -- simply has not payed off. What worries me is that our research quality gates are not able to improve the QoS of the underlying research. And with my 'management hat on' -- I am reaching out for this quote by Einstein. "Insanity: doing the same thing over and over again and expecting different results. The OP paper is not the first one pointing at the lack of reproducible results, and it not just for cancer research "... But it may also be due to current state of science. Scientists themselves are becoming increasingly concerned about the unreliability – that is, the lack of reproducibility — of many experimental or observational results. ..." [2] There needs to be a bit of a revolution in the science of the cancer research and the way money is allocated to it. Clearly the current model does not work and likely is encouraging the pseudo science to prosper. [1] http://www.csicop.org/si/show/war_on_cancer_a_progress_report_for_skeptics/ http://www.csicop.org/si/show/war_on_cancer_a_progress_repor... [2] http://www.forbes.com/sites/henrymiller/2014/01/08/the-trouble-with-scientific-research-today-a-lot-thats-published-is-junk/ http://www.forbes.com/sites/henrymiller/2014/01/08/the-troub... [3]http://blogs.reuters.com/stories-id-like-to-see/2014/09/09/the-money-spent-in-fighting-cancer-and-alibabas-risk-factor/ http://blogs.reuters.com/stories-id-like-to-see/2014/09/09/t...
nagaiah 11y ago
Is it possible to inject cancer resistant DNA into humans?

156 points by tlb 11y ago

It seems like a sensible check is for this collaboration to include original studies, like the one mentioned in the article lede, that have already been replicated elsewhere. (Ideally they would keep blind the relevant members of the collaboration to this fact.) Then when you say "we failed to replicate X% of the studies" you also say "of the subgroup that had already be replicated, we failed to replicate Y%". If Y isn't much smaller than X, you know the replication collaboration is probably botching this.

nmrm2 11y ago

I can't state how strongly I disagree with the conclusion that papers should be providing excruciating detail about protocol just because "pharmaceutical companies can't reproduce key cancer papers [without the help of the original scientists]". Science has rarely been done like this. It would be like Google complaining that they can't copy psuedocode verbatim out of a paper and have a highly performant algorithm. Or Microsoft complaining that a static analysis defined in a paper wasn't accompanied by a production-ready implementation. Producing protocols that literally anyone could replicate without expending effort is not the business of Science. Replication should focus on the veracity of the underlying truth claim, not the economics of reproducing the results.

toufka 11y ago
To follow up, the papers can sometimes be deliberately vague with respect to actual volumes/masses/materials, as theoretically the concentration, & activity and other state-functions sufficiently describe the system. The exact DNA sequence I use might be important, but if I just say I used gene, 'hGENE1', that should be sufficient for the published science. Sure, my hGene1 might have a mutation, or yours might, but we'd never learn that if we just kept passing around the same plasmid, calling it hGene1 (I'm looking at you, cell-lines...). That you used brand-X tubes should generally not be important to the science (conceptually, though sometimes they are physically), and so should be left out of the paper's protocol precisely so that another lab does not use that tube during a replication study. It's not bad science that using brand-X tube yields different results than brand-Y tube, though it can be frustrating to learn that is the crux of the difference you've been seeing. It is precisely that kind of variable that must be ferreted out, and can often only be ferreted out when labs attack the same problem from a different angle.
- mrestko 11y ago
  In terms of listing certain brands, you're completely correct that this should not cause there to be different results. But I disagree that these details should be left out of the paper. Any competent scientist attempting to replicate a result can make assumptions about which equipment (like tubing) should not reasonably be expected to change the results. It should be up to the replicating scientist to make the equipment substitutions that he deems reasonable. An example might be listing that a certain brand of pippetor tips or centrifuge were used during the experiment. It is highly unlikely that a lab is going to go out of their way to acquire the exact brand of pipettor tips or centrifuge in order to replicate the experiment. But should they have difficulty reproducing the results, having the additional specific data in the original write up allows for the replicating lab to being troubleshooting on their own. They can begin looking at small level details which may have affected their results.
return0 11y ago
The specifics of the animals used for example may be the cause of severe errors and cause wildly different results.
- nmrm2 11y ago
  My criticism was only of the justification given. And honestly, the attitude in scientific publishing has never been "science should be written so that people in industry can reproduce the results at minimal effort/expense and without acquiring scientists who know the area and techniques well." (Also, your post would have worked just as well without the first sentence.)
  - deleted 11y ago
    [deleted]
  - return0 11y ago
    yeah, sorry about that. but biological systems are very complex so it's more important to know whether the claimed argument is verified. Finding a negative result under a different protocol would be very hard to analyze and pinpoint the source of the discrepancy. Also, empirical science is very much the art of precise things. That's the difference between 20th century physics and ancient philosophy.
    - nmrm2 11y ago
      Agreed. I think results should be reproducible. I guess my point is that the amount of time spent writing up and communicating protocol should be governed by an analysis of the scientific benefits of doing so, rather than commercial benefits. I'm not saying that protocols can't be improved at appropriate cost. Rather, I'm saying that the cost of creating protocols that random person at pharma company can use correctly the first time without asking for help is a going to result in an enormous amount of wasted time getting to a level of reproducibility that isn't necessary for doing good science (where good != profitable). And at the frontiers of science where we're using a novel protocol for the first time, I think it's OK to publish without extensive documentation. If the protocol ends up being useful and important enough that other people want to use it, then that's the time to start investing time in high-quality documentation. Which is one of the scientific benefits of collaborative reproduction. (FWIW I think michaelhoffman did a better job of capturing the essence of my complaint in a few sentences than I've done over multiple posts.)
michaelhoffman 11y ago
Yes, it's the truth of the biological knowledge that is most important. And we get at that best not by replicating a particular experimental design in two labs, but by examining it in the context of a diversity of different experiments with different assumptions. The Reproducibility Project is run by people in the business of replicating experimental design for profit. They argue that a proportion of public research money should be given to businesses like theirs for this process. This will take a lot of resources that would otherwise be spent examining biological knowledge from different angles rather than the same angle multiple times. In the end, we will know less.
kazinator 11y ago
It also casts a doubt on the independence of the verification. It will only catch outright cases of falsification, but innocent situations where the procedure is erroneous or the conclusions are wrongly drawn will slip through. If the reproducers don't quite know what they are doing if left to their own devices, they will invariably be influenced by the original researchers into being blind to exactly the same mistakes and making the same wrong inferences. Another software analogy: it's like QA people needing detailed hand-holding from the original developer in testing some program, instead of independently looking for ways to break it. That is then no better than the original functional testing by the developer.
- jsprogrammer 11y ago
  There are at least two distinct concepts here. 1) Exactly (as much as possible, given apparent environmental constraints) reproducing the procedure and results claimed and 2) attacking the procedure or results as ineffective or misleading (to what is actually being claimed). I'm not sure either is quite analogous to QA testing in the way you claimed though. Yes, QA testing should be reproducible and yes, QA testing should search for novel tests, but the scientific process is still a bit different and there are multiple distinct and important phases and concepts at play.
ColinDabritz 11y ago
That may be true, but one of the key ideals in science is that the findings are reproduceable. If I can't reproduce it, it's helpful for the science to be fairly specific, in case the findings depended on details that were thought to be unimportant, but we later find out were critical. The most interesting science comes in unexpected results. I feel it's like a piece of closed source software releasing "the source code" and yet no one can make it compile. You don't have to have a clean codebase, it can be a bit tricky to build, but it has to be able to be compiled before anyone can have any confidence in your claims. If you don't provide that, it's much less useful. The separate issues of for profit labs and what support they should expect in this context are tough issues, but we shouldn't undermine the science to thwart them.
- nmrm2 11y ago
  Thanks for giving an excellent example. NuPRL is exactly like this, but no one in their right mind would say the group that put out NuPRL hasn't done good science for 20+ years.
  - ColinDabritz 11y ago
    To clarify, the NuPRL project is like this in what way? They discuss cs topics and algorithms but no source code exists? or it exists but they don't provide it? Or (unlikely) they provide source code no one can compile? I don't want to mix up the side issue of "should code-based research always provide source code" with the metaphors being discussed. I definitely think useful CS research can happen without source code needing to exist, which I hope is the NuPRL project's case. I do feel code based research should provide source code that was created as part of the research when it is material to their claims, including 'we ran a simulation' and similar findings. If you already have that level of detail, you should include it. If you make claims based on data, you need to provide that data. If you make claims based on source code, you need to provide that source code.
    - nmrm2 11y ago
      > the NuPRL project is like this in what way? Difficult to obtain and get running (I'm told by everyone I've asked). So definitely not the first thing; mostly those last two things. And yeah, I think that high-quality source code is definitely a pretty impressive feat when researchers pull in off. And I also agree source code should be provided. But beyond providing a VM, I think it's really dangerous to sort of fixate on building software that's easy to setup/use. There's a pretty significant time cost there in some cases, and it's worth asking whether that's what we want to be spending our Science dollars on.
raldi 11y ago
You make a lot of assertions, but the closest you come to explaining the reason for any of those assertions is, "That's the way it's always been done." Can you elaborate on why this new approach isn't even worth trying?
- nmrm2 11y ago
  Lots of the replies have covered the whys: * Because this is in the context of a for profit company arguing that they should get a slice of the Science funding pie so that pharmaceutical companies have an easier time at reproducing results. It's unclear that this is a better use of funding than investing in new discoveries (unless you are profoundly suspicious of scientists and think they are all fabricating results), but it's fairly obvious why this aligns with the profit motives of pharmaceutical companies. I hope it's obvious why that's troubling to me and others. * Because there is value in reproducing results without doing everything exactly identically (see other posts). * Because there are other mechanisms for obtaining high-quality descriptions of protocols and it doesn't necessarily always make sense to invest lots of time in this early in the scientific process. * Because public morality panics rarely result in good policy making.
Panoramix 11y ago
Your analogies are completely off the mark. It is not the same at all. The cornerstone of science is that other people can reproduce your results. Period. There is no use publishing otherwise. Withholding key information, which is so ubiquitous now, is a great disservice to the scientific community.
- nmrm2 11y ago
  The question is not whether reproducibility is good, it's how much labs should invest upfront in producing descriptions of protocols. My argument is that they should probably invest more than they do now, but not enough that pharmacutical companies are able to reproduce a given experiment without talking to the lab. Science is a collaborative process. There's nothing wrong with collaboration being part of the reproducibility process, as long as the person doing the reproduction maintains their objectivity.
- coliveira 11y ago
  People often misunderstand the idea of reproducibility in science. The idea is not that scientists need to given a complete, easy to follow set of procedures that will let anyone reproduce an observation. In some areas this is close to impossible. For example, in high energy physics there is only one equipment in the world that can reproduce (with luck) some key experiment. The idea is that, with enough effort and funding somebody else could possibly reach the same conclusions. If you don't agree with some published results, there is an easy way to do it: create your own experiment and publish the results. If the result conflicts, then a new step is to determine why and in which conditions there is a conflict. Science evolves through the debate of ideas and observations, not because somebody is sharing a cookie-cut recipe.
mcguire 11y ago
Are the researches at the pharmaceutical companies really "literally anyone"? How do you "focus on the veracity" of the claim if you cannot reproduce it?
tjradcliffe 11y ago
If I claim, "An implementation of this algorithm (described in pseudocode) achieves O(N) peformance on that task" and no one can reproduce it based on the pseudocode, I've published a lie. This is a phenomenon I've seen many times in my work as a consultant translating academic discoveries to commercial code. This work invovles digging through both published papers and actual code to find that some grad student has implemented a clever heuristic that is not described in the paper because it is "inessential" that is what the claimed performance actually relies on. The article says, "It's unrealistic to think contract labs or university core facilities can get the same results as a highly specialized team of academic researchers, they say." I can understand this sentiment: science is hard, and at the cutting edge it's really hard. I've certainly got results that I think it would be hard for others to achieve and I've succeed with experiments where others have failed simply because I have good hands. I once took over an experiment that had been worked on by a heavy smoker with shaky hands. He had developed a complicated apparatus for streching ultra-thin plastic foils over a frame. I just used my fingers. If I had published the way I did it and someone like him had tried to reproduce it, they would have failed. However... the kind of reproducibilty being talked about here is not (primarily) technique, it is results. If a result can't be reproduced because the cell line was so very special or whatever, then it likely isn't robust enough to be very interesting in practical biology. It certainly isn't of much interest in drug discovery. So while the science may be OK, results that are incredibly hard to reproduce are of very low value for biotech, and that knowledge is worth something in and of itself.

astazangasta 11y ago

Looking at this through the lens of drug discovery is the wrong way to do this. The problem is with our drug discovery strategy, generally, not with the reproducibility of our research. STK33, for example, is definitely implicated in cancer through a wide variety of mechanisms. It is often mutated in tumors, and multiple studies have picked it up as having a role in driving migration, metastasis, etc. This doesn't mean we can make good drugs to it. Making drugs is hard - they need to be available in the tissue in the right concentrations, often difficult to achieve with a weird-shaped, sticky molecule. They need to have specificity for the tumor, they need to have specificity for the gene target(s) of interest. They need to be effective at modulating the target. More importantly, though, the drug is modulating a target (gene) that is involved in a biological system that involves complex systems of feedback control, produces adaptive responses, and otherwise behaves in unexpected ways in response to modulation. In my experience this is usually underappreciated by most drug discovery strategies, which merely seek to "inhibit the target" as if its involvement in the tumor process means we can simply treat it as an "on-off" switch for cancer. This assumption is asinine, and of course will (and does) lead to frequent failure. STK33 is not an on-off switch, and attempting to treat it that way will likely result in a drug that does nothing.

toufka 11y ago
This is absolutely correct. Pharma companies are running into a wall and are flailing to figure out what to do. It's quite clear that from first principles bathing the entire body in trillions of little molecules hoping that they only and completely shut down a single kind of protein, at the right time, in the right place, of the right cell, and do nothing more, is insane. There is some logic behind the ability for a small molecule to help against invading diseases (the antibiotics of the 20th century), but the same strategy will philosophically not work for entire classes of cancers or other innate biological problems. Though we all somehow just assumed that it would. The pharmacy of the future will be entirely curative to non-invading diseases, will repair the DNA that's been mutated, will express or inhibit the proteins that need to be expressed or inhibited. And small molecules will be the payload of these fancy protein-based nano-machines. But this hunt to bring down the cost of those small-molecule targets at the cost of the reputation of the science itself might be foolhardy when the cost is near-infinite in the first place because they're looking from the wrong perspective. tldr; Pharmaceutical companies' hammer that worked so well against the nails of bacterial infection is in no way suited to the plumbing of cancer. And now they're 'investigating the plumbers' to figure out why their fancy new 50-billion-dollar "water-hammers" don't work so well to unclog pipes.

lettergram 11y ago

Interesting... Over the past year I was tickling the idea of making an organization/company/website which automatically would give tax dollars to research. The idea being, you can maximize the amount you right off in taxes and donate to research you desire (i.e. more NASA, less children killed around the globe). The idea stemmed from the the idea that people want access to research (public) AND reproducible. The funds would go directly to research groups, and as an incentive, reproducibility would have bounties based on what people were willing to donate. Because virtually every research group with public research is supported by a non-profit, no one losses additional money, but more funds go towards public interest research GROUPS not organizations with bureaucracy. Somewhat off topic, but this seems another reason for me to start the project.

anonbanker 11y ago

the group clarified that they did not want to replicate the 30 or so experiments in the Cell paper, but just four described in a single key figure. And those experiments would be performed not by another academic lab working in the same area, but by an unnamed contract research organization. sounds like someone wants to quietly weaponize this.

gwern 11y ago

> This past January, the cancer reproducibility project published its protocol for replicating the experiments, and the waiting began for Young to see whether his work will hold up in their hands. He says that if the project does match his results, it will be unsurprising—the paper's findings have already been reproduced. If it doesn't, a lack of expertise in the replicating lab may be responsible. Either way, the project seems a waste of time, Young says. “I am a huge fan of reproducibility. But this mechanism is not the way to test it.” One swallow does not make a spring. With a belief like 'one replication is enough', I'm not sure Young actually appreciates how large sampling error is under usual significance levels or how high heterogeneity between labs is.

joe_the_user 11y ago
As far as I can tell from the article, Young's stance is that the contract lab doing the reproduction doesn't seem competent enough to correctly reproduce the experiment. I don't know enough to know if that's true or not. But it seems at least possible that in the rush to reproduce much, the project is cutting costs by using less-skilled contract labs.
- jsprogrammer 11y ago
  Presumably the replicating lab will record and document their approach and results so that it can be compared with the original experiment.
  - joe_the_user 11y ago
    Indeed, we can hope this. But as the op mentions, lab techniques vary and not everything can be documented. It's reasonable for a scientist to ask for a certain level in a lab attempting reproduction. I don't know if these scientists are demanding too much and have something to hide or if there are problem with the lab involved. However, the article actually mentions numerous scientists expressing misgivings.
    - jsprogrammer 11y ago

TisButMe 11y ago

This is the same behaviour I've seen time and time again in biology labs. People there are re-doing the same experiment over and over until it gives them the result they want, and then they publish that. It's the only field where I've heard people saying "Oh, yeah, my experiment failed, I have to do it again". What does it even mean that an experiment failed? It did exactly what it was supposed to: it gave you data. It didn't fit your expectations? Good, now you have a tool to refine your expectations. But instead, we see PhD students and post-doc working 70h hours week on experiments with seemingly random results until the randomness goes their way. A lot of them have no clue about statistical treatment of data, making a proper model to try and test assumptions against reality. Since they deal with insanely complicated system, with hidden variables all over the place, a proper statistical analysis would be the minimum expected to be able to extract any information from the data, but no matter, once you have a good looking figure, you're done. In cellular/molecular biology, nobody cares about what a p-value is, so as long as Excel tells you it's <0.05, you're golden. The scientific process has been forgotten in biology. Right now it's basically what alchemy was to chemistry. I very happy to see efforts like this one. Sure, they might show that a lot of "key" papers are very wrong, but that's not the crux of it. If there is a reason for biologists to make sure that their results are real, they might try to put a little more effort into checking their work. And when they figure out how much of it is bullshit, they might even try to slow down a little on the publications and go back to the basics for a little while. I'm sorry about this rant, but I've been driven away from a career in virology by those same issues, despite my love for the discipline, so I'm a bit bitter.

JoshTriplett 11y ago
https://xkcd.com/882/ https://xkcd.com/882/ Hopefully the people trying to reproduce results don't follow the same strategy. Also see https://en.wikipedia.org/wiki/Data_dredging https://en.wikipedia.org/wiki/Data_dredging .
- TisButMe 11y ago
  Yeah, I've seen this xkcd. It's on point as usual, but it doesn't show the frightening thing: it's the scientists who are going "Whoa" and believing in the results they've produced...

ryanobjc 11y ago

As a complete outsider reading the attitudes behind the original scientists, it seems to me that they resent the oversight and hate to do extra work. In defending their practices they fall back on "expert work" and essentially are arguing that what they are doing is too complex for anyone else to do and they should be left alone to continue to do it. And from their point of view, it seems all very reasonable. But from the rest of humanity who is being asked to materially support them, and waits for their conclusions to make the world a better place, it seems ... frankly... lazy and selfish. 30 emails, wow! 2 weeks of a graduate student's time -- these are the people who are the least paid right? Below minimum wage even? The demands on their time seem so low, yet the complaints are so high, that one can't help but wonder if the concern really is that their results are too 'magical' and irreproducible and they just fear other people learning about it. I've seen this behavior in professional settings, and ultimately it comes down to a lack of confidence in oneself, the tools and technology and the quality of work being done. Careers are at stake, but is the alternative to just give people a free pass?

nmrm2 11y ago
Sorry, but how is this not just anti-intellectualism? > 30 emails, wow! It can take up to half a day to reply to an email containing even one fairly technical question. Obviously the time invested depends on the content of those 30 emails, but it's not at all difficult to imagine this eating a week worth of a PI's time. > 2 weeks of a graduate student's time -- these are the people who are the least paid right? Below minimum wage even? The typical attitude is that this is a justification for not making them spend their time managing these sorts of tasks. You don't get to say "we get to pay you crap because your job is exciting and rewarding" and then load that same person down with a bunch of grunt work (on top of all the normal grunt work). That's how you bleed students and kill the lab's productivity. Also, grad students are not well paid but they are typically no cheaper than post docs; PIs pay tuition. > The demands on their time seem so low, yet the complaints are so high So let's imagine a world where people calling for massive improvements to reproducability get their way. Let's say it is a month worth of time for the lab. For each paper. Multiple papers a year. That's a pretty massive time investment. If you're a top lab, that could become a full-time position. And believe me, you're not going to be able to fill that position with a grad student. That person will have to be well-paid, because their job is going to suck. So it's reasonable that scientists are peeved when they invest all this time and don't perceive their collaborators as acting in good faith, or feel like their collaborators are trying to cut corners to pinch pennies.

azernik 11y ago

> It's unrealistic to think contract labs or university core facilities can get the same results as a highly specialized team of academic researchers, they say. Often a graduate student has spent years perfecting a technique using novel protocols, Young says. Then they need to spend the time documenting those protocols. My dad worked in biological research, and his attitude has always been: if you don't write it down, you might as well not have done the work at all. ESPECIALLY in research.

coliveira 11y ago

People often misunderstand the idea of reproducibility in science. The idea is not that scientists need to given a complete, easy to follow set of procedures that will let anyone reproduce an observation. In some areas this is close to impossible. For example, in high energy physics there is only one equipment in the world that can reproduce (with luck) some key experiment. The idea is that, with enough effort and funding somebody else could possibly reach the same conclusions. If you don't agree with some published results, there is an easy way to do it: create your own experiment and publish the results. If the result conflicts, then a new step is to determine why and in which conditions there is a conflict. Science evolves through the debate of ideas and observations, not because somebody is sharing a cookie-cut recipe.

chris_wot 11y ago

So hold on a moment. These researchers are doing experiments so badly that they can't find the actual procedures they used to get their results? And now they are tracking down old postdocs and lab technicians just to pick their Brian's as to what they actually did?!? How the heck did this stuff get through peer review? Surely I'm missing something critical?

infamouscow 11y ago
Academic peer review has nothing to do with quality control and everything to do with maintaining the status quo. Don't take my word for it, read Retraction Watch: http://retractionwatch.com/ http://retractionwatch.com/

x0054 11y ago

I tend to agree that the biology papers often lack proper documentation of procedures and methodologies. This is a wonderful effort to reproduce some of the key experiments. That being said, I think it's also very important to look at the quality and qualifications of the labs doing the reproductions. I don't have any direct link to cancer research, so I can't speak with authority on the subject, but I have been involved in the past with a company working in the Preimplantation Genetic Diagnosis field. The basics of their procedure is to create one or more human embryos via IVF, incubate the embryos for up to 6 days, than either freeze them, or transplant them into the prospective mother. On day 3 or 5 of incubations the embryo is biopsied, and the genetic material is tested to make sure there are no aneuploidy defects. We were also able to test for some other types of genetic abnormalities. This is for people who are having problem becoming pregnant. In any case, some time in the mid 2000s there were 3 papers published in Europe claiming that performing biopsy on Day 3 is extremely detrimental to the embryo, and their conclusion was that PGD with Day 3 should not be performed. The experiments were conducted by people who were unskilled in micro manipulation. They did follow proper protocols, and I am sure they did their best to replicate proper procedures. But micro manipulation is as much skill as it is knowledge. For instance, I can write a detailed procedure on how to shoot a compound bow, and you can follow that procedure exactly. But, without practice, you are not going to hit the bullseye on the first try. Because we were in the business of providing services to doctors, not publishing papers, we constantly tracked our embryo mortality rates, birth rates, and accuracy of testing. The better our results were, the more business we would get. And we couldn't fake the results, because clinics ordering the test would be the once recording all of those statistics for us. Any way, long story short, none of our data agreed with the papers claiming that Day 3 biopsy was detrimental to the embryo. In fact, quite the contrary, many of our statistics suggested that Day 3 biopsy and Day 4 or Day 5 transfer would result in better implantation rates. But, the papers were published, and referenced, and then it became "common knowledge" that Day 3 biopsy is bad, and the medical industry moved on to Day 5 biopsy and embryo cryopreservation, and so has the company I worked with. To the company I worked for it's all the same, money is money. Day 3 or Day 5 biopsy, they make money all the same. But the patients are not more limited. From the stats we have seen, it doesn't look like Day 4 or 5 biopsy is worse for the embryo, but being frozen isn't a walk in the park. With Day 5 biopsy you have to freeze the embryo, in order to allow time for the test results to come back. Any way, it's my 2 cents. Reproducibility is important, but I think it's just as important to change the incentives of those who publish papers. If you goal is to be published, then of course your research will suffer. It's the publish or parish mentality in academia that is the problem, I think.

cwyers 11y ago

> Jeff Settleman, who left academia for industry 5 years ago and is now at Calico Life Sciences in South San Francisco, California, agrees. “You can't give me and Julia Child the same recipe and expect an equally good meal,” he says. Settleman has two papers being replicated. Uh, Julia Child WROTE FREAKING COOKBOOKS. The entire point of Julia Child was that she tried to develop recipes in such a way that another cook could produce an equally good meal. Now, yes, if I went into a boiler room at Goldman Sachs and picked 10 guys at random, I doubt that most would be able to duplicate the recipe. If I picked 10 professional sous chefs at random and none of them were able to make a dish as good as Julia Child's from her recipe, I would start to have my doubts about the recipe. By the same token, I don't expect rank amateurs to be able to duplicate state of the art cancer research. But if labs run by pharma companies and academic institutions are having the failure rate at reproducing research that the article claims, I think it's more than reasonable to start questioning the paper that documented that research, if not the research itself.

untilHellbanned 11y ago
Analogy doesn't hold up. The complexity of cancer research is vastly greater than cooking. But anyway, chefs claim that there is special, non-reproducible elements about their environment and creations, e.g., oven, ingredients, etc. all the time. > none of them were able to make a dish as good Defining "good" is the problem for both cancer research and cooking. You can have two experts saying this is "good" or this is "bad" and not really be able to prove who is right. Fine, maybe for cancer research you can ultimately prove who is right but its not realistic in most cases given the resource constraints of even top-flight labs.
- cwyers 11y ago
  Well, it's not my analogy. But I think it's a good one, it just undermines the point of the person who made it. If I go into Momofuko, I'm looking for a good meal. A good meal is one that tastes good, in this case. If I'm looking at a Julia Child cookbook, I'm looking for a good recipe. One of the criteria for a good recipe may be that it tastes good, or that it produces a healthy meal -- there's several different criteria you can use here. But one criteria for a good recipe is reproducibility -- in order for a recipe to be good it must contain enough information and be accurate enough for me to create the dish that the recipe is for. A recipe for a tasty meal that does not contain the right ingredients or enough detail in the steps to prepare it is a bad recipe. By the same token, an experiment that cannot be repeated is a bad experiment. It may not be false. But its explanatory value is limited -- if a reaction can only take place in water that's treated a certain way or has/lacks certain minerals, then a paper that doesn't tell me that is leaving out important information. Regardless of whether or not you define the point of cancer research in purely scientific terms -- that is, to learn more about cancer -- or in more pragmatic terms -- that is, to allow us to create better cancer treatments -- omitted information about the circumstances surrounding the test that has a significant effect on the test result gives us less information and is less likely to lead to better cancer treatments.

platform 11y ago

I agree with the overall sentiment, the scientific community working on cancer cure(s) is failing us, the patients and their families. And they are failing us because of some fundamental gaps in how the research, and subsequent review/dissemination/presentation of finding is done. I suspect there are multiple failures in the process. The standards of scientific proof and repeatability, used by mathematicians, physicists and chemists are not followed The net result is the following disappointing statistic: "... In 1971, President Nixon and Congress declared war on cancer. Since then, the federal government has spent well over $105 billion on the effort (Kolata 2009b). ... Gina Kolata pointed out in The New York Times that the cancer death rate, adjusted for the size and age of the population, has decreased by only 5 percent since 1950 (Kolata 2009a)." [1] And this was just the US federal government investment. Not counting the private donations, and private company research. Today the annual fed investment is 5bln annually [3] I do not mean to sound totally discouraged, as clearly the screenings have helped many to detect cancers before they metastasized. And I would say the science results are showing that that part of the research is working well. However,for the cancers that can be rarely be detected before the spread (eg pancreatic cancer and others) -- the investment our country and other societies have put in -- simply has not payed off. What worries me is that our research quality gates are not able to improve the QoS of the underlying research. And with my 'management hat on' -- I am reaching out for this quote by Einstein. "Insanity: doing the same thing over and over again and expecting different results. The OP paper is not the first one pointing at the lack of reproducible results, and it not just for cancer research "... But it may also be due to current state of science. Scientists themselves are becoming increasingly concerned about the unreliability – that is, the lack of reproducibility — of many experimental or observational results. ..." [2] There needs to be a bit of a revolution in the science of the cancer research and the way money is allocated to it. Clearly the current model does not work and likely is encouraging the pseudo science to prosper. [1] http://www.csicop.org/si/show/war_on_cancer_a_progress_report_for_skeptics/ http://www.csicop.org/si/show/war_on_cancer_a_progress_repor... [2] http://www.forbes.com/sites/henrymiller/2014/01/08/the-trouble-with-scientific-research-today-a-lot-thats-published-is-junk/ http://www.forbes.com/sites/henrymiller/2014/01/08/the-troub... [3]http://blogs.reuters.com/stories-id-like-to-see/2014/09/09/the-money-spent-in-fighting-cancer-and-alibabas-risk-factor/ http://blogs.reuters.com/stories-id-like-to-see/2014/09/09/t...

nagaiah 11y ago

Is it possible to inject cancer resistant DNA into humans?