4 ms·
Negative results sit in a file drawer, or the trial keeps going in hopes the results turn positive. Therein lies the problem. We know the solution: tweak the p
by yummyfajitas 16y ago
Negative results sit in a file drawer, or the trial keeps going in hopes the results turn positive.
Therein lies the problem. We know the solution: tweak the publishing process.
Currently: 1) Do study 2) Submit paper, including conclusions. 3) Journal accepts, based on correctness of methodology and interestingness of conclusion.
To prevent this result, just change the order of things.
1) Submit paper explaining methodology, and "alternate endings" (positive and negative). 2) Journal accepts paper based on methodology only. Methodology section of paper is published. 3) Authors do study and submit results. Journal publishes appropriate conclusion section. 4) If necessary, authors submit additional observational results.
- niels_olson 16y agoThe International Committee of Medical Journal Editors has done something fairly similar (http://www.icmje.org/urm_main.html http://www.icmje.org/urm_main.html), and, in the US the government backs it up with a mandatory prepublication database of all ongoing trials: http://clinicaltrials.gov/ http://clinicaltrials.gov/ Also see Ionnidis's paper on why most published research is wrong: http://medicine.plosjournals.org/perlserv/?request=get-document&doi=10%2E1371%2Fjournal%2Epmed%2E0020124 http://medicine.plosjournals.org/perlserv/?request=get-docum...
- weego 16y ago_Negative results sit in a file drawer, or the trial keeps going in hopes the results turn positive._ There is a more unfortunate scenario that often plays out. That is that they get a different statistician in who has a goal of getting positive looking results out of a negative result set though various means and then often carefully manipulating the wording of the success criteria to match their best attempt at making the data work. This is quite common, my partner (in the girlfriend sense) work within medial journal publishing and sees it quite a lot.
- yummyfajitas 16y ago...manipulating the wording of the success criteria to match their best attempt at making the data work. While not impossible, it is very tricky to do this before you actually have the data. I'm advocating that the wording of the test criteria, statistical tests to perform, etc, are all decided before the trial. Bringing in a different statistician post-hoc is pointless, all he could do is run the exact predetermined statistical test. In fact, you can do even better: submit code to the journal before the study, and submit data after the study. The journal runs the code on the data and decides which conclusion to publish.
- deleted 16y ago[deleted]
- rdtsc 16y agoVery good point. The "good" statistician then gets a bonus at the end of the year for being productive and helpful. The "honest" statistician gets slowly pushed out. Accumulate this over the whole range of companies and studies and we end with a disaster.
- blahblahblah 16y agoIt's not just the publishing process that's broken. The funding process is problematic as well. Among the competitors for grant money, it is the investigators that promise to do a lot with the minimal amount of funding that can reasonably be believed to be feasible for the project who end up winning the grant money. This optimizes the funding process to select for proposals that aim to collect the bare minimum of data that the statistician says is required to see a "significant" result (because data collection is the expensive part of doing a study). If the investigator's estimate of the effect size is off, then the sample size is insufficient and spurious results are more likely. Change the funding process to select for proposals that involve collecting a sample size that is larger than strictly necessary for the expected effect size by some margin of safety and you'll get research results that are more likely to actually be true.
- forensic 16y agoThe people commissioning these studies need to somehow be liable for the accuracy of the results. If you were doing a private study where the results would only be for your own private usage, you would go the extra mile and pay the extra money for good, accurate science. But the financiers here don't really care about whether the result is accurate. They usually are optimizing something else entirely and accuracy is just some minimum threshold to avoid being accused of fraud.
- krakensden 16y agoLiable seems too strong- there are lots of ways for studies to go wrong, that you may not figure out for years, through no particular malice or lack of competence.
- protomyth 16y ago> Negative results sit in a file drawer, or the trial keeps going in hopes the results turn positive. I can't say a whole lot about it (I am pretty sure I am still bound by agreements and truthfully it wouldn't be polite), but I once was employed in clinical trials and this line is a load of crap. I have seen trials stopped. Some came back, but it wasn't the same trial.
- nkurz 16y agoI presume you are right, but this doesn't inspire confidence in the system. Of course it's not the same trial: if you have a weak effect, it's better to modify a superficial variable and start the trial again. Why try to climb out of the hole when you can roll the dice again? Do this 20 times and you might stumble on something statistically significant. From a 'true science' perspective, the right thing to do is to carry the trial through to conclusion and publish the negative results. But it's hard to make a living doing this.
- protomyth 16y agoGiven that a FDA trial can last years and cost a lot of money, a company can't really afford to "carry on" if it is obvious to themselves and the people they have conducting the trial that something is very different from their predicted results. They had a belief in what the outcome should be before the trial (else you would not go through with / spend money on the trial). A deviation from the expected results really needs to be looked at and you really need to find out why all the prelim testing is not accurately predicting what is actually going on.
- nkurz 16y agoYes, it's not that the drug companies are actually running trials and random hoping for spurious correlations, but that the traditional measures of statistical significance can't be used in cases where trials might be stopped early. Even inspecting the results midway through a trial to see if the effect is being shown is problematic. Yes, the deviation from expectation really needs to be looked at. The problem is that the financial reward for the companies might be greater to just 'move on' and never publish those unexpectedly negative results. But this means that the unexpectedly positive results for the next test are given undue weight. I don't think the problem is really the science, or the publishers, or the drug companies, but the unrealistic weight placed on erroneously calculated statistics. This combined with a lot of money at stake forces the companies to "produce results" whether or not this is of long term public benefit.