8 ms·

To Get More Out of Science, Show the Rejected Research

204 points by andrewl 12y ago

tim333 12y ago
The process generally seems a bit broken what with the concerns in the article and with Elsevier making everything only available to those who pay loads. You'd think you could have something like arXiv and a rating service to figure out which research was good and worth reading and should be rewarded career wise. Something for a YC start up to fix?
- capnrefsmmat 12y ago
  There is something like that in biology: http://f1000.com/prime http://f1000.com/prime ...but getting sufficient widespread adoption is the big problem. And given that peer reviewers are not particularly effective in improving paper quality except in the most egregious cases, we should wonder whether an entirely different model is more appropriate.
Shivetya 12y ago
There is the big problem now of negative results not being published, mainly because of the competition for federal (government worldwide) funding does not lend itself to proving something "otherwise". http://eloquentscience.com/wp-content/uploads/2012/02/Fanelli12-NegativeResults.pdf http://eloquentscience.com/wp-content/uploads/2012/02/Fanell...
denzil_correa 12y ago
There are two issues here - (1) Irreproducible research and (2) Negative results. (1) is clearly a problem and must be dealt with by scientific community and processes. Let me talk about (2). For negative results to be published, they too should follow basic patterns of positive results - innovative, scientifically rigorous. There are always more negative results possible than positive results. A negative result should be something which people think would intuitively work but wouldn't. For example - an apple falling from a tree and floating in thin air isn't a negative result because we all know that it's supposed to fall down to the ground via gravity. Edit - We also have a Journal of Negative Results http://jnr-eeb.org/index.php/jnr http://jnr-eeb.org/index.php/jnr
- biehl 12y ago
  A negative result is also interesting if a positive result has been previously reported.
  - denzil_correa 12y ago
    That would be irreproducible research.
netcan 12y ago
We kind of think of science as an embodiment of modernity, and therefore modern. What it is though is an institution, similar to academia and fairly old. Human institutions take time to change. In any case, I'm pretty excited that it's coming under pressure to improve. Publication is really a method of communication and the revolution in communication of the last generation is a profound step change in human history, in my opinion. To use some terms that our great predecessors would have been comfortable with, science is a way to uncover the truth using light. Experimentation, debate, publication, review: these are all ways of making light. Bringing modern communication into science and the collaborative opportunities inherent in better communication is a potentially very bright light. Nature, and Nature's Laws lay hid in Night. God said, 'Let Newton be!' and all was Light. -- Alexander Pope Reproducibility and negative results are two parts of the same problem, and fundamental one in science since the beginning. A better method for solving it (using a computer (: ), is probably coming. If not now, within ten years. Maybe twenty. Soon.
vanderZwan 12y ago
There's many more issues distorting or distorted by academic publishing. For example, grants being awarded to the most productive members. Sounds fine at face value, but compare biologists working with ecological systems and biologists working with DNA. In the first case collecting data by definition has to take decades, in the other it's a matter of hours these days. The ecologists needs more money because it inherently takes longer to research, yet the grants are more likely to be awarded to the DNA researchers because they publish more. DISCLAIMER: I'm not a biologist myself - this is a second-hand story from a biologist friend so if the story doesn't hold up under close scrutiny, my apologies.
- Malarkey73 12y ago
  Yes and No. Yes ecologists take more time to collect data. However ecology is much cheaper than mol. biology. Essentially its ecology = manpower vs mol. biology = manpower + expensive technology + expensive chemistry. Also ecology has been one of the most resistant fields to sharing data whereas genomics (the big data end of mol. biology) has been the most enthusiastic. There was a vey noticeable divide in opinion on twitter when PLOS journals mooted making authors make all their data public. I think this is precisely because ecologists invest so much on collecting their field data, so they become possessive. Ecologists can become very senior PI by essentially owning a long term ecological study and assigning their students to continuing it. Genomics and to a lesser extent Mol. Biol. has been keener recently to share as there is a strong open source ethos coming in from the computing side - and it is often a very expensive multi-centre, multi-agency collaboration.
- pcrh 12y ago
  Within fields this is taken into account. Some areas can be very quick to generate data, whereas others take longer and /or more manpower. One would not expect someone doing studies in mice, or clinical studies, to generate the same number of papers as one doing biochemistry, for example. The incentive to publish frequently is not universal, however, some reviewing committees ask only for one to present the best 5 papers in the last 10 years, for example.
mherdeg 12y ago
Just gonna leave a quick link to my favorite general-purpose academic journal: http://www.jasnh.com/ http://www.jasnh.com/ The Journal of Articles in Support of Null Hypothesis collects experiments that didn't work. Not very much volume, not a huge area of prestige, but there should be no shame in publishing there. The content is very diverse and pretty fun. Titles like "No Effect of a Brief Music Intervention on Test Anxiety and Exam Scores in College Undergraduates"; "Parenting Style Trumps Work Role in Life Satisfaction of Midlife Women"; "Does Fetal Malnourishment Put Infants at Risk of Caregiver Neglect Because Their Faces Are Unappealing?"; "Is There an Effect of Subliminal Messages in Music on Choice Behavior?". Plenty more cool stuff.
- chrisBob 12y ago
  This is interesting, but I think there needs to be an option that takes the next step. While this journal is for psychology only I would like to see a broad Journal of Negative Results. This would help limit repeated, failed experiments, and could do as much to help new discoveries as the current publications which are a collection of techniques that happened to work. Everyone says you should learn from your mistakes, but then they are never shared for others to learn from.
  - mherdeg 12y ago
    Yeah, it does seem like in general in science, we only see "we tried to do X and it didn't work" papers AFTER someone has published a paper that says "surprisingly, we tried to do X which shouldn't work and it did!". The example that comes to mind is pluripotent stem cells (see e.g. discussion at http://www.plosbiology.org/article/info%3Adoi%2F10.1371%2Fjournal.pbio.0050338 http://www.plosbiology.org/article/info%3Adoi%2F10.1371%2Fjo... ) where publications only really advance knowledge about the "dogma" of a field when someone finds a way to publish results that oppose it.
mnw21cam 12y ago
For an instructive insight into the dangers of only publishing positive results, see http://xkcd.com/882/ http://xkcd.com/882/
smaldj 12y ago
Ben Goldacre has been rallying around this cause for years now. He has: 1. presented a TED talk (http://www.ted.com/talks/ben_goldacre_battling_bad_science?language=en http://www.ted.com/talks/ben_goldacre_battling_bad_science?l...), 2. authored books (http://www.amazon.co.uk/dp/0007350740/ http://www.amazon.co.uk/dp/0007350740/), and 3. written numerous articles (http://www.theguardian.com/profile/bengoldacre http://www.theguardian.com/profile/bengoldacre) on the misrepresentation of statistics within the pharmaceutical community. Probably the best summary post of his is this one: http://www.theguardian.com/commentisfree/2011/nov/04/bad-science-eight-years http://www.theguardian.com/commentisfree/2011/nov/04/bad-sci...
- duwease 12y ago
  Bad Pharma is truly an eye-opener as to the extent of the troubles faced in getting truth out of medical research. It's going to take many, many, different group combining foresight, technology, and good-old-fashioned political clout to even start fixing it. John Ioannidis' work is fascinating in that respect as well.. not just in the medical field, but science in general.
- StavrosK 12y ago
  He also recommended that a registry be instituted, where you have to register before starting research, and you HAVE to publish your result, negative or not. If you haven't registered beforehand, you can't publish at all. Sounds like a simple and effective solution.
CognitiveLens 12y ago
The suggestion in the article to pre-register trials is a good one, but I'm extremely wary of a more general effort to "publish rejected research" because there is a huge quantity of very poorly conducted research that really does not deserve publication. Most fields are already drowning in a sea of journal articles - few researchers are aware of all the published studies that might be relevant to their own work - and greatly increasing the quantity of published material will dilute the pool even further. Replicating findings should be given higher priority, pre-registering methods and analyses should be encouraged/required, but it's important to stop short of "publish all the things".
- jonwachob91 12y ago
  So don't publish the shit research in quality journals. Publish them in a general journal for negative findings. I'd also like to know who conducts quality research and who conducts shit research, cause that might influence my funding/spending patterns (if I had money to spend).
  - tectec 12y ago
    "Shit research" is not equal to negative findings. In fact it may be the opposite. I suspect most bad research has the positive findings the researchers are looking for.
unclebunkers 12y ago
I don't think they are aware of how much bad science is out there and how many people are trying to publish it. It wouldn't be a journal every month, it would be a phone book every week. Corporations would simply drown out real science with papers designed to support whatever narrative they were promoting. Where this may make sense is when Watson grows up, and you can aggregate the volume of garbage to fill in the holes of knowledge. But that's more than a couple years off I suspect.
- svsaraf 12y ago
  I think the author addressed your concern: > Ask journal editors and scientific peers to review study designs and analysis plans and commit to publish the results if the study is conducted and reported in a professional manner (which will be ensured by a second round of peer review). If the study design is peer reviewed, I would hazard a guess that there would be less bad science, not more. Currently study design takes second priority to significant results, which is perhaps why we have the problems of inconsistent research in the first place.
  - nhstanley 12y ago
    I think that would be a great step. Science would be greatly advanced if there was more focus on whether a good question was asked and whether the experiments were well chosen, than whether the experiments gave positive/negative results.
- pcrh 12y ago
  A lot of bad science is published. I would guess that more than 50% of published articles in the biomedical field are not worth anything, even as "negative results".
kyro 12y ago
I have always had a huge problem with the non-reproducible nature of medical research and its acceptance within the field. Being in medicine, every day you hear a physician citing some study from 15 years ago conducted on a sample size of 20/50/100 or so patients as a way to justify their clinical decisions. And it always worries me that we tend to put so much faith in these "landmark" studies, as if their findings are somehow legitimate and true because statistical significance was reached at least once, and seem to forget all of the misaligned incentives and game-playing that goes on in research, and the amount of data burying by the FDA, all that almost certainly influence and manipulate data, publication, etc in a negative way. I tend to take the majority of medical research with a grain of salt, for the reasons listed here and in the article, unless there's some very convincing meta-analysis or successfully reproduced evidence. Call me overly cynical, but that we calculate a parameter or administer a drug or change our methods because of some article you read last month in NEJM is beyond bogus.
- biomcgary 12y ago
  I'm a computational biologist working at the intersection of basic research and medical research. I would recommended more than one grain of salt with medical research. Medically trained folks often fail to understand study design and are rarely rigorous thinkers. The heavy emphasis on memorization in medical school is a substantial part of the problem. And, those are problems even before perverse incentives kick in.
  - pcrh 12y ago
    In medical research there is also the additional handicap of career structure. The extremely hierarchical nature of medical careers, and the substantial financial rewards for "playing along", mean that there are strong disincentives to rock the boat by criticizing the hypotheses of senior members. This often results in byzantine twists of logic being used to justify continuing with particular lines of enquiry, such as the amyloid hypothesis in Alzheimer's disease (my field).
- DMac87 12y ago
  Ditto for academic finance - so many sketchy results get advertised, and billions if not trillions of dollars get invested based on false or uncertain hypotheses... Sometimes I feel that 90% of my job is to be skeptical of all results, and convince managers not to do anything stupid - a sort of "first, do no harm" philosophy.
tjradcliffe 12y ago
In physics, we have been aggressively publishing negative results for decades. There is an entire field dedicated to such work, called "Physics Beyond the Standard Model" (spoiler: there isn't any). I've seen entire careers of extremely good experimentalists dedicated to "failing to reject the null hypothesis" at more an more stringent limits (neutrinoless double beta decay is a good example of this) and have been to week-long conferences where every single paper was either a crazy theory or a negative experimental result. I left the field 15 years ago because I didn't want to spend my career measuring zero, and presumably over time it will eventually dry up. The conditions for its existence seem to be more to do with having a highly trained group of people who have exhausted all plausible avenues of research in a given area and are left chasing a few scraps. In areas where there are still plenty of positive results to be had, the tendency will always be to emphasize the positive. As a partial solution to this tendency, in my applied physics work, where I did get positive results, I tried to include a section in papers entitled "Things That Didn't Work So Well" that sketched failed approaches to save other people the trouble of trying stuff that seemed like a good idea but didn't pan out... at the very least we should expect that from the average publication, and be suspicious of any experimental paper that does not include some description of the blind alleys.
- ISL 12y ago
  A counterpoint -- neutrino oscillation, dark matter, dark energy, and gravity all exist outside the standard model. If we are to believe that there's some larger theory that ties everything together, then there's something missing. That we have done such extensive searches without finding anything is an amazing property of nature. The growing body of experimental null searches serve to provide tighter and tighter constraints on our understanding of nature. If you put your best new idea to the test, and it fails, it's not a failure, it's a success. Now you know your best idea, the one you thought could lead you forward, is a dead end. Your next steps will be better-guided and take you further forward.
  - eru 12y ago
    Yes. It's just that biologists don't think like this (yet?).
vedtopkar 12y ago
A lot of this has to do with incentive structures, especially in Life Science research. The grant landscape is intensely competitive, and writing up results is incredibly time-consuming. There is little incentive to take the time to write up and submit negative results to relevant journals. If institutions and grant committees were to require this practice, it wouldn't be nearly as big of a problem.
bmh100 12y ago
It is tragic to imagine the amount of time wasted by repeating the unpublished experiments of others. It is even more tragic to imagine that someone might be able to gain a hidden insight by finding the gaps in various negative results, which might remain undiscovered for a long time otherwise.
- monksy 12y ago
  The problem is that there is such a huge priority on successful research and there is a self-consciousness about funding/research that didn't go as well as expected.
  - bmh100 12y ago
    Indeed. It would be ironic if some of those less successful results were the result of hiding earlier, less successful results.
schrodingersCat 12y ago
I haven't read all the responses, so I hope I am not repeating anyone's insight. I'm finishing my PhD in biophysics, and I wanted to share my perspective from conversations I've had with my boss/PI and other investigators. In life science research, there are huge incentives to not repeat others research and furthermore to not publish negative results. The first disincentive comes from funding bodies: NIH et. al (NIGMS, NIEHS, ...) don't like to pay for you to do "someone else's science". If you manage to get a grant, and it comes out in a progress report that you did repeat too much of other people work, be prepared to get that funding reduced and or cut. Academic departments strongly discourage new hires from publishing negative results and /or repeating other peoples work (mostly because this will likely decrease chances of getting published and funded). Academic journals hate to publish negative results, but seemingly have no problem publishing bad science (yes Nature, I'm looking at you: http://retractionwatch.com/2014/09/11/potentially-groundbreaking-highly-provocative-nature-stap-cell-peer-reviews-published/ http://retractionwatch.com/2014/09/11/potentially-groundbrea...). Early in my PI's career, she tried to publish a very important negative fining in a high impact journal. The article's acceptance was accompanied by a personal letter from the editor urging her to consider other journals for negative results. Another barrier quite honestly is ego. While it may sound as if my boss is "one of the good ones", alas, she is not. On occasions that I have asked to repeat other group's seemingly unbelievable results myself, I've been flatly denied on grounds that this kind of work does not express the sort of originality of research produced by her lab. In other words, nobody wants to be known as "that lab", the nay-sayers of the field, those that would dare to question a colleague's ideas. Finally, this lead me to the last barrier I have observed: scientific communities / societies. If you are of the lucky few that end up publishing negative results of major significance, prepare to not be invited to dinner at next years Society for X annual meeting. Yes, in many ways life-science is stratified just like high school. You have the cool kids on track for the nobel, the weirdoes in their corner pushing the boundaries of what is possible, the "jocks"/ career scientists who manage to turn a couple of tricks and some charisma into a living, and finally the tattle-tales who seem to piss everyone off with their negative results. These are HUGE oversimplifications / generalizations, but I really think that all of these barriers need to be addressed in some way to fix life science.
- eru 12y ago

204 points by andrewl 12y ago

The process generally seems a bit broken what with the concerns in the article and with Elsevier making everything only available to those who pay loads. You'd think you could have something like arXiv and a rating service to figure out which research was good and worth reading and should be rewarded career wise. Something for a YC start up to fix?

capnrefsmmat 12y ago
There is something like that in biology: http://f1000.com/prime http://f1000.com/prime ...but getting sufficient widespread adoption is the big problem. And given that peer reviewers are not particularly effective in improving paper quality except in the most egregious cases, we should wonder whether an entirely different model is more appropriate.

Shivetya 12y ago

There is the big problem now of negative results not being published, mainly because of the competition for federal (government worldwide) funding does not lend itself to proving something "otherwise". http://eloquentscience.com/wp-content/uploads/2012/02/Fanelli12-NegativeResults.pdf http://eloquentscience.com/wp-content/uploads/2012/02/Fanell...

denzil_correa 12y ago

There are two issues here - (1) Irreproducible research and (2) Negative results. (1) is clearly a problem and must be dealt with by scientific community and processes. Let me talk about (2). For negative results to be published, they too should follow basic patterns of positive results - innovative, scientifically rigorous. There are always more negative results possible than positive results. A negative result should be something which people think would intuitively work but wouldn't. For example - an apple falling from a tree and floating in thin air isn't a negative result because we all know that it's supposed to fall down to the ground via gravity. Edit - We also have a Journal of Negative Results http://jnr-eeb.org/index.php/jnr http://jnr-eeb.org/index.php/jnr

biehl 12y ago
A negative result is also interesting if a positive result has been previously reported.
- denzil_correa 12y ago
  That would be irreproducible research.

netcan 12y ago

We kind of think of science as an embodiment of modernity, and therefore modern. What it is though is an institution, similar to academia and fairly old. Human institutions take time to change. In any case, I'm pretty excited that it's coming under pressure to improve. Publication is really a method of communication and the revolution in communication of the last generation is a profound step change in human history, in my opinion. To use some terms that our great predecessors would have been comfortable with, science is a way to uncover the truth using light. Experimentation, debate, publication, review: these are all ways of making light. Bringing modern communication into science and the collaborative opportunities inherent in better communication is a potentially very bright light. Nature, and Nature's Laws lay hid in Night. God said, 'Let Newton be!' and all was Light. -- Alexander Pope Reproducibility and negative results are two parts of the same problem, and fundamental one in science since the beginning. A better method for solving it (using a computer (: ), is probably coming. If not now, within ten years. Maybe twenty. Soon.

vanderZwan 12y ago

There's many more issues distorting or distorted by academic publishing. For example, grants being awarded to the most productive members. Sounds fine at face value, but compare biologists working with ecological systems and biologists working with DNA. In the first case collecting data by definition has to take decades, in the other it's a matter of hours these days. The ecologists needs more money because it inherently takes longer to research, yet the grants are more likely to be awarded to the DNA researchers because they publish more. DISCLAIMER: I'm not a biologist myself - this is a second-hand story from a biologist friend so if the story doesn't hold up under close scrutiny, my apologies.

Malarkey73 12y ago
Yes and No. Yes ecologists take more time to collect data. However ecology is much cheaper than mol. biology. Essentially its ecology = manpower vs mol. biology = manpower + expensive technology + expensive chemistry. Also ecology has been one of the most resistant fields to sharing data whereas genomics (the big data end of mol. biology) has been the most enthusiastic. There was a vey noticeable divide in opinion on twitter when PLOS journals mooted making authors make all their data public. I think this is precisely because ecologists invest so much on collecting their field data, so they become possessive. Ecologists can become very senior PI by essentially owning a long term ecological study and assigning their students to continuing it. Genomics and to a lesser extent Mol. Biol. has been keener recently to share as there is a strong open source ethos coming in from the computing side - and it is often a very expensive multi-centre, multi-agency collaboration.
pcrh 12y ago
Within fields this is taken into account. Some areas can be very quick to generate data, whereas others take longer and /or more manpower. One would not expect someone doing studies in mice, or clinical studies, to generate the same number of papers as one doing biochemistry, for example. The incentive to publish frequently is not universal, however, some reviewing committees ask only for one to present the best 5 papers in the last 10 years, for example.

mherdeg 12y ago

Just gonna leave a quick link to my favorite general-purpose academic journal: http://www.jasnh.com/ http://www.jasnh.com/ The Journal of Articles in Support of Null Hypothesis collects experiments that didn't work. Not very much volume, not a huge area of prestige, but there should be no shame in publishing there. The content is very diverse and pretty fun. Titles like "No Effect of a Brief Music Intervention on Test Anxiety and Exam Scores in College Undergraduates"; "Parenting Style Trumps Work Role in Life Satisfaction of Midlife Women"; "Does Fetal Malnourishment Put Infants at Risk of Caregiver Neglect Because Their Faces Are Unappealing?"; "Is There an Effect of Subliminal Messages in Music on Choice Behavior?". Plenty more cool stuff.

chrisBob 12y ago
This is interesting, but I think there needs to be an option that takes the next step. While this journal is for psychology only I would like to see a broad Journal of Negative Results. This would help limit repeated, failed experiments, and could do as much to help new discoveries as the current publications which are a collection of techniques that happened to work. Everyone says you should learn from your mistakes, but then they are never shared for others to learn from.
- mherdeg 12y ago
  Yeah, it does seem like in general in science, we only see "we tried to do X and it didn't work" papers AFTER someone has published a paper that says "surprisingly, we tried to do X which shouldn't work and it did!". The example that comes to mind is pluripotent stem cells (see e.g. discussion at http://www.plosbiology.org/article/info%3Adoi%2F10.1371%2Fjournal.pbio.0050338 http://www.plosbiology.org/article/info%3Adoi%2F10.1371%2Fjo... ) where publications only really advance knowledge about the "dogma" of a field when someone finds a way to publish results that oppose it.

mnw21cam 12y ago

For an instructive insight into the dangers of only publishing positive results, see http://xkcd.com/882/ http://xkcd.com/882/

smaldj 12y ago

Ben Goldacre has been rallying around this cause for years now. He has: 1. presented a TED talk (http://www.ted.com/talks/ben_goldacre_battling_bad_science?language=en http://www.ted.com/talks/ben_goldacre_battling_bad_science?l...), 2. authored books (http://www.amazon.co.uk/dp/0007350740/ http://www.amazon.co.uk/dp/0007350740/), and 3. written numerous articles (http://www.theguardian.com/profile/bengoldacre http://www.theguardian.com/profile/bengoldacre) on the misrepresentation of statistics within the pharmaceutical community. Probably the best summary post of his is this one: http://www.theguardian.com/commentisfree/2011/nov/04/bad-science-eight-years http://www.theguardian.com/commentisfree/2011/nov/04/bad-sci...

duwease 12y ago
Bad Pharma is truly an eye-opener as to the extent of the troubles faced in getting truth out of medical research. It's going to take many, many, different group combining foresight, technology, and good-old-fashioned political clout to even start fixing it. John Ioannidis' work is fascinating in that respect as well.. not just in the medical field, but science in general.
StavrosK 12y ago
He also recommended that a registry be instituted, where you have to register before starting research, and you HAVE to publish your result, negative or not. If you haven't registered beforehand, you can't publish at all. Sounds like a simple and effective solution.

CognitiveLens 12y ago

The suggestion in the article to pre-register trials is a good one, but I'm extremely wary of a more general effort to "publish rejected research" because there is a huge quantity of very poorly conducted research that really does not deserve publication. Most fields are already drowning in a sea of journal articles - few researchers are aware of all the published studies that might be relevant to their own work - and greatly increasing the quantity of published material will dilute the pool even further. Replicating findings should be given higher priority, pre-registering methods and analyses should be encouraged/required, but it's important to stop short of "publish all the things".

jonwachob91 12y ago
So don't publish the shit research in quality journals. Publish them in a general journal for negative findings. I'd also like to know who conducts quality research and who conducts shit research, cause that might influence my funding/spending patterns (if I had money to spend).
- tectec 12y ago
  "Shit research" is not equal to negative findings. In fact it may be the opposite. I suspect most bad research has the positive findings the researchers are looking for.

unclebunkers 12y ago

I don't think they are aware of how much bad science is out there and how many people are trying to publish it. It wouldn't be a journal every month, it would be a phone book every week. Corporations would simply drown out real science with papers designed to support whatever narrative they were promoting. Where this may make sense is when Watson grows up, and you can aggregate the volume of garbage to fill in the holes of knowledge. But that's more than a couple years off I suspect.

svsaraf 12y ago
I think the author addressed your concern: > Ask journal editors and scientific peers to review study designs and analysis plans and commit to publish the results if the study is conducted and reported in a professional manner (which will be ensured by a second round of peer review). If the study design is peer reviewed, I would hazard a guess that there would be less bad science, not more. Currently study design takes second priority to significant results, which is perhaps why we have the problems of inconsistent research in the first place.
- nhstanley 12y ago
  I think that would be a great step. Science would be greatly advanced if there was more focus on whether a good question was asked and whether the experiments were well chosen, than whether the experiments gave positive/negative results.
pcrh 12y ago
A lot of bad science is published. I would guess that more than 50% of published articles in the biomedical field are not worth anything, even as "negative results".

kyro 12y ago

I have always had a huge problem with the non-reproducible nature of medical research and its acceptance within the field. Being in medicine, every day you hear a physician citing some study from 15 years ago conducted on a sample size of 20/50/100 or so patients as a way to justify their clinical decisions. And it always worries me that we tend to put so much faith in these "landmark" studies, as if their findings are somehow legitimate and true because statistical significance was reached at least once, and seem to forget all of the misaligned incentives and game-playing that goes on in research, and the amount of data burying by the FDA, all that almost certainly influence and manipulate data, publication, etc in a negative way. I tend to take the majority of medical research with a grain of salt, for the reasons listed here and in the article, unless there's some very convincing meta-analysis or successfully reproduced evidence. Call me overly cynical, but that we calculate a parameter or administer a drug or change our methods because of some article you read last month in NEJM is beyond bogus.

biomcgary 12y ago
I'm a computational biologist working at the intersection of basic research and medical research. I would recommended more than one grain of salt with medical research. Medically trained folks often fail to understand study design and are rarely rigorous thinkers. The heavy emphasis on memorization in medical school is a substantial part of the problem. And, those are problems even before perverse incentives kick in.
- pcrh 12y ago
  In medical research there is also the additional handicap of career structure. The extremely hierarchical nature of medical careers, and the substantial financial rewards for "playing along", mean that there are strong disincentives to rock the boat by criticizing the hypotheses of senior members. This often results in byzantine twists of logic being used to justify continuing with particular lines of enquiry, such as the amyloid hypothesis in Alzheimer's disease (my field).
DMac87 12y ago
Ditto for academic finance - so many sketchy results get advertised, and billions if not trillions of dollars get invested based on false or uncertain hypotheses... Sometimes I feel that 90% of my job is to be skeptical of all results, and convince managers not to do anything stupid - a sort of "first, do no harm" philosophy.

tjradcliffe 12y ago

In physics, we have been aggressively publishing negative results for decades. There is an entire field dedicated to such work, called "Physics Beyond the Standard Model" (spoiler: there isn't any). I've seen entire careers of extremely good experimentalists dedicated to "failing to reject the null hypothesis" at more an more stringent limits (neutrinoless double beta decay is a good example of this) and have been to week-long conferences where every single paper was either a crazy theory or a negative experimental result. I left the field 15 years ago because I didn't want to spend my career measuring zero, and presumably over time it will eventually dry up. The conditions for its existence seem to be more to do with having a highly trained group of people who have exhausted all plausible avenues of research in a given area and are left chasing a few scraps. In areas where there are still plenty of positive results to be had, the tendency will always be to emphasize the positive. As a partial solution to this tendency, in my applied physics work, where I did get positive results, I tried to include a section in papers entitled "Things That Didn't Work So Well" that sketched failed approaches to save other people the trouble of trying stuff that seemed like a good idea but didn't pan out... at the very least we should expect that from the average publication, and be suspicious of any experimental paper that does not include some description of the blind alleys.

ISL 12y ago
A counterpoint -- neutrino oscillation, dark matter, dark energy, and gravity all exist outside the standard model. If we are to believe that there's some larger theory that ties everything together, then there's something missing. That we have done such extensive searches without finding anything is an amazing property of nature. The growing body of experimental null searches serve to provide tighter and tighter constraints on our understanding of nature. If you put your best new idea to the test, and it fails, it's not a failure, it's a success. Now you know your best idea, the one you thought could lead you forward, is a dead end. Your next steps will be better-guided and take you further forward.
- eru 12y ago
  Yes. It's just that biologists don't think like this (yet?).

vedtopkar 12y ago

A lot of this has to do with incentive structures, especially in Life Science research. The grant landscape is intensely competitive, and writing up results is incredibly time-consuming. There is little incentive to take the time to write up and submit negative results to relevant journals. If institutions and grant committees were to require this practice, it wouldn't be nearly as big of a problem.

bmh100 12y ago

It is tragic to imagine the amount of time wasted by repeating the unpublished experiments of others. It is even more tragic to imagine that someone might be able to gain a hidden insight by finding the gaps in various negative results, which might remain undiscovered for a long time otherwise.

monksy 12y ago
The problem is that there is such a huge priority on successful research and there is a self-consciousness about funding/research that didn't go as well as expected.
- bmh100 12y ago
  Indeed. It would be ironic if some of those less successful results were the result of hiding earlier, less successful results.

schrodingersCat 12y ago

I haven't read all the responses, so I hope I am not repeating anyone's insight. I'm finishing my PhD in biophysics, and I wanted to share my perspective from conversations I've had with my boss/PI and other investigators. In life science research, there are huge incentives to not repeat others research and furthermore to not publish negative results. The first disincentive comes from funding bodies: NIH et. al (NIGMS, NIEHS, ...) don't like to pay for you to do "someone else's science". If you manage to get a grant, and it comes out in a progress report that you did repeat too much of other people work, be prepared to get that funding reduced and or cut. Academic departments strongly discourage new hires from publishing negative results and /or repeating other peoples work (mostly because this will likely decrease chances of getting published and funded). Academic journals hate to publish negative results, but seemingly have no problem publishing bad science (yes Nature, I'm looking at you: http://retractionwatch.com/2014/09/11/potentially-groundbreaking-highly-provocative-nature-stap-cell-peer-reviews-published/ http://retractionwatch.com/2014/09/11/potentially-groundbrea...). Early in my PI's career, she tried to publish a very important negative fining in a high impact journal. The article's acceptance was accompanied by a personal letter from the editor urging her to consider other journals for negative results. Another barrier quite honestly is ego. While it may sound as if my boss is "one of the good ones", alas, she is not. On occasions that I have asked to repeat other group's seemingly unbelievable results myself, I've been flatly denied on grounds that this kind of work does not express the sort of originality of research produced by her lab. In other words, nobody wants to be known as "that lab", the nay-sayers of the field, those that would dare to question a colleague's ideas. Finally, this lead me to the last barrier I have observed: scientific communities / societies. If you are of the lucky few that end up publishing negative results of major significance, prepare to not be invited to dinner at next years Society for X annual meeting. Yes, in many ways life-science is stratified just like high school. You have the cool kids on track for the nobel, the weirdoes in their corner pushing the boundaries of what is possible, the "jocks"/ career scientists who manage to turn a couple of tricks and some charisma into a living, and finally the tattle-tales who seem to piss everyone off with their negative results. These are HUGE oversimplifications / generalizations, but I really think that all of these barriers need to be addressed in some way to fix life science.

eru 12y ago