6 ms·

Right at the end is a key thing: reward failure better. Publish negative results. I'm kinda embarrassed by my negative results, such as they are, but I'm clea

Right at the end is a key thing: reward failure better. Publish negative results. I'm kinda embarrassed by my negative results, such as they are, but I'm clear in this latest case that I wanted the method reviewed most, since others then may be able to use it. (Warning: not a real scientist, but being validated by real scientists in the case in question.)

tenaciousDaniel 9y ago
I'm not a scientist, but this is very surprising to me. I would have imagined that our scientific culture would be one where any result is recognized as something from which valuable knowledge can be gained. The idea that unanticipated results reflects some failure on the part of the researcher seems so strange and unscientific. But again, I'm not a scientist.
- allenz 9y ago
  In an ideal world, scientists would be judged solely on their study design and execution without considering the outcome. But that takes a lot of effort. It's much easier to judge based on the results that the scientist achieved ("I discovered the priming effect"). We remember Watson and Crick, and forget many other scientists who worked just as hard.
  - KGIII 9y ago
    Science doesn't really have a place for participation trophies. That and it's pretty natural to judge based on results. When handing out recognition and fame, the results are probably the best criteria - though that should include those who helped and that each stood on the shoulders of giants. That information should be included in the papers for those who are curious.
    - allenz 9y ago
      Negative results are real science, and science is collaborative. When we don't incentivize negative results, scientists don't take the time to analyze and share those findings, which results in a shallower understanding and possibly wasted effort by other scientists. It also biases scientists towards claiming results where none exist (e.g. p-hacking).
      - KGIII 9y ago
        The solution isn't publishing negative results. The solution is publishing better results. The solution is holding people to a higher standard and insisting on reproducibility before accepting the findings as factual. There are times when negative results pose new problems or offer new questions or methods. Those should be published but don't normally get you much praise. A blanket statement about publishing negative results is just going to get overwhelming information that nobody has time or interest in. Frankly, there are more productive things that occupy a researcher's time. If you want to know what a researcher has tried and failed, then chances are you're a professional in the domain. You should already know how to contact them and you can ask them to share their journals and logs with you. But, nobody is going to submit all their negative results. Nobody is going to index, catalogue, cross-reference, fund, and disseminate these massive negative results. If you want to know all the things I've tried and failed (I am a retired scientist), it would take multiple volumes and far more time than I'm willng to devote. Prior to retirement, I had positive results to publish, more things to try, and to find a way to make it commercially viable. Those acts are far more valuable than publishing things that got a negative result. Publishing those would take all my time and I'd never have gotten anything done. The system works. The system was designed as it was for a reason. Yes, some negative results should be published but they are exceedingly rare. The problem is a lack of effort to reproduce prior to acceptance and low-quality peer review. The solution isn't to publish more stuff, it's to publish less and to ensure the quality is there. The solution probably should also include open publishing. Instead of locking it up in expensive journals, make it more open so that more people, even layman, have the chance to review and replicate it. Speaking of which, I mention again that I lament the lack of emphasis, appreciation, and support for the citizen scientist. I'm a scientist. I'm nothing special. I'm wrong more times, by 09:00, than most people will be all day. Stop putting scientists on a pedestal or in ivory towers. You really don't want our many, many volumes of negative results. Just because it's got a Ph.D. tacked onto the end of the author's name doesn't mean it is valuable. Really, just uphold the standards that many seem to have forgotten. You're correct in that some negative results should be published, but they are few and far between. If anything, we have too much being published and a distinct lack of quality. I will add this... Journals only want a trimmed down paper. A journal may want you to trim your paper down to 10 pages, some want even shorter. Your research may be 1,000 to 2,000 pages but only a tiny bit gets published, or even submitted. That full research will have many of the negative results, as may the 'private' journal that the researcher worked from. If you want that information, you can probably get it emailed to you. The institution may have it in their archives, if the researcher is no longer available. So, negative results are out there, but most don't really need to be published. Most researchers don't have time, resources, or inclination to publish negative results with dubious value. Sometimes a negative result is potentially valuable, as it introduces something novel. Those get published with some frequency, though rating it as novel and anticipating its ability to further the arts is easier said than done. Even still, those rarities are even more rarely deserving of accolades. There are no participation trophies in science. That sort of thinking is kind of how we got into this mess. Somewhere along the lines, someone should have said, "No, that's garbage. Don't publish it under the heading of science or with the prestige of this journal/institution attached."
    - watwut 9y ago
      There was no relationship between variables is not participation trophy. It is a result and should count as one. As long as science is concerned with finding truth.
      - KGIII 9y ago
        If you want to see the negative results, contact the researcher. There is no room, publishing mechanism, or benefit to most negative results. If the negative results are novel or reveal a new method, they will get published. Science doesn't care that someone tried hard. Science isn't about applauding effort but is about results. Negative results with significance may, very seldom, be worthy of publishing. Publishing the vast majority of negative results would be a waste of time, funding, and resources. Maybe you're unaware of the many failed experiments across the domains, but they are plentiful. If you want to read them, the data is still collected - it's just not published. There's no good reason for it to be published in all but the rarest of cases. Just email the researcher, it will be in their journal. If they published, it's in the stack of notes and was trimmed out because it takes space, time, and money while having negligible value. It's great that someone put a lot of effort in, science does not care. Frankly, we are publishing too much junk as it is. Adding to the list of junk isn't going to solve any problems, it's just going to create problems. The solution is to increase the quality if what is being published, not to increase the noise to appease some emotions. Again, there is no participation trophy - nor should there be.
        tenaciousDaniel 9y ago
        What if there were an open platform where these results could be published? Not only published, but the data related to variables could be fed into a database. So if you wanted to check which variables have been found to not be correlated, you could query the database. It seems like that could be useful for scientists who want to avoid wasting their time.
- SapphireSun 9y ago
  Most ideas people have suck. That's sort of the problem of new ideas in particular. When you're working, their failures aren't recognized as some kind of thing you need to tell the world about, you just tried something speculative and it didn't work out. Writing papers sucks (though it can be ego satisfying if you did something really cool), and writing papers about negative results that no one will publish sucks even more (or give you money for). Also, if you do something really dumb, now everyone will know about it.
  - simplexion 9y ago
    Yet people still do studies on alternative "medicines" and other stupidity like "wind turbine sickness." That is literally doing something dumb that everyone will hear about.
    - vacri 9y ago
      Doing studies on those things you think are stupid is the right thing to do. Counter-intuitive stuff is found all the time in science. Occasionally those alternative medicines get tested and turn out to work, in which case they become regular medicine. By declaring something as "too dumb to study", you're a priori declaring the result, which is bad science.
  - SapphireSun 9y ago
    For what it's worth, I'm just describing the ground reality of research, not the utopian vision we should aspire to.
- godelski 9y ago
  Which is funny, because in science we narrow down answers by crossing ideas off a list. Not so much proving an answer as disproving others. But you don't get funding, especially in the softer fields, when you don't produce positive results. Unfortunately this also sows distrust in other fields.
- carbocation 9y ago
  Underpowered negative results are a class of potentially genuinely uninteresting results. Interestingly, underpowered positive results are likely to be published, but unreproducible. That is, the ratio of false positives will be higher among underpowered studies. So, the literature of underpowered studies will be enriched for false positives. So, underpowered results, whether positive or negative, are less interesting.
Asdfbla 9y ago
Pre-registration of studies is an interesting concept in that regard (also briefly mentioned in the article), where researchers can submit their study design and goals to a journal before conducting the study. It will then be accepted or rejected for publication based on the study design etc., not on the eventual outcome (as long as the study was actually conducted as promised and with care, of course). The idea is probably only applicable to a small subset of scientific work (not good for exploratory and very novel stuff, for instance), but in certain specific areas it might make publication of negative results a more viable path for scientists because they don't have to fear that their idea was bad or they seem incompetent because of a negative result. (Since their design was accepted, at least some other people thought it was a good idea too.)
- throwaway2048 9y ago
  The problem is the incentives don't change with a system like that, all that will happen is you have people rigging the pre-registers after they have already completed the study then sitting on results until the correct amount of time has passed.
  - wtallis 9y ago
    That's a very overt form of fraud, and it probably requires very different countermeasures than the kind of unintentional biases under discussion here.
  - deleted 9y ago
    [deleted]
kurthr 9y ago
Many have discussed this, but like writing bad/buggy code, there are just too many (combinatorial) ways to get a negative result, and most of them are frankly boring, distracting, or worse than useless. You would need a review paper of all the most common failures, screwups, and anti-patterns to be particularly useful... preferably with a good write-up of proper procedure. Some labs publish good protocols, but many (academics!) consider them to be secrets to used for advantage over other labs. Of course, based on the current p-hacking, publish at all costs environment, it's not clear you need to be particularly useful to be published. I like the idea that has been promoted of requiring a certain number of novel replications in a tenure application as one way forward... at least as a goal or professional shaming tool.
- pjschlic 9y ago
  This sounds like a great case for publishing negative results: it will result in meta-analysis about what practices routinely cause bad results and then those papers will be used to reduce the number of bad results. Labs keeping secrets in how to do research seems like a bug not a feature. That's just the externalities around publishing negative results. The major reason being that it will tell people where the minefields are, resulting in people not all reproducing the same negative results.
- admiralpumpkin 9y ago
  I agree, there are always more way to make mistakes, 2nd law and all, but the process of assessing good study design, good protocol, good statistical methods, etc. are all the same as for manuscripts with non-null results. Many of these issues will, in theory, have been sorted at the funding level. What holds us back, it seems to me, isn't process, it's culture and inertia.
- nzjrs 9y ago
  The code analogy is ok, but remember that the natural sciences are not engineering - a problem to be overcome. There is great value in publishing negative results in the form "I hypothesized X, did Y, and turned out X was wrong". It's my experience these negative results are often well controlled and trustworthy as there are few things that make one trust a method as much as holding onto it for dear life because you have spent so much time building it. Incidentally that's not always as true for positive results where the incentives are tilted towards stopping once you get ahead.
godelski 9y ago
I fully agree. Also, I think (as embarrassing as it is) we need raw data and code. Would make peer review much easier as you could perform the same analysis on the same data. Makes it easy to correlate results and validate premises of data sampling methods. (Warning: In Scientist-Engineer limbo)
Alex3917 9y ago
> Publish negative results. Or just don't publish the results at all, only publish the methodology and make the results available elsewhere along with the data and source code.
KGIII 9y ago
You don't need a degree to be a real scientist. You just need to follow the process. If anything, I think the discouragement of the citizen scientist is a bad thing. I think more people should try.
- DamonHD 9y ago
  FWIW I have a couple of degrees, but the topic I'm outlining is most definitely not rocket science. Indeed, it's rabout educing something people assume is hard and expensive to some meter readings and a bit of simple Excel jockeying or similar. That method is published and being validated (and we should get a shiny cert from a prestigious lab soon), along with the code and inconclusive results (not enough data, so uncertainty too high).
  - KGIII 9y ago
    Absolutely go for the gusto. Get your name attached to the paper, if you indeed earned it. It's even more awesome if you can get your name listed as the primary. It's rare, but sometimes people without any degrees get published in reputable journals. If I were in your shoes, I'd do what I can to get published in something reputable and buy a bunch of printed copies. I'd then autograph them and hand them out like a rock star. When I was first published in a journal, I brought a copy of the journal and a couple of 'official' single article prints for my mother. One of them she hung on the fridge and even stuck a gold star sticker on it. Of course, buying a copy of the whole journal and single article reprints was a lot less expensive then. I'd still get physical copies, just to show them off. Best of luck and feel free to yell, both when you've got it published and if you have any questions about formatting or want a proofreader. If it encourages you to publish, and encourages people like you, I'll do it. I really do wish there were more citizen scientists, or at least people who lack the standard credentials, submitting their ideas, being interested, and being willing to share their insights and research. Either way, best of luck! You've made my night. I love stories like that. uninvolved@outlook.com (generic public email address)
    - DamonHD 9y ago
      I think that you may be confusing me with someone clever and industrious! I do have my name on a pukka scientific paper from many moons ago, as the summer help for a PhD, where the real authors slung me in the author list out of kindness. But I do like the idea of a gold star. Then I can be 'award winning'! B^> That was molecular biology, this topic is home heating efficiency. That one is online for all to see, this one is too but is a bunch of very boring docs and part of a GitHub repo!

tenaciousDaniel 9y ago

I'm not a scientist, but this is very surprising to me. I would have imagined that our scientific culture would be one where any result is recognized as something from which valuable knowledge can be gained. The idea that unanticipated results reflects some failure on the part of the researcher seems so strange and unscientific. But again, I'm not a scientist.

allenz 9y ago
In an ideal world, scientists would be judged solely on their study design and execution without considering the outcome. But that takes a lot of effort. It's much easier to judge based on the results that the scientist achieved ("I discovered the priming effect"). We remember Watson and Crick, and forget many other scientists who worked just as hard.
- KGIII 9y ago
  Science doesn't really have a place for participation trophies. That and it's pretty natural to judge based on results. When handing out recognition and fame, the results are probably the best criteria - though that should include those who helped and that each stood on the shoulders of giants. That information should be included in the papers for those who are curious.
  - allenz 9y ago
    Negative results are real science, and science is collaborative. When we don't incentivize negative results, scientists don't take the time to analyze and share those findings, which results in a shallower understanding and possibly wasted effort by other scientists. It also biases scientists towards claiming results where none exist (e.g. p-hacking).
    - KGIII 9y ago
      The solution isn't publishing negative results. The solution is publishing better results. The solution is holding people to a higher standard and insisting on reproducibility before accepting the findings as factual. There are times when negative results pose new problems or offer new questions or methods. Those should be published but don't normally get you much praise. A blanket statement about publishing negative results is just going to get overwhelming information that nobody has time or interest in. Frankly, there are more productive things that occupy a researcher's time. If you want to know what a researcher has tried and failed, then chances are you're a professional in the domain. You should already know how to contact them and you can ask them to share their journals and logs with you. But, nobody is going to submit all their negative results. Nobody is going to index, catalogue, cross-reference, fund, and disseminate these massive negative results. If you want to know all the things I've tried and failed (I am a retired scientist), it would take multiple volumes and far more time than I'm willng to devote. Prior to retirement, I had positive results to publish, more things to try, and to find a way to make it commercially viable. Those acts are far more valuable than publishing things that got a negative result. Publishing those would take all my time and I'd never have gotten anything done. The system works. The system was designed as it was for a reason. Yes, some negative results should be published but they are exceedingly rare. The problem is a lack of effort to reproduce prior to acceptance and low-quality peer review. The solution isn't to publish more stuff, it's to publish less and to ensure the quality is there. The solution probably should also include open publishing. Instead of locking it up in expensive journals, make it more open so that more people, even layman, have the chance to review and replicate it. Speaking of which, I mention again that I lament the lack of emphasis, appreciation, and support for the citizen scientist. I'm a scientist. I'm nothing special. I'm wrong more times, by 09:00, than most people will be all day. Stop putting scientists on a pedestal or in ivory towers. You really don't want our many, many volumes of negative results. Just because it's got a Ph.D. tacked onto the end of the author's name doesn't mean it is valuable. Really, just uphold the standards that many seem to have forgotten. You're correct in that some negative results should be published, but they are few and far between. If anything, we have too much being published and a distinct lack of quality. I will add this... Journals only want a trimmed down paper. A journal may want you to trim your paper down to 10 pages, some want even shorter. Your research may be 1,000 to 2,000 pages but only a tiny bit gets published, or even submitted. That full research will have many of the negative results, as may the 'private' journal that the researcher worked from. If you want that information, you can probably get it emailed to you. The institution may have it in their archives, if the researcher is no longer available. So, negative results are out there, but most don't really need to be published. Most researchers don't have time, resources, or inclination to publish negative results with dubious value. Sometimes a negative result is potentially valuable, as it introduces something novel. Those get published with some frequency, though rating it as novel and anticipating its ability to further the arts is easier said than done. Even still, those rarities are even more rarely deserving of accolades. There are no participation trophies in science. That sort of thinking is kind of how we got into this mess. Somewhere along the lines, someone should have said, "No, that's garbage. Don't publish it under the heading of science or with the prestige of this journal/institution attached."
  - watwut 9y ago
    There was no relationship between variables is not participation trophy. It is a result and should count as one. As long as science is concerned with finding truth.
    - KGIII 9y ago
      If you want to see the negative results, contact the researcher. There is no room, publishing mechanism, or benefit to most negative results. If the negative results are novel or reveal a new method, they will get published. Science doesn't care that someone tried hard. Science isn't about applauding effort but is about results. Negative results with significance may, very seldom, be worthy of publishing. Publishing the vast majority of negative results would be a waste of time, funding, and resources. Maybe you're unaware of the many failed experiments across the domains, but they are plentiful. If you want to read them, the data is still collected - it's just not published. There's no good reason for it to be published in all but the rarest of cases. Just email the researcher, it will be in their journal. If they published, it's in the stack of notes and was trimmed out because it takes space, time, and money while having negligible value. It's great that someone put a lot of effort in, science does not care. Frankly, we are publishing too much junk as it is. Adding to the list of junk isn't going to solve any problems, it's just going to create problems. The solution is to increase the quality if what is being published, not to increase the noise to appease some emotions. Again, there is no participation trophy - nor should there be.
      - tenaciousDaniel 9y ago
        What if there were an open platform where these results could be published? Not only published, but the data related to variables could be fed into a database. So if you wanted to check which variables have been found to not be correlated, you could query the database. It seems like that could be useful for scientists who want to avoid wasting their time.
SapphireSun 9y ago
Most ideas people have suck. That's sort of the problem of new ideas in particular. When you're working, their failures aren't recognized as some kind of thing you need to tell the world about, you just tried something speculative and it didn't work out. Writing papers sucks (though it can be ego satisfying if you did something really cool), and writing papers about negative results that no one will publish sucks even more (or give you money for). Also, if you do something really dumb, now everyone will know about it.
- simplexion 9y ago
  Yet people still do studies on alternative "medicines" and other stupidity like "wind turbine sickness." That is literally doing something dumb that everyone will hear about.
  - vacri 9y ago
    Doing studies on those things you think are stupid is the right thing to do. Counter-intuitive stuff is found all the time in science. Occasionally those alternative medicines get tested and turn out to work, in which case they become regular medicine. By declaring something as "too dumb to study", you're a priori declaring the result, which is bad science.
- SapphireSun 9y ago
  For what it's worth, I'm just describing the ground reality of research, not the utopian vision we should aspire to.
godelski 9y ago
Which is funny, because in science we narrow down answers by crossing ideas off a list. Not so much proving an answer as disproving others. But you don't get funding, especially in the softer fields, when you don't produce positive results. Unfortunately this also sows distrust in other fields.
carbocation 9y ago
Underpowered negative results are a class of potentially genuinely uninteresting results. Interestingly, underpowered positive results are likely to be published, but unreproducible. That is, the ratio of false positives will be higher among underpowered studies. So, the literature of underpowered studies will be enriched for false positives. So, underpowered results, whether positive or negative, are less interesting.

Asdfbla 9y ago

Pre-registration of studies is an interesting concept in that regard (also briefly mentioned in the article), where researchers can submit their study design and goals to a journal before conducting the study. It will then be accepted or rejected for publication based on the study design etc., not on the eventual outcome (as long as the study was actually conducted as promised and with care, of course). The idea is probably only applicable to a small subset of scientific work (not good for exploratory and very novel stuff, for instance), but in certain specific areas it might make publication of negative results a more viable path for scientists because they don't have to fear that their idea was bad or they seem incompetent because of a negative result. (Since their design was accepted, at least some other people thought it was a good idea too.)

throwaway2048 9y ago
The problem is the incentives don't change with a system like that, all that will happen is you have people rigging the pre-registers after they have already completed the study then sitting on results until the correct amount of time has passed.
- wtallis 9y ago
  That's a very overt form of fraud, and it probably requires very different countermeasures than the kind of unintentional biases under discussion here.
- deleted 9y ago
  [deleted]

kurthr 9y ago

Many have discussed this, but like writing bad/buggy code, there are just too many (combinatorial) ways to get a negative result, and most of them are frankly boring, distracting, or worse than useless. You would need a review paper of all the most common failures, screwups, and anti-patterns to be particularly useful... preferably with a good write-up of proper procedure. Some labs publish good protocols, but many (academics!) consider them to be secrets to used for advantage over other labs. Of course, based on the current p-hacking, publish at all costs environment, it's not clear you need to be particularly useful to be published. I like the idea that has been promoted of requiring a certain number of novel replications in a tenure application as one way forward... at least as a goal or professional shaming tool.

pjschlic 9y ago
This sounds like a great case for publishing negative results: it will result in meta-analysis about what practices routinely cause bad results and then those papers will be used to reduce the number of bad results. Labs keeping secrets in how to do research seems like a bug not a feature. That's just the externalities around publishing negative results. The major reason being that it will tell people where the minefields are, resulting in people not all reproducing the same negative results.
admiralpumpkin 9y ago
I agree, there are always more way to make mistakes, 2nd law and all, but the process of assessing good study design, good protocol, good statistical methods, etc. are all the same as for manuscripts with non-null results. Many of these issues will, in theory, have been sorted at the funding level. What holds us back, it seems to me, isn't process, it's culture and inertia.
nzjrs 9y ago
The code analogy is ok, but remember that the natural sciences are not engineering - a problem to be overcome. There is great value in publishing negative results in the form "I hypothesized X, did Y, and turned out X was wrong". It's my experience these negative results are often well controlled and trustworthy as there are few things that make one trust a method as much as holding onto it for dear life because you have spent so much time building it. Incidentally that's not always as true for positive results where the incentives are tilted towards stopping once you get ahead.

godelski 9y ago

I fully agree. Also, I think (as embarrassing as it is) we need raw data and code. Would make peer review much easier as you could perform the same analysis on the same data. Makes it easy to correlate results and validate premises of data sampling methods. (Warning: In Scientist-Engineer limbo)

Alex3917 9y ago

> Publish negative results. Or just don't publish the results at all, only publish the methodology and make the results available elsewhere along with the data and source code.

KGIII 9y ago

You don't need a degree to be a real scientist. You just need to follow the process. If anything, I think the discouragement of the citizen scientist is a bad thing. I think more people should try.

DamonHD 9y ago
FWIW I have a couple of degrees, but the topic I'm outlining is most definitely not rocket science. Indeed, it's rabout educing something people assume is hard and expensive to some meter readings and a bit of simple Excel jockeying or similar. That method is published and being validated (and we should get a shiny cert from a prestigious lab soon), along with the code and inconclusive results (not enough data, so uncertainty too high).
- KGIII 9y ago
  Absolutely go for the gusto. Get your name attached to the paper, if you indeed earned it. It's even more awesome if you can get your name listed as the primary. It's rare, but sometimes people without any degrees get published in reputable journals. If I were in your shoes, I'd do what I can to get published in something reputable and buy a bunch of printed copies. I'd then autograph them and hand them out like a rock star. When I was first published in a journal, I brought a copy of the journal and a couple of 'official' single article prints for my mother. One of them she hung on the fridge and even stuck a gold star sticker on it. Of course, buying a copy of the whole journal and single article reprints was a lot less expensive then. I'd still get physical copies, just to show them off. Best of luck and feel free to yell, both when you've got it published and if you have any questions about formatting or want a proofreader. If it encourages you to publish, and encourages people like you, I'll do it. I really do wish there were more citizen scientists, or at least people who lack the standard credentials, submitting their ideas, being interested, and being willing to share their insights and research. Either way, best of luck! You've made my night. I love stories like that. uninvolved@outlook.com (generic public email address)
  - DamonHD 9y ago
    I think that you may be confusing me with someone clever and industrious! I do have my name on a pukka scientific paper from many moons ago, as the summer help for a PhD, where the real authors slung me in the author list out of kindness. But I do like the idea of a gold star. Then I can be 'award winning'! B^> That was molecular biology, this topic is home heating efficiency. That one is online for all to see, this one is too but is a bunch of very boring docs and part of a GitHub repo!