5 ms·

> put it out there. To write a good, technical blog post takes 10+ hours. To write a complete academic paper with references, related work, methods, graphics,

by beambot 10y ago

> put it out there. To write a good, technical blog post takes 10+ hours. To write a complete academic paper with references, related work, methods, graphics, etc can take days, weeks, or even months (eg. for thesis work). It's not just that negative results largely go un-valued by the academic status quo... it's also often just not worth the effort to write-up beyond simple documentation in your lab notebook. As a researcher, the lasting impact that we care about is (simply put) in the positive results, and being the first to achieve them.

kneel 10y ago
>To write a complete academic paper with references, related work, graphics, etc can take days, weeks, or even months (eg. for thesis work). Doesn't have to be high quality, tell a convincing story and back it up with evidence proof. Just tell people what you tried and why it didn't work. There is a lot of redundant work in science because of pride.
- beambot 10y ago
  Anecdotally, researchers generally won't do exhaustive (emphasis added) literature reviews from less reputable sources for negative results before attempting reproduction. At the very least, you'd need better organizational structures to make finding negative results easier -- something other than citations alone. Maybe this could be a section on each Nature paper with links to positive & negative reproduction notes?
  - kneel 10y ago
    Just the fact that it was attempted and noted would be great. A researcher could then attempt to contact the author and ask a couple questions about details. Currently, as it stands everyone is walking around in the dark.
- tertius 10y ago
  > There is a lot of redundant work in science because of pride. Or they're hard working and cannot afford to waste time that won't lead them to success, i.e. more grant money.
  - kneel 10y ago
    Incentives in science are skewed. The focus shouldn't be on positive results, the focus should be on publishing results. A correctly incentivized researcher would strive to publish, and strive to improve knowledge in their field.
    - SolaceQuantum 10y ago
      What prevents scientists from publishing several failures and arbitrarily jumping ship off any experiments because it's faster to fail than it is to succeed?
      - kneel 10y ago
        If you never get a positive result then you're probably just not good at science. Also I can't imagine any scientists who gets up and tries to fail everyday. However, they're still advancing knowledge. Just because your experiment doesn't work doesn't mean it wasn't worthwhile. If you publish your failure then future researchers don't have to waste their time walking down the same path.
        tertius 10y ago
        > If you never get a positive result then you're probably just not good at science. Also I can't imagine any scientists who gets up and tries to fail everyday. Science is this: Try to disprove (i.e. fail) your hypothesis. If you've failed sufficiently, then you have something that can be published. > Just because your experiment doesn't work doesn't mean it wasn't worthwhile. If experiments don't work then you need to redo them to figure out why they didn't work. An experiment with a negative result is not an experiment that didn't work. It worked, but gave you an answer you did not expect. Verify. Every scientist I know (a lot) and have interacted with understand that a negative result is not a worthwhile result. > If you publish your failure then future researchers don't have to waste their time walking down the same path. Scientists often publish negative results. This is mostly done if it disproves some other published work that you do not believe, through some experience of your own, and want to test with a different test or a more rigorous test. But if it does not challenge something that's accepted, it's unlikely to get acknowledged and is therefore of low value to the lab and to the scientist that is publishing. They cannot afford to put that above their own career success. And if they did they would find it harder and harder to get grant money and therefore will fail in the academic field completely. You cannot (except for extreme edge cases) contribute to science without money.
mtdewcmu 10y ago
You could publish notes in rough form and clean them up on demand.
- rspeer 10y ago
  Publishing poorly-written crap will make you look worse than publishing nothing at all. There's already a terrible stigma around negative results. Grant committees and tenure committees would subconsciously like to believe that you are not a scientist who need to test hypotheses, but a clairvoyant superhero who only has correct hypotheses. Yes, this is ridiculous and anti-scientific.
  - mtdewcmu 10y ago
    Everyone would have to agree to publish their rough notes (on the web). If this was the norm, then no one would be singled out for criticism. Of course, norms would have to change.
    - rspeer 10y ago
      > Of course, norms would have to change. Sure. How? Changing the norms of academia will take a generation, and would require some actual incentive to change them in the correct direction.
      - mtdewcmu 10y ago
        The article is calling for demanding more effort to get a paper published. That would be a change of norms. There's no real way to solve the reproducibility problem without changing norms. Publishing notes seems easiest. It would be like "open source" research.
  - arthur_pryor 10y ago
    > There's already a terrible stigma around negative results. Grant committees and tenure committees would subconsciously like to believe that you are not a scientist who need to test hypotheses, but a clairvoyant superhero who only has correct hypotheses. Yes, this is ridiculous and anti-scientific. this is exactly why an archive of these sorts of results is necessary. so that we can see that this is part of the process, de-stigmatize it, and use those findings to make more progress and design better experiments to gain further insight into the world. replication and null results are an important part of the scientific process and building up our overall store of knowledge. it can't all be sexy groundbreaking proofs of new hypotheses.
kriro 10y ago
Just publishing the raw data and the methods section would be enough imo. Alas trying to get scientists to publish the data sets for their research is surprisingly hard for various reasons.

by beambot 10y ago

kneel 10y ago

>To write a complete academic paper with references, related work, graphics, etc can take days, weeks, or even months (eg. for thesis work). Doesn't have to be high quality, tell a convincing story and back it up with evidence proof. Just tell people what you tried and why it didn't work. There is a lot of redundant work in science because of pride.

beambot 10y ago
Anecdotally, researchers generally won't do exhaustive (emphasis added) literature reviews from less reputable sources for negative results before attempting reproduction. At the very least, you'd need better organizational structures to make finding negative results easier -- something other than citations alone. Maybe this could be a section on each Nature paper with links to positive & negative reproduction notes?
- kneel 10y ago
  Just the fact that it was attempted and noted would be great. A researcher could then attempt to contact the author and ask a couple questions about details. Currently, as it stands everyone is walking around in the dark.
tertius 10y ago
> There is a lot of redundant work in science because of pride. Or they're hard working and cannot afford to waste time that won't lead them to success, i.e. more grant money.
- kneel 10y ago
  Incentives in science are skewed. The focus shouldn't be on positive results, the focus should be on publishing results. A correctly incentivized researcher would strive to publish, and strive to improve knowledge in their field.
  - SolaceQuantum 10y ago
    What prevents scientists from publishing several failures and arbitrarily jumping ship off any experiments because it's faster to fail than it is to succeed?
    - kneel 10y ago
      If you never get a positive result then you're probably just not good at science. Also I can't imagine any scientists who gets up and tries to fail everyday. However, they're still advancing knowledge. Just because your experiment doesn't work doesn't mean it wasn't worthwhile. If you publish your failure then future researchers don't have to waste their time walking down the same path.
      - tertius 10y ago
        > If you never get a positive result then you're probably just not good at science. Also I can't imagine any scientists who gets up and tries to fail everyday. Science is this: Try to disprove (i.e. fail) your hypothesis. If you've failed sufficiently, then you have something that can be published. > Just because your experiment doesn't work doesn't mean it wasn't worthwhile. If experiments don't work then you need to redo them to figure out why they didn't work. An experiment with a negative result is not an experiment that didn't work. It worked, but gave you an answer you did not expect. Verify. Every scientist I know (a lot) and have interacted with understand that a negative result is not a worthwhile result. > If you publish your failure then future researchers don't have to waste their time walking down the same path. Scientists often publish negative results. This is mostly done if it disproves some other published work that you do not believe, through some experience of your own, and want to test with a different test or a more rigorous test. But if it does not challenge something that's accepted, it's unlikely to get acknowledged and is therefore of low value to the lab and to the scientist that is publishing. They cannot afford to put that above their own career success. And if they did they would find it harder and harder to get grant money and therefore will fail in the academic field completely. You cannot (except for extreme edge cases) contribute to science without money.

mtdewcmu 10y ago

You could publish notes in rough form and clean them up on demand.

rspeer 10y ago
Publishing poorly-written crap will make you look worse than publishing nothing at all. There's already a terrible stigma around negative results. Grant committees and tenure committees would subconsciously like to believe that you are not a scientist who need to test hypotheses, but a clairvoyant superhero who only has correct hypotheses. Yes, this is ridiculous and anti-scientific.
- mtdewcmu 10y ago
  Everyone would have to agree to publish their rough notes (on the web). If this was the norm, then no one would be singled out for criticism. Of course, norms would have to change.
  - rspeer 10y ago
    > Of course, norms would have to change. Sure. How? Changing the norms of academia will take a generation, and would require some actual incentive to change them in the correct direction.
    - mtdewcmu 10y ago
      The article is calling for demanding more effort to get a paper published. That would be a change of norms. There's no real way to solve the reproducibility problem without changing norms. Publishing notes seems easiest. It would be like "open source" research.
- arthur_pryor 10y ago
  > There's already a terrible stigma around negative results. Grant committees and tenure committees would subconsciously like to believe that you are not a scientist who need to test hypotheses, but a clairvoyant superhero who only has correct hypotheses. Yes, this is ridiculous and anti-scientific. this is exactly why an archive of these sorts of results is necessary. so that we can see that this is part of the process, de-stigmatize it, and use those findings to make more progress and design better experiments to gain further insight into the world. replication and null results are an important part of the scientific process and building up our overall store of knowledge. it can't all be sexy groundbreaking proofs of new hypotheses.

kriro 10y ago

Just publishing the raw data and the methods section would be enough imo. Alas trying to get scientists to publish the data sets for their research is surprisingly hard for various reasons.