6 ms·

I have an idea: if a research study doesn't go the way you thought it would, put it out there. We need a central repository like Arxiv where we dump the experi

I have an idea: if a research study doesn't go the way you thought it would, put it out there. We need a central repository like Arxiv where we dump the experiments that didn't work out so that we can quickly compare a "successful" one to ones done before. That gives us a better idea of if the data is just a fluke. The papers wouldn't have to be super involved. What did you do? What were statistical conclusions. Give an upper-level undergraduate or an early masters student some experience writing up a procedure. Shouldn't take more than a couple hours but it could save a lot of time dealing with publication bias

jonlucc 10y ago
BioRxiv [1] attempts to be this, but it isn't widely used as it is in the physics/math/compSci world. [1] http://biorxiv.org/ http://biorxiv.org/
untilHellbanned 10y ago
Not going to happen. You don't understand the forces at play when biomedical researchers (I'm one too) do the work. A major reason more specific to this type of researcher that he/she wants to hold back the data is in the event it can be re-purposed for later goals. It's capital. The time and money spent acquiring the types of negative data the article talks about (mouse experiments) take big dollars. I'm not going to throw it away in an archive. It's like donating to a Goodwill bin some really expensive clothes you saved your money to buy only to realize after you bought it that it's the wrong size. You're totally going to try to make it work often for years before you give up on it. If you take issue with my comment and have ever worked for a startup that didn't pan out (should be many of us here on HN), think of the scenario if someone told you, "hey, why don't you just be a good person and open source your app and give all your customers to XYZ?" (Setting aside the armchair quarterback guilt imposed on you) There are a few who do that, but if you've spent any length of time on your business, you're going to think of ways to re-purpose the investment you made it in other ways before you just go dumping it in an archive like Github or whatever.
- alexryan 10y ago
  There seems to be a lot of sharing amongst artificial intelligence researchers nowadays and it seems to be accelerating innovation. Sharing of medical research could potential accelerate innovation that could save a lot of lives (including those of the researchers themselves and their loved ones). It's curious that these industries are so different in this respect. What would have to happen in order to increase the amount of sharing of medical research?
  - jandrewrogers 10y ago
    I would say this is more illusory than real. It is common in areas that have intense private R&D competition, such as AI, databases, et al to regularly publish research that makes you appear relevant without publishing any internal research that actually matters. The incentives are aligned to encourage this. It is unfortunate but I have seen it first-hand in a few interesting computer science research areas, and academia simply doesn't have the resources to keep pace with the output that happens under NDA at many private research labs.
- uptime 10y ago
  I think the notion that negative outcomes are capital is a good one. And if the research has public money involved then we have a right to see them so we are not spending dwindling public money on unnecessary redundancy (as opposed to productive or confirmatory kinds.) The second point about open sourcing tech that do not achieve its goal is also a good point. People who do that with great docs etc get a lot of respect. People who dump badly done work do not. Businesses that fail but care about their customers often "donate" them to a competitor. I think these ideas are quite valid and can and should be realized more often.
  - mattkrause 10y ago
    > People who do that with great docs etc get a lot of respect. That's the issue. Publishing run-of-the-mill negative results right now doesn't get you much of anything. I would love to live in a world where hiring committees and funding agencies say "Well, 3/5 of his experiments flopped, but we looked at the papers and they were all carefully thought out and well-run. Hire him/give him some cash!" Alas, I don't. The article even says, "We trust that reviewers and tenure committees will find appropriate ways to credit papers that include confirmation." The issue is that needs to come first for anything to happen.
chrisseaton 10y ago
How can an experiment 'not work out'? Do you mean a negative result? Not getting evidence for your hypothesis is not 'not working'. That's a crazy way to approach science. It is more information to adjust your hypothesis. Or do you mean a failure such as broken equipment or an infected sample meaning you have no data? Well then what would you put in the paper?
- nine_k 10y ago
  Assumptions proved wrong?
  - chrisseaton 10y ago
    I don't understand - experiments only work if they prove your assumptions? If your assumption was wrong then the experiment is working if it won't confirm it.
    - robotresearcher 10y ago
      Or your experiment was faulty. Or the effect was too small to show up in your experiment, or any of many things that cause an experiment to be inconclusive. Getting a negative result with high confidence is real work. Failing to get a positive result is not the same thing.
    - Fomite 10y ago
      "Experiment A is based on the assumption that Y is true. Halfway through doing A, it's quite clear that Y is at the very least not true. Thus, Experiment A doesn't work." Most people I know who have experiments not work out mean something like that - there's something not successful about the experiment itself, not just that they didn't get the answer they were hoping for.
- deleted 10y ago
  [deleted]
- julian_1 10y ago
  An experiment doesn't 'work out', if it doesn't lead to continued grant funding.
djsumdog 10y ago
Negative results are one of the biggest things missing from publications. I highly suggest the book "The Antidote" by Burkeman. There are several chapters on our current western view of failure and how our attempt to distance ourselves from it can hinder research, product development and even personal development. I think there was even an article posted here a few months back on how the lack of negative results in science really hurts the community. People may work on something for half a year, consider it a failure, and just dump it; and other people go down the same exact path with the same methods. (If you publish a negative result, someone may pick it up and ask, "I wonder if they tried x or y" and attempt the experiment again).
beambot 10y ago
> put it out there. To write a good, technical blog post takes 10+ hours. To write a complete academic paper with references, related work, methods, graphics, etc can take days, weeks, or even months (eg. for thesis work). It's not just that negative results largely go un-valued by the academic status quo... it's also often just not worth the effort to write-up beyond simple documentation in your lab notebook. As a researcher, the lasting impact that we care about is (simply put) in the positive results, and being the first to achieve them.
- kneel 10y ago
  >To write a complete academic paper with references, related work, graphics, etc can take days, weeks, or even months (eg. for thesis work). Doesn't have to be high quality, tell a convincing story and back it up with evidence proof. Just tell people what you tried and why it didn't work. There is a lot of redundant work in science because of pride.
  - beambot 10y ago
    Anecdotally, researchers generally won't do exhaustive (emphasis added) literature reviews from less reputable sources for negative results before attempting reproduction. At the very least, you'd need better organizational structures to make finding negative results easier -- something other than citations alone. Maybe this could be a section on each Nature paper with links to positive & negative reproduction notes?
    - kneel 10y ago
      Just the fact that it was attempted and noted would be great. A researcher could then attempt to contact the author and ask a couple questions about details. Currently, as it stands everyone is walking around in the dark.
  - tertius 10y ago
    > There is a lot of redundant work in science because of pride. Or they're hard working and cannot afford to waste time that won't lead them to success, i.e. more grant money.
mtdewcmu 10y ago
What if you basically take your lab notes and blog them?
arthur_pryor 10y ago
a friend of mine (who just finished a clinical psych PhD) had exactly this idea a couple years ago. a null results archive, so that experiments that don't "work out" don't just get thrown away.

jonlucc 10y ago

BioRxiv [1] attempts to be this, but it isn't widely used as it is in the physics/math/compSci world. [1] http://biorxiv.org/ http://biorxiv.org/

untilHellbanned 10y ago

Not going to happen. You don't understand the forces at play when biomedical researchers (I'm one too) do the work. A major reason more specific to this type of researcher that he/she wants to hold back the data is in the event it can be re-purposed for later goals. It's capital. The time and money spent acquiring the types of negative data the article talks about (mouse experiments) take big dollars. I'm not going to throw it away in an archive. It's like donating to a Goodwill bin some really expensive clothes you saved your money to buy only to realize after you bought it that it's the wrong size. You're totally going to try to make it work often for years before you give up on it. If you take issue with my comment and have ever worked for a startup that didn't pan out (should be many of us here on HN), think of the scenario if someone told you, "hey, why don't you just be a good person and open source your app and give all your customers to XYZ?" (Setting aside the armchair quarterback guilt imposed on you) There are a few who do that, but if you've spent any length of time on your business, you're going to think of ways to re-purpose the investment you made it in other ways before you just go dumping it in an archive like Github or whatever.

alexryan 10y ago
There seems to be a lot of sharing amongst artificial intelligence researchers nowadays and it seems to be accelerating innovation. Sharing of medical research could potential accelerate innovation that could save a lot of lives (including those of the researchers themselves and their loved ones). It's curious that these industries are so different in this respect. What would have to happen in order to increase the amount of sharing of medical research?
- jandrewrogers 10y ago
  I would say this is more illusory than real. It is common in areas that have intense private R&D competition, such as AI, databases, et al to regularly publish research that makes you appear relevant without publishing any internal research that actually matters. The incentives are aligned to encourage this. It is unfortunate but I have seen it first-hand in a few interesting computer science research areas, and academia simply doesn't have the resources to keep pace with the output that happens under NDA at many private research labs.
uptime 10y ago
I think the notion that negative outcomes are capital is a good one. And if the research has public money involved then we have a right to see them so we are not spending dwindling public money on unnecessary redundancy (as opposed to productive or confirmatory kinds.) The second point about open sourcing tech that do not achieve its goal is also a good point. People who do that with great docs etc get a lot of respect. People who dump badly done work do not. Businesses that fail but care about their customers often "donate" them to a competitor. I think these ideas are quite valid and can and should be realized more often.
- mattkrause 10y ago
  > People who do that with great docs etc get a lot of respect. That's the issue. Publishing run-of-the-mill negative results right now doesn't get you much of anything. I would love to live in a world where hiring committees and funding agencies say "Well, 3/5 of his experiments flopped, but we looked at the papers and they were all carefully thought out and well-run. Hire him/give him some cash!" Alas, I don't. The article even says, "We trust that reviewers and tenure committees will find appropriate ways to credit papers that include confirmation." The issue is that needs to come first for anything to happen.

chrisseaton 10y ago

How can an experiment 'not work out'? Do you mean a negative result? Not getting evidence for your hypothesis is not 'not working'. That's a crazy way to approach science. It is more information to adjust your hypothesis. Or do you mean a failure such as broken equipment or an infected sample meaning you have no data? Well then what would you put in the paper?

nine_k 10y ago
Assumptions proved wrong?
- chrisseaton 10y ago
  I don't understand - experiments only work if they prove your assumptions? If your assumption was wrong then the experiment is working if it won't confirm it.
  - robotresearcher 10y ago
    Or your experiment was faulty. Or the effect was too small to show up in your experiment, or any of many things that cause an experiment to be inconclusive. Getting a negative result with high confidence is real work. Failing to get a positive result is not the same thing.
  - Fomite 10y ago
    "Experiment A is based on the assumption that Y is true. Halfway through doing A, it's quite clear that Y is at the very least not true. Thus, Experiment A doesn't work." Most people I know who have experiments not work out mean something like that - there's something not successful about the experiment itself, not just that they didn't get the answer they were hoping for.
deleted 10y ago
[deleted]
julian_1 10y ago
An experiment doesn't 'work out', if it doesn't lead to continued grant funding.

djsumdog 10y ago

Negative results are one of the biggest things missing from publications. I highly suggest the book "The Antidote" by Burkeman. There are several chapters on our current western view of failure and how our attempt to distance ourselves from it can hinder research, product development and even personal development. I think there was even an article posted here a few months back on how the lack of negative results in science really hurts the community. People may work on something for half a year, consider it a failure, and just dump it; and other people go down the same exact path with the same methods. (If you publish a negative result, someone may pick it up and ask, "I wonder if they tried x or y" and attempt the experiment again).

beambot 10y ago

> put it out there. To write a good, technical blog post takes 10+ hours. To write a complete academic paper with references, related work, methods, graphics, etc can take days, weeks, or even months (eg. for thesis work). It's not just that negative results largely go un-valued by the academic status quo... it's also often just not worth the effort to write-up beyond simple documentation in your lab notebook. As a researcher, the lasting impact that we care about is (simply put) in the positive results, and being the first to achieve them.

kneel 10y ago
>To write a complete academic paper with references, related work, graphics, etc can take days, weeks, or even months (eg. for thesis work). Doesn't have to be high quality, tell a convincing story and back it up with evidence proof. Just tell people what you tried and why it didn't work. There is a lot of redundant work in science because of pride.
- beambot 10y ago
  Anecdotally, researchers generally won't do exhaustive (emphasis added) literature reviews from less reputable sources for negative results before attempting reproduction. At the very least, you'd need better organizational structures to make finding negative results easier -- something other than citations alone. Maybe this could be a section on each Nature paper with links to positive & negative reproduction notes?
  - kneel 10y ago
    Just the fact that it was attempted and noted would be great. A researcher could then attempt to contact the author and ask a couple questions about details. Currently, as it stands everyone is walking around in the dark.
- tertius 10y ago
  > There is a lot of redundant work in science because of pride. Or they're hard working and cannot afford to waste time that won't lead them to success, i.e. more grant money.

mtdewcmu 10y ago

What if you basically take your lab notes and blog them?

arthur_pryor 10y ago

a friend of mine (who just finished a clinical psych PhD) had exactly this idea a couple years ago. a null results archive, so that experiments that don't "work out" don't just get thrown away.