7 ms·

It's an interesting point, but it's a tough one. Real example: I'm a social scientist, and I have a hypothesis about a better method of analysing discourse (bo

It's an interesting point, but it's a tough one. Real example: I'm a social scientist, and I have a hypothesis about a better method of analysing discourse (borne from theoretical musings, mostly). Once I spend the next 6 months testing that empirically, and it turns out to be nonsense or completely unfounded, I'm not sure how much value there is in putting it out there - a lot of social science hypotheses are (imo) quite ephemeral and move on quite quickly. Part of that's a spillover from the fact that it's difficult to scale qualitative data gathering and analysis as an (often individual) researcher, and a frequent assumption that only quantitative data 'counts', but yeah. Maybe there should be a way to summarise null findings, to stop people replicating unsuccessful or unfounded work.

onion2k 12y ago
I'm not sure how much value there is in putting it out there There's a huge amount of value in sharing 'failed' experimental data and analysis (so long as the experimental methodology was sound). People can learn new things from meta-analysis of failures and correlating disproved theories against proven ones. In fact, there are journals dedicated to it; http://www.jnrbm.com/ http://www.jnrbm.com/ and http://www.arjournals.com/ojs/ http://www.arjournals.com/ojs/ are examples.
- sp332 12y ago
  Exactly! A negative result is not a failed experiment, it's a successful experiment. Think of all the 0's and 1's in your computer files - now imagine if someone just deleted all the 0's because they were boring. Negative results have just as much information as positive results, if you did the experiment right.
  - JoeAltmaier 12y ago
    Not quite. As information they are much, much weaker. There are an almost unlimited number of negative results, and only a few positive ones. Imagine you are tuning a radio with a dial. Nobody wants to hear 'static at 100.3MHz! Static at 100.31MHz! Static at 100.32MHz' They'd rather hear 'Signal at 103.5MHz!'
    - sp332 12y ago
      How would you know there is no signal at 100.3MHz if you didn't look? Edit: A standard FM radio filter is +/- 90kHz around the tuned-in frequency, so if you choose 100.3 on your dial, you're really getting everything from 100.21 to 100.39 all at once. And 100.5 is really 100.41 to 100.59, so we're really only skipping 20% of the frequencies.
    - lutusp 12y ago
      > Not quite. As information they [negative results] are much, much weaker. No, that is false, and further, it distorts entire fields. In psychology, for example, phony mental illnesses like Asperger's become established as real things only because of all the positive results, not accompanied by an equal number of negative ones that were never published. And in drug therapies, many drugs are validated and sold only because there are only positive published results. In one correction to this trend, the FDA funded a study in which all studies of a given class of drugs -- published and unpublished -- were analyzed in a meta-analysis. The result was that the statistical results fell below clinical significance for the majority of subjects: http://www.plosmedicine.org/article/info%3Adoi%2F10.1371%2Fjournal.pmed.0050045 http://www.plosmedicine.org/article/info%3Adoi%2F10.1371%2Fj... "Drug–placebo differences increased as a function of initial severity, rising from virtually no difference at moderate levels of initial depression to a relatively small difference for patients with very severe depression, reaching conventional criteria for clinical significance only for patients at the upper end of the very severely depressed category."
      - JoeAltmaier 12y ago
        Of course negative results are valuable. But as signal-to-noise there have to be many, many more of them to help define what is there by showing what isn't. They have individually less weight than a positive value, is the argument.
        lutusp 12y ago
        > They have individually less weight than a positive value, is the argument. Yes, and that argument is quite false. The history of science shows that positive and negative results have equal weight. But for that to remain true in modern times, the negative results must be published.
lutusp 12y ago
> Maybe there should be a way to summarise null findings, to stop people replicating unsuccessful or unfounded work. But repeating failed experiments has the same scientific value as repeating successful ones -- it solidifies the result, increase our certainly about it. In real science, null results, and their replication, are extremely important. My favorite example is the Michelson-Morley experiment that expected to find differences in the speed of light due to the motion of the earth through the hypothetical ether. The failure of this experiment, its publication, and efforts to replicate the result, also dutifully published, finally created a revolution in physics when Einstein explained why there was a null result, and coincidentally gave birth to the modern world.
- iamwithnail 12y ago
  'Real science' - thanks for that. ;-) I think the issue I'm trying to get at - I suspect there is less value in null proving a lot of social science hypotheses - the impacts are smaller than physical constants of the universe (or finding say, that the Higgs Boson didn't exist) - if I prove, because my empirical work says so, that discourse can be meaningfully (for a given value of meaningful...) analysed in a new way, that's great, and will advance some other work. But proving (even repeatedly) that it can't be won't really change anything, imo. I guess though, the main issue is: I'm sure as hell not going to get published saying it, which ties back nicely to the point made in Nature, really.
  - lutusp 12y ago
    > I suspect there is less value in null proving a lot of social science hypotheses Yes, but that applies equally to proving positive social science results. My point wasn't to compare social science to other sciences, it was only to point out the value of replicating null results in general -- that it stands equal to replicating positive results. > if I prove, because my empirical work says so, that discourse can be meaningfully (for a given value of meaningful...) analysed in a new way, that's great, and will advance some other work. But proving (even repeatedly) that it can't be won't really change anything, imo. Both findings have equal weight, therefore both findings deserve equal attention from replicators. By contrast, ignoring null results is a disaster for whatever science they occur in -- it may mean that other people might spend a lot of time spinning their wheels studying things that have already been found wanting.
dspillett 12y ago
> I'm not sure how much value there is in putting it out there I'm sure there is great value in putting it out there if people find it later. The problem is two fold: firstly where do you put it (journals with limited space and limited time for reviewing content aren't going to be all that interested, and people aren't going to trust some random page on the Internet), and secondly will people even look for it before starting their work? This affects many scientific area and not just this one - just think how many instances there have been of medical experiments going ahead based on the results of a study or two despite the fact that a bunch of other studies contradicted the findings? Those studies either were not where they could be easily found, were not searched for, or were not published at all at the time. This is particularly problematic when there is a conflict of interests due to studies being commissioned by those selling the products/processes/ideas they hope the studies will support: there is a commercial drive to publish every success and down play every less positive result.
- harvestmoon 12y ago
  Maybe this type of issue would be ideal for a highly curated, online database. At the moment, for instance, I've done a lot of research into biomedical topics, and get pretty much all my info through online pdfs linked to from PubMed. (Though, of course, most of those have been published in a reputable journal.) Still, if the content was both checked for being reasonable and also classified well, then this type of online database could possibly provide a lot of value to researchers.
p4bl0 12y ago
> I'm not sure how much value there is in putting it out there. A lot. Failed experiments can be the basis for an improved one attempting to fix what made it go wrong the first time, even if "only" for a master thesis study. It can also be a real asset for anyone writing a paper on a similar subject with working results since they can cite the failed experiment to quickly justify why they didn't test some variations of their hypothesis etc. In a world where paper are often limited in number of pages, being available to list failed attempts with citations is awesome compared to having to write a small paragraph with approximative and most often bad explanations for each failures. A first step might be to put such (non-)results in short papers and upload them to an open access repository in the kind of arXiv or HAL for instance.
harvestmoon 12y ago
Assuming that your idea is reasonable, then there could be a lot of value. What if I am a researcher who has a similar idea and is saved 6 months testing because of your work? Or what if I read it and see that some of it is valuable and look into a related area. Of course, if there is a tremendous amount of research published on, say, how to make a perpetual motion device, I can see how that could waste people's time.

onion2k 12y ago

I'm not sure how much value there is in putting it out there There's a huge amount of value in sharing 'failed' experimental data and analysis (so long as the experimental methodology was sound). People can learn new things from meta-analysis of failures and correlating disproved theories against proven ones. In fact, there are journals dedicated to it; http://www.jnrbm.com/ http://www.jnrbm.com/ and http://www.arjournals.com/ojs/ http://www.arjournals.com/ojs/ are examples.

sp332 12y ago
Exactly! A negative result is not a failed experiment, it's a successful experiment. Think of all the 0's and 1's in your computer files - now imagine if someone just deleted all the 0's because they were boring. Negative results have just as much information as positive results, if you did the experiment right.
- JoeAltmaier 12y ago
  Not quite. As information they are much, much weaker. There are an almost unlimited number of negative results, and only a few positive ones. Imagine you are tuning a radio with a dial. Nobody wants to hear 'static at 100.3MHz! Static at 100.31MHz! Static at 100.32MHz' They'd rather hear 'Signal at 103.5MHz!'
  - sp332 12y ago
    How would you know there is no signal at 100.3MHz if you didn't look? Edit: A standard FM radio filter is +/- 90kHz around the tuned-in frequency, so if you choose 100.3 on your dial, you're really getting everything from 100.21 to 100.39 all at once. And 100.5 is really 100.41 to 100.59, so we're really only skipping 20% of the frequencies.
  - lutusp 12y ago
    > Not quite. As information they [negative results] are much, much weaker. No, that is false, and further, it distorts entire fields. In psychology, for example, phony mental illnesses like Asperger's become established as real things only because of all the positive results, not accompanied by an equal number of negative ones that were never published. And in drug therapies, many drugs are validated and sold only because there are only positive published results. In one correction to this trend, the FDA funded a study in which all studies of a given class of drugs -- published and unpublished -- were analyzed in a meta-analysis. The result was that the statistical results fell below clinical significance for the majority of subjects: http://www.plosmedicine.org/article/info%3Adoi%2F10.1371%2Fjournal.pmed.0050045 http://www.plosmedicine.org/article/info%3Adoi%2F10.1371%2Fj... "Drug–placebo differences increased as a function of initial severity, rising from virtually no difference at moderate levels of initial depression to a relatively small difference for patients with very severe depression, reaching conventional criteria for clinical significance only for patients at the upper end of the very severely depressed category."
    - JoeAltmaier 12y ago
      Of course negative results are valuable. But as signal-to-noise there have to be many, many more of them to help define what is there by showing what isn't. They have individually less weight than a positive value, is the argument.
      - lutusp 12y ago
        > They have individually less weight than a positive value, is the argument. Yes, and that argument is quite false. The history of science shows that positive and negative results have equal weight. But for that to remain true in modern times, the negative results must be published.

lutusp 12y ago

> Maybe there should be a way to summarise null findings, to stop people replicating unsuccessful or unfounded work. But repeating failed experiments has the same scientific value as repeating successful ones -- it solidifies the result, increase our certainly about it. In real science, null results, and their replication, are extremely important. My favorite example is the Michelson-Morley experiment that expected to find differences in the speed of light due to the motion of the earth through the hypothetical ether. The failure of this experiment, its publication, and efforts to replicate the result, also dutifully published, finally created a revolution in physics when Einstein explained why there was a null result, and coincidentally gave birth to the modern world.

iamwithnail 12y ago
'Real science' - thanks for that. ;-) I think the issue I'm trying to get at - I suspect there is less value in null proving a lot of social science hypotheses - the impacts are smaller than physical constants of the universe (or finding say, that the Higgs Boson didn't exist) - if I prove, because my empirical work says so, that discourse can be meaningfully (for a given value of meaningful...) analysed in a new way, that's great, and will advance some other work. But proving (even repeatedly) that it can't be won't really change anything, imo. I guess though, the main issue is: I'm sure as hell not going to get published saying it, which ties back nicely to the point made in Nature, really.
- lutusp 12y ago
  > I suspect there is less value in null proving a lot of social science hypotheses Yes, but that applies equally to proving positive social science results. My point wasn't to compare social science to other sciences, it was only to point out the value of replicating null results in general -- that it stands equal to replicating positive results. > if I prove, because my empirical work says so, that discourse can be meaningfully (for a given value of meaningful...) analysed in a new way, that's great, and will advance some other work. But proving (even repeatedly) that it can't be won't really change anything, imo. Both findings have equal weight, therefore both findings deserve equal attention from replicators. By contrast, ignoring null results is a disaster for whatever science they occur in -- it may mean that other people might spend a lot of time spinning their wheels studying things that have already been found wanting.

dspillett 12y ago

> I'm not sure how much value there is in putting it out there I'm sure there is great value in putting it out there if people find it later. The problem is two fold: firstly where do you put it (journals with limited space and limited time for reviewing content aren't going to be all that interested, and people aren't going to trust some random page on the Internet), and secondly will people even look for it before starting their work? This affects many scientific area and not just this one - just think how many instances there have been of medical experiments going ahead based on the results of a study or two despite the fact that a bunch of other studies contradicted the findings? Those studies either were not where they could be easily found, were not searched for, or were not published at all at the time. This is particularly problematic when there is a conflict of interests due to studies being commissioned by those selling the products/processes/ideas they hope the studies will support: there is a commercial drive to publish every success and down play every less positive result.

harvestmoon 12y ago
Maybe this type of issue would be ideal for a highly curated, online database. At the moment, for instance, I've done a lot of research into biomedical topics, and get pretty much all my info through online pdfs linked to from PubMed. (Though, of course, most of those have been published in a reputable journal.) Still, if the content was both checked for being reasonable and also classified well, then this type of online database could possibly provide a lot of value to researchers.

p4bl0 12y ago

> I'm not sure how much value there is in putting it out there. A lot. Failed experiments can be the basis for an improved one attempting to fix what made it go wrong the first time, even if "only" for a master thesis study. It can also be a real asset for anyone writing a paper on a similar subject with working results since they can cite the failed experiment to quickly justify why they didn't test some variations of their hypothesis etc. In a world where paper are often limited in number of pages, being available to list failed attempts with citations is awesome compared to having to write a small paragraph with approximative and most often bad explanations for each failures. A first step might be to put such (non-)results in short papers and upload them to an open access repository in the kind of arXiv or HAL for instance.

harvestmoon 12y ago

Assuming that your idea is reasonable, then there could be a lot of value. What if I am a researcher who has a similar idea and is saved 6 months testing because of your work? Or what if I read it and see that some of it is valuable and look into a related area. Of course, if there is a tremendous amount of research published on, say, how to make a perpetual motion device, I can see how that could waste people's time.