3 ms·

Making the publishing of negative data as prestigious and scientifically rigorous as positive data.

by kellogs 17y ago

maximilian 17y ago
Make the publishing of anything scientific independent from Journals and freely available to anyone.
- michael_dorfman 17y ago
  Isn't that Arxiv.org?
  - olalonde 17y ago
    Right, although Arxiv.org is so 1995.
  - natch 17y ago
    Do tenure committees and other such parts of the scientific establishment give publication on arxiv.org the same weight they give to publication in traditional scientific journals? They didn't used to -- has this changed?
    - kellogs 17y ago
      nope, it definitely hasn't.
tel 17y ago
This bothered me for a while because I'm not sure what you mean. I can take it three ways. 1. Negative means data that contradicts the popular hypothesis. This is frequently reported very prestigiously, though sometimes care has to be taken depending on just how deeply held the refuted belief is. 2. Negative means data that is highly inconclusive. This data is often non-reported since people continue experiments until they have a story or just drop inquiry. It should still (in some cases) be reported along with the final, telling data but probably isn't. 3. Negative means data that supports already "known" beliefs. Totally "uninteresting" results which just serve to give us the tiniest greater justification to believe what we do. Rarely reported. I can see (2) but mostly (3) being suitable targets, but you might have a different idea in mind for what negative data is. In the end it sounds like a need to make big advances in meta-analysis in order to have data reported against a more quantified context. Global, public, and automatic meta-analysis of the state of scientific belief is a hard problem I would love to see solved.
- kellogs 17y ago
  Oh, I meant #2, inconclusive results. I believe that this research is extremely valuable. Scientific contribution occurs not just by finding a story and some supporting data - another source of great contribution could be saving time of many researchers by saying "this is exactly what i did. this is why i did it. and look, nothing came up." another researcher can look at it and think about improving the methodology. if published, this search to get it right becomes a-lot-of-people mission instead of a one-phd-student venture to do all the mistakes that everyone else did, and then start doing things right. and to echo the above comments - arxiv is not prestigious. you wouldn't write a job market paper or pride yourself in terms of 10 publications in arxiv rather than 2 publications in Nature. i havent thought about the 'how', but i am convinced that the prejudice against negative data has to disappear for this whole thing to work.
  - tel 17y ago
    I totally agree with supporting (2), it's actually something I'm interested in as a research topic. Like I mentioned, I think part of the difficulty is the non-generality of being able to summarize that sort of material. There are (orders of magnitude) more inconclusive results than telling ones. Now the idea of being able to search a scientific question and somehow find a meta-analytical summary of all the inconclusive results ever reported would be flat magical. If the summaries were really useful it'd likely generate a large number of telling results almost automatically. Of course, a resource like that is almost mythical sounding for today.
    - kellogs 17y ago
      Oh, nice. Is this research in the pipeline or are you incubating the idea for the future? I am also thinking about this as a research topic - independent summer study, to be more specific. I am hoping to give the matter some serious thought - why negative data is not considered valuable, what sorts of conditions would make it valuable, and how the framework of publishing should be, so that others can easily access, understand, and reason about the data. Are you interested in a subset of these? Do you know any resources on where to start thinking? I know there's the Journal of Negative Results in Biomedicine, but I don't know of other efforts or perhaps societies/groups for this cause.

Making the publishing of negative data as prestigious and scientifically rigorous as positive data.

by kellogs 17y ago

Making the publishing of negative data as prestigious and scientifically rigorous as positive data.

maximilian 17y ago

Make the publishing of anything scientific independent from Journals and freely available to anyone.

michael_dorfman 17y ago
Isn't that Arxiv.org?
- olalonde 17y ago
  Right, although Arxiv.org is so 1995.
- natch 17y ago
  Do tenure committees and other such parts of the scientific establishment give publication on arxiv.org the same weight they give to publication in traditional scientific journals? They didn't used to -- has this changed?
  - kellogs 17y ago
    nope, it definitely hasn't.

tel 17y ago

This bothered me for a while because I'm not sure what you mean. I can take it three ways. 1. Negative means data that contradicts the popular hypothesis. This is frequently reported very prestigiously, though sometimes care has to be taken depending on just how deeply held the refuted belief is. 2. Negative means data that is highly inconclusive. This data is often non-reported since people continue experiments until they have a story or just drop inquiry. It should still (in some cases) be reported along with the final, telling data but probably isn't. 3. Negative means data that supports already "known" beliefs. Totally "uninteresting" results which just serve to give us the tiniest greater justification to believe what we do. Rarely reported. I can see (2) but mostly (3) being suitable targets, but you might have a different idea in mind for what negative data is. In the end it sounds like a need to make big advances in meta-analysis in order to have data reported against a more quantified context. Global, public, and automatic meta-analysis of the state of scientific belief is a hard problem I would love to see solved.

kellogs 17y ago
Oh, I meant #2, inconclusive results. I believe that this research is extremely valuable. Scientific contribution occurs not just by finding a story and some supporting data - another source of great contribution could be saving time of many researchers by saying "this is exactly what i did. this is why i did it. and look, nothing came up." another researcher can look at it and think about improving the methodology. if published, this search to get it right becomes a-lot-of-people mission instead of a one-phd-student venture to do all the mistakes that everyone else did, and then start doing things right. and to echo the above comments - arxiv is not prestigious. you wouldn't write a job market paper or pride yourself in terms of 10 publications in arxiv rather than 2 publications in Nature. i havent thought about the 'how', but i am convinced that the prejudice against negative data has to disappear for this whole thing to work.
- tel 17y ago
  I totally agree with supporting (2), it's actually something I'm interested in as a research topic. Like I mentioned, I think part of the difficulty is the non-generality of being able to summarize that sort of material. There are (orders of magnitude) more inconclusive results than telling ones. Now the idea of being able to search a scientific question and somehow find a meta-analytical summary of all the inconclusive results ever reported would be flat magical. If the summaries were really useful it'd likely generate a large number of telling results almost automatically. Of course, a resource like that is almost mythical sounding for today.
  - kellogs 17y ago
    Oh, nice. Is this research in the pipeline or are you incubating the idea for the future? I am also thinking about this as a research topic - independent summer study, to be more specific. I am hoping to give the matter some serious thought - why negative data is not considered valuable, what sorts of conditions would make it valuable, and how the framework of publishing should be, so that others can easily access, understand, and reason about the data. Are you interested in a subset of these? Do you know any resources on where to start thinking? I know there's the Journal of Negative Results in Biomedicine, but I don't know of other efforts or perhaps societies/groups for this cause.