5 ms·

This is a pipe dream of mine, but I would love a Wikipedia of null results: nullpedia! Nobody publishes null results bc they’re not exciting, but I think a lot

This is a pipe dream of mine, but I would love a Wikipedia of null results: nullpedia! Nobody publishes null results bc they’re not exciting, but I think a lot of NSF money would be saved if “failed” experiments were aggregated somewhere in a searchable way. There’s lot of questions. How to organize it? How to encourage participation? How to maximize usefulness while at the same time minimizing volunteer effort? How to encourage discussion (suggesting changes for a better exp design) rather than manipulation (stealing the seed of a bad experiment to publish at your better funded lab)? I don’t know how to do it, but I think if done right people would really like it.

mariushn 7y ago
Good idea, and great challenges indeed. Taking this further, I'd call it experimenpedia, with experiments published as they are thought out. Be open for comments/review and showing related experiments before the actual experiment being done. This might prevent potential failures and let owners tweak the planned experiment before being actually done. Then do the experiment and publish the results, whatever they are. I guess secrecy would actually win though, and nobody would use this?
- biophysboy 7y ago
  eLife is sorta like this. The experiment isn’t live, but the publishing process is. https://elifesciences.org/about/peer-review https://elifesciences.org/about/peer-review It’s pretty interesting
ken 7y ago
Snopes and Mythbusters showed how you can actually make entertainment out of "This thing we thought might be true, actually isn't true / doesn't work at all". As for the fairness issue, my first thought is to have it moderated by scientists but from different fields. It's like we do in other areas, like law. You need an expert on the general process, but if they have any personal connection to the topic at hand, they must recuse themselves. In a sense, it's just a specific type of scientific journal, right? An online journal of only null results.
TeMPOraL 7y ago
Great idea, would love to see it. Related, there were journals like this at some point, e.g. International Journal of Negative & Null Results, or ournal of Articles in Support of the Null Hypothesis, and I think at least one more. Not sure how they fare, though.
godelski 7y ago
To do this, it somehow needs to gain the reputation of a publication but also not encourage people doing just frivolous research (at least not on their own dime). But I think this kind of thing would be INSANELY useful. Especially if data was attached. (This could possibly help with the above problem because reputations develop).
bilifuduo 7y ago
The Series of Unsurprising Results (SURE) Journal is starting to do this, but only for economics papers: https://blogs.canterbury.ac.nz/surejournal/ https://blogs.canterbury.ac.nz/surejournal/ Vox recently ran a good profile on them and the importance of null results in general: https://www.vox.com/future-perfect/2019/5/17/18624812/publication-bias-economics-journal https://www.vox.com/future-perfect/2019/5/17/18624812/public...
fghtr 7y ago
I am not sure this would be as useful as the new method of "pre-registered studies" [0], where the null findings appear automatically in the publications. Null findings are not the goal of this approach, but a side effect. Important are the initial hipotheses, which pass peer-review. [0] https://www.nature.com/articles/d41586-018-07118-1 https://www.nature.com/articles/d41586-018-07118-1 via https://news.ycombinator.com/item?id=18297724 https://news.ycombinator.com/item?id=18297724
grumdan 7y ago
In a similar vein, I'd love to see some form of reward for replication studies, which are also hard to publish, since they're not novel and exciting. However, due to this focus on novelty, we got plenty of unreproducible results: https://en.wikipedia.org/wiki/Replication_crisis https://en.wikipedia.org/wiki/Replication_crisis In my ideal world, a scientific result would not be taken as meaningful until it's been replicated at least once.

mariushn 7y ago

Good idea, and great challenges indeed. Taking this further, I'd call it experimenpedia, with experiments published as they are thought out. Be open for comments/review and showing related experiments before the actual experiment being done. This might prevent potential failures and let owners tweak the planned experiment before being actually done. Then do the experiment and publish the results, whatever they are. I guess secrecy would actually win though, and nobody would use this?

biophysboy 7y ago
eLife is sorta like this. The experiment isn’t live, but the publishing process is. https://elifesciences.org/about/peer-review https://elifesciences.org/about/peer-review It’s pretty interesting

ken 7y ago

Snopes and Mythbusters showed how you can actually make entertainment out of "This thing we thought might be true, actually isn't true / doesn't work at all". As for the fairness issue, my first thought is to have it moderated by scientists but from different fields. It's like we do in other areas, like law. You need an expert on the general process, but if they have any personal connection to the topic at hand, they must recuse themselves. In a sense, it's just a specific type of scientific journal, right? An online journal of only null results.

TeMPOraL 7y ago

Great idea, would love to see it. Related, there were journals like this at some point, e.g. International Journal of Negative & Null Results, or ournal of Articles in Support of the Null Hypothesis, and I think at least one more. Not sure how they fare, though.

godelski 7y ago

To do this, it somehow needs to gain the reputation of a publication but also not encourage people doing just frivolous research (at least not on their own dime). But I think this kind of thing would be INSANELY useful. Especially if data was attached. (This could possibly help with the above problem because reputations develop).

bilifuduo 7y ago

The Series of Unsurprising Results (SURE) Journal is starting to do this, but only for economics papers: https://blogs.canterbury.ac.nz/surejournal/ https://blogs.canterbury.ac.nz/surejournal/ Vox recently ran a good profile on them and the importance of null results in general: https://www.vox.com/future-perfect/2019/5/17/18624812/publication-bias-economics-journal https://www.vox.com/future-perfect/2019/5/17/18624812/public...

fghtr 7y ago

I am not sure this would be as useful as the new method of "pre-registered studies" [0], where the null findings appear automatically in the publications. Null findings are not the goal of this approach, but a side effect. Important are the initial hipotheses, which pass peer-review. [0] https://www.nature.com/articles/d41586-018-07118-1 https://www.nature.com/articles/d41586-018-07118-1 via https://news.ycombinator.com/item?id=18297724 https://news.ycombinator.com/item?id=18297724

grumdan 7y ago

In a similar vein, I'd love to see some form of reward for replication studies, which are also hard to publish, since they're not novel and exciting. However, due to this focus on novelty, we got plenty of unreproducible results: https://en.wikipedia.org/wiki/Replication_crisis https://en.wikipedia.org/wiki/Replication_crisis In my ideal world, a scientific result would not be taken as meaningful until it's been replicated at least once.