4 ms·

I agree with this article and have always thought about this but never voiced it. I think a few things could improve the quality and discovery of published pap

by Vaslo 7y ago

I agree with this article and have always thought about this but never voiced it. I think a few things could improve the quality and discovery of published papers: -After so many publications, could it be mandatory for random samples in that author’s publication be tested? I get that there is a limit on resources, but some advanced undergrads could do this with guidance. -I would love to see some version of a journal of failures. That is, well intentioned research that had poor outcomes. It was so frequently in Chemistry research that my compounds were useless or the methods to synthesize them did not work, and it would be helpful to document that. Unfortunately, there is no “Journal of Failed Chemistry.” Only research that ostensibly make a contribution with a clear outcome get published. So much time wasted experimenting where you could save another scientist time and encourage them down another path.

djtango 7y ago
>It was so frequently in Chemistry research that my compounds were useless or the methods to synthesize them did not work, and it would be helpful to document that. This mirrors my experience where a lot of the post-docs carried war stories of things like lab/country-specific humidity playing a role in synthetic methods succeeding (or failing). There were a lot of dark arts/tricks of the trade that people carried around with them: stuff like going that extra mile to dry things of water super thoroughly (even if it was not mentioned in the paper we were referencing).
- ISL 7y ago
  Many "dark arts" aren't known as dark arts to the people who know them. They're just the things you always do, because if you don't do them, nothing works. An approximate CS analog: Writing great commit messages, and using an SCM. You don't have to do them, nobody writes it up because those who know regard it as trivial, but if you don't do them, almost nothing works. Everyone who knows what they're doing does it.
  - djtango 7y ago
    >An approximate CS analog: Writing great commit messages, and using an SCM. This is definitely not the analogue. The analogue is always doing a clean install of the OS before running your experiment. Or only ever using Arch version x.y.z for replicating lab 1 and maybe a.b.c for replicating lab 2. It's knowing all the magic undocumented JVM flags ahead of running the application. Some you know to use as part of war scars/best practices. Some are just pure inside information from working in that lab or having a personal relationship with people in that lab.
codegladiator 7y ago
Partly the issue is because of the naming... Like "poor outcomes" and "failed"... Obviously there would be no awards for the poor outcomes and failed results and we would be back to square one where no one is incentivized to look at them positively.
tasogare 7y ago
What are you waiting for creating one? Motivate a few scientists to form a credible review comity, fill one paper [1] and voila you have the journal you dreamt about on a recognized and perennial web platform. This is of course some work, but if you really want it to exists it’s better not to wait for someone else to do it. Also allowing anonymous authors would help I think (to avoid the name being associated with failure). [1] https://www.openedition.org/15974?file=1 https://www.openedition.org/15974?file=1 Edit: openedition.org is for humanities, but there probably is alternative for other fields.
- MaxBarraclough 7y ago
  > What are you waiting for creating one? Creating a journal is a tremendous task, and only highly respected researchers have any chance. > if you really want it to exists it’s better not to wait for someone else to do it Again, most people simply cannot create a credible journal. Let's be realistic here.
wolfram74 7y ago
maybe a more serious version of the journal of irreproducible results? https://en.wikipedia.org/wiki/Journal_of_Irreproducible_Results https://en.wikipedia.org/wiki/Journal_of_Irreproducible_Resu...
glofish 7y ago
This is a variant of "the carrot and stick" approach that happens to be very short on the carrot and be very large on stick. Very unlikely to be effective. Time and again people think that when they have a problem it must mean the punishment was not severe enough. It won't work. The problem is that good science is not rewarded enough. Nobody rewards you for having published reliable stuff five years ago.
- weberc2 7y ago
  Unless the GP edited, there's no suggestion of punishment in his comment.
cepp 7y ago
On failures: many years ago I stumbled across the idea of a `CV of Failures` where you list all of the accomplishments you missed/were rejected from, etc. I think it's a very humbling idea and can see some applicability here. Example: https://www.princeton.edu/~joha/Johannes_Haushofer_CV_of_Failures.pdf https://www.princeton.edu/~joha/Johannes_Haushofer_CV_of_Fai...
bumby 7y ago
I recently noticed SURE journal, intent on publishing "unsurprising" economic articles. I just wish there were more, or at least more journals rewarding sound methodology instead of unsound "results". Who knows how much effort is wasted reproducing sound, but unremarkable studies simply because they weren't published
Al-Khwarizmi 7y ago
There are a bunch of journals like that. If you google "Journal of Negative Results", you will see several for different fields. However, for some I know, it's really difficult to get in, because there are many more journals of positive than negative results, and there are probably many more negative results than positive... so your negative result had better be especially interesting.

by Vaslo 7y ago

djtango 7y ago

>It was so frequently in Chemistry research that my compounds were useless or the methods to synthesize them did not work, and it would be helpful to document that. This mirrors my experience where a lot of the post-docs carried war stories of things like lab/country-specific humidity playing a role in synthetic methods succeeding (or failing). There were a lot of dark arts/tricks of the trade that people carried around with them: stuff like going that extra mile to dry things of water super thoroughly (even if it was not mentioned in the paper we were referencing).

ISL 7y ago
Many "dark arts" aren't known as dark arts to the people who know them. They're just the things you always do, because if you don't do them, nothing works. An approximate CS analog: Writing great commit messages, and using an SCM. You don't have to do them, nobody writes it up because those who know regard it as trivial, but if you don't do them, almost nothing works. Everyone who knows what they're doing does it.
- djtango 7y ago
  >An approximate CS analog: Writing great commit messages, and using an SCM. This is definitely not the analogue. The analogue is always doing a clean install of the OS before running your experiment. Or only ever using Arch version x.y.z for replicating lab 1 and maybe a.b.c for replicating lab 2. It's knowing all the magic undocumented JVM flags ahead of running the application. Some you know to use as part of war scars/best practices. Some are just pure inside information from working in that lab or having a personal relationship with people in that lab.

codegladiator 7y ago

Partly the issue is because of the naming... Like "poor outcomes" and "failed"... Obviously there would be no awards for the poor outcomes and failed results and we would be back to square one where no one is incentivized to look at them positively.

tasogare 7y ago

What are you waiting for creating one? Motivate a few scientists to form a credible review comity, fill one paper [1] and voila you have the journal you dreamt about on a recognized and perennial web platform. This is of course some work, but if you really want it to exists it’s better not to wait for someone else to do it. Also allowing anonymous authors would help I think (to avoid the name being associated with failure). [1] https://www.openedition.org/15974?file=1 https://www.openedition.org/15974?file=1 Edit: openedition.org is for humanities, but there probably is alternative for other fields.

MaxBarraclough 7y ago
> What are you waiting for creating one? Creating a journal is a tremendous task, and only highly respected researchers have any chance. > if you really want it to exists it’s better not to wait for someone else to do it Again, most people simply cannot create a credible journal. Let's be realistic here.

wolfram74 7y ago

maybe a more serious version of the journal of irreproducible results? https://en.wikipedia.org/wiki/Journal_of_Irreproducible_Results https://en.wikipedia.org/wiki/Journal_of_Irreproducible_Resu...

glofish 7y ago

This is a variant of "the carrot and stick" approach that happens to be very short on the carrot and be very large on stick. Very unlikely to be effective. Time and again people think that when they have a problem it must mean the punishment was not severe enough. It won't work. The problem is that good science is not rewarded enough. Nobody rewards you for having published reliable stuff five years ago.

weberc2 7y ago
Unless the GP edited, there's no suggestion of punishment in his comment.

cepp 7y ago

On failures: many years ago I stumbled across the idea of a `CV of Failures` where you list all of the accomplishments you missed/were rejected from, etc. I think it's a very humbling idea and can see some applicability here. Example: https://www.princeton.edu/~joha/Johannes_Haushofer_CV_of_Failures.pdf https://www.princeton.edu/~joha/Johannes_Haushofer_CV_of_Fai...

bumby 7y ago

I recently noticed SURE journal, intent on publishing "unsurprising" economic articles. I just wish there were more, or at least more journals rewarding sound methodology instead of unsound "results". Who knows how much effort is wasted reproducing sound, but unremarkable studies simply because they weren't published

Al-Khwarizmi 7y ago

There are a bunch of journals like that. If you google "Journal of Negative Results", you will see several for different fields. However, for some I know, it's really difficult to get in, because there are many more journals of positive than negative results, and there are probably many more negative results than positive... so your negative result had better be especially interesting.