3 ms·

Imo, A more realistic thing to do is "replicability review" and/or requirement to submit "methodology map" to each paper. The former would be a back and forth

Imo, A more realistic thing to do is "replicability review" and/or requirement to submit "methodology map" to each paper. The former would be a back and forth between a reviewer that inquire and ask questions (based on the paper) with the goal to reproduce the result, but don't have to actually reproduce it. This is usually good to find out missing details in the paper that the writer just took for granted everyone in the field knows (I've met Bio PHD that have wasted Months of their life tracking up experimental details not mentioned in a paper) The latter would be the result of the former. Instead of having pages long "appendix" section in the main paper, you produce another document with meticulous details of the experiment/methodology with every stone turned together with an peer reviewer. Stamp it with the peer reviewes name so they can't get away with hand wavy review. I've read too many papers where important information to reproduce the result is omitted. (for ML/RL) If the code is included I've countless of times found implementation details that is not mentioned in the paper. In matter of fact, there's even results suggesting that those details are the make or break of certain algorithms. [1] I've also seen breaking details only mentioned in code comments... Another atrocious thing I've witnessed is a paper claiming they evaluated their method on a benchmark and if you check the benchmark, the task they evaluated on doesn't exit! They forked the benchmark and made their own task without being clear about it! [2] Shit like this make me lose faith in certain science directions. And I've seen a couple of junior researcher giving it all up because they concluded it's all just house of cards. [1] https://arxiv.org/abs/2005.12729 https://arxiv.org/abs/2005.12729 [2] https://arxiv.org/abs/2202.02465 https://arxiv.org/abs/2202.02465 Edit: also if you think that's too tedious/costly, reminder that publishers rake in record profits so the resources are already there https://youtu.be/ukAkG6c_N4M https://youtu.be/ukAkG6c_N4M

kergonath 3y ago
> I've met Bio PHD that have wasted Months of their life tracking up experimental details not mentioned in a paper Same. Now, when I review manuscripts, I pay much more attention to whether there is enough information to replicate the experiment or simulation. We can put out a paper with wrong interpretations and that’s fine because other people will realise that when doing their own work. We cannot let papers get published if their results cannot be replicated. > The latter would be the result of the former. Instead of having pages long "appendix" section in the main paper, you produce another document with meticulous details of the experiment/methodology with every stone turned together with an peer reviewer. Stamp it with the peer reviewes name so they can't get away with hand wavy review Things that take too much space to go in the experimental section should go to a electronic supplementary information document. But then it would be nice if the ESI were appended to the article when we download a PDF because tracking them is a pain in the backside. Some fields are better than others about this, for example in materials characterisation studies it’s very common to have ESI with a whole bunch of data and details. Large dataset should go to a repository or a dataset journal, that way the method is still peer reviewed and the dataset has a doi and is much easier to re-use. It’s also a nice way of doubling a student’s papers count by the end of their PhD. > Another atrocious thing I've witnessed is a paper claiming they evaluated their method on a benchmark and if you check the benchmark, the task they evaluated on doesn't exit! They forked the benchmark and made their own task without being clear about it! [2] That’s just evil!
- Maxion 3y ago
  > Large dataset should go to a repository or a dataset journal, that way the method is still peer reviewed and the dataset has a doi and is much easier to re-use. This may be possible in some sciences, but not in epidemiology or biomed. Often the study is based on tissue samples owned by some entity, with permission granted only to some certain entity. Datasets in epidemiology are often full of PII, and cannot be shared publicly for many reasons.

kergonath 3y ago

> I've met Bio PHD that have wasted Months of their life tracking up experimental details not mentioned in a paper Same. Now, when I review manuscripts, I pay much more attention to whether there is enough information to replicate the experiment or simulation. We can put out a paper with wrong interpretations and that’s fine because other people will realise that when doing their own work. We cannot let papers get published if their results cannot be replicated. > The latter would be the result of the former. Instead of having pages long "appendix" section in the main paper, you produce another document with meticulous details of the experiment/methodology with every stone turned together with an peer reviewer. Stamp it with the peer reviewes name so they can't get away with hand wavy review Things that take too much space to go in the experimental section should go to a electronic supplementary information document. But then it would be nice if the ESI were appended to the article when we download a PDF because tracking them is a pain in the backside. Some fields are better than others about this, for example in materials characterisation studies it’s very common to have ESI with a whole bunch of data and details. Large dataset should go to a repository or a dataset journal, that way the method is still peer reviewed and the dataset has a doi and is much easier to re-use. It’s also a nice way of doubling a student’s papers count by the end of their PhD. > Another atrocious thing I've witnessed is a paper claiming they evaluated their method on a benchmark and if you check the benchmark, the task they evaluated on doesn't exit! They forked the benchmark and made their own task without being clear about it! [2] That’s just evil!

Maxion 3y ago
> Large dataset should go to a repository or a dataset journal, that way the method is still peer reviewed and the dataset has a doi and is much easier to re-use. This may be possible in some sciences, but not in epidemiology or biomed. Often the study is based on tissue samples owned by some entity, with permission granted only to some certain entity. Datasets in epidemiology are often full of PII, and cannot be shared publicly for many reasons.