4 ms·

This criticism is mostly outdated- including the analysis code is now required for most good journals. As a computational biology academic PI, I review a lot of

by _yb2s 2y ago

This criticism is mostly outdated- including the analysis code is now required for most good journals. As a computational biology academic PI, I review a lot of papers, and don’t approve anything without code unless the analysis is trivial- like a standard off the shelf statistical test. In the past this objection has been overruled by editors, but not in the last few years. That said, people misunderstand what peer review is and trust it too much. It is just a quick sanity check, and serves mostly to get feedback on how to make a paper more understandable. Good faith is assumed, reviewers do not look for fraud or hidden mistakes, that isn’t necessary or the point of peer review- fraud will eventually come out and consequences are severe. In most cases when I review a paper I have less than an hour to look it over, and mostly try to suggest things to make the paper easier to understand.

belter 2y ago
I can see your analysis being used by participants cited as paper co-authors... "...my name's on the paper, but I mostly just cheered from the sidelines. I didn't dive into the data or run any analyses. Honestly, most co-authors don't. We just add names to look impressive. If the paper's not great, it's because real work is hard, and I was more of a supportive figure rather than an actual contributor..." :-)
Al-Khwarizmi 2y ago
I don't think the criticism is outdated, in spite of what you mention. Indeed, in the last few decades, releasing the code has gone from uncommon to standard practice. But in my view, both you and the post author are greatly overstating the importance of releasing the code. Sure, releasing it is clearly better than not releasing it, it's a step forward. But it makes very little difference to peer review, if at all, since peer reviewers hardly ever actually open the code. You already are lucky if you can find reviewers that read the whole paper rather than skimming through the dense central sections - good luck finding reviewers that will actually check the code, considering that it takes much more time and effort than reading a paper. When I'm myself a reviewer, I also penalize papers with no code - but papers with code might as well link the Minesweeper source and most of the time I wouldn't notice, because I don't have time to check - and I devote more time per review than most people I know. Regarding your second paragraph, that is what peer review actually is in the present time, but the point is that it's supposed to be much more than that. And the problem is not only outright fraud that might have severe consequences, the post itself mentions "soft fraud" that will hardly ever have any consequence (who is going to notice that someone made several experiments and only published the one favoring their conclusion?). And there are also problems not related to fraud - for example, peer review is supposed to be not only about correctness but choosing the best work for top-tier venues, good but not that impactful work for second tier, etc. Shallow peer review often means that work that is not fraudulent, but not stellar is accepted in better venues due to famous authors/institutions or just authors that are especially skilled in "pitching" their work, and vice versa.
- SiempreViernes 2y ago
  In practice fraud and file-drawer effects observed (estimated from all the large N replication projects that have sprung up) are much less common than what the replication crisis headlines imply, a lot of it seems to simply be that in practice a "null hypothesis" is better understood as a continuous distribution because the null model doesn't match reality perfectly. See "The replication crisis is not a crisis of false positives" Coleman et al 2024, https://osf.io/preprints/socarxiv/rkyf7 https://osf.io/preprints/socarxiv/rkyf7
- JR1427 2y ago
  In my experience, getting someone else's code from a paper to run is a PITA. In life science, the person who wrote the code is often not really that experienced in this kind of thing, and certainly didn't have ease of sharing and running in mind when they started the project. I've long had in the back of my mind some kind of cloud-based platform for writing, running, storing, and sharing scientific code. It would be the standard, and journals would (in time) ask that analysis be done using it, or that reasons are provided as to why it could not be used. Maybe one already exists?
  - JR1427 2y ago
    One success story is that of the imageJ/Fiji project, which provided/provides a standard way of writing and running image analysis workflows.
  - jltsiren 2y ago
    > I've long had in the back of my mind some kind of cloud-based platform for writing, running, storing, and sharing scientific code. And now there are 15 competing standards. Everyone uses the computing resources available at their institution, and each institution makes its own choices. Running someone else's code almost always involves migrating it to a new environment. Even using something as common as Docker is problematic, as it doesn't always interact nicely with the rest of the environment (such as Slurm).
- ufmace 2y ago
  In a perfect world, studies should not only be required to include the code, any of the following should also be a reason to reject a publication: * The code can't be run outside the original researcher's undocumented special snowflake system * A fellow scientist who understands the domain can't understand what the code is doing and so is unable to verify that it isn't making any mistakes * It'd be really nice if there were some sort of test cases that could verify that the code in question produced results in line with what the scope of the paper in question considers to be established science in that domain before it was applied to the experimental data - probably not always possible, but it'd be good to at least try Yeah I know this doesn't touch the dozens of other reasons the whole institution is in trouble, and for those reasons usually nobody who has the skill and time to properly review the code actually reviews it carefully. For that matter, most scientists aren't very good at writing code that's easy to understand and will run everywhere. Just one more set of things that ought to go on the list of things that really should be changed.

by _yb2s 2y ago

belter 2y ago

I can see your analysis being used by participants cited as paper co-authors... "...my name's on the paper, but I mostly just cheered from the sidelines. I didn't dive into the data or run any analyses. Honestly, most co-authors don't. We just add names to look impressive. If the paper's not great, it's because real work is hard, and I was more of a supportive figure rather than an actual contributor..." :-)

Al-Khwarizmi 2y ago

I don't think the criticism is outdated, in spite of what you mention. Indeed, in the last few decades, releasing the code has gone from uncommon to standard practice. But in my view, both you and the post author are greatly overstating the importance of releasing the code. Sure, releasing it is clearly better than not releasing it, it's a step forward. But it makes very little difference to peer review, if at all, since peer reviewers hardly ever actually open the code. You already are lucky if you can find reviewers that read the whole paper rather than skimming through the dense central sections - good luck finding reviewers that will actually check the code, considering that it takes much more time and effort than reading a paper. When I'm myself a reviewer, I also penalize papers with no code - but papers with code might as well link the Minesweeper source and most of the time I wouldn't notice, because I don't have time to check - and I devote more time per review than most people I know. Regarding your second paragraph, that is what peer review actually is in the present time, but the point is that it's supposed to be much more than that. And the problem is not only outright fraud that might have severe consequences, the post itself mentions "soft fraud" that will hardly ever have any consequence (who is going to notice that someone made several experiments and only published the one favoring their conclusion?). And there are also problems not related to fraud - for example, peer review is supposed to be not only about correctness but choosing the best work for top-tier venues, good but not that impactful work for second tier, etc. Shallow peer review often means that work that is not fraudulent, but not stellar is accepted in better venues due to famous authors/institutions or just authors that are especially skilled in "pitching" their work, and vice versa.

SiempreViernes 2y ago
In practice fraud and file-drawer effects observed (estimated from all the large N replication projects that have sprung up) are much less common than what the replication crisis headlines imply, a lot of it seems to simply be that in practice a "null hypothesis" is better understood as a continuous distribution because the null model doesn't match reality perfectly. See "The replication crisis is not a crisis of false positives" Coleman et al 2024, https://osf.io/preprints/socarxiv/rkyf7 https://osf.io/preprints/socarxiv/rkyf7
JR1427 2y ago
In my experience, getting someone else's code from a paper to run is a PITA. In life science, the person who wrote the code is often not really that experienced in this kind of thing, and certainly didn't have ease of sharing and running in mind when they started the project. I've long had in the back of my mind some kind of cloud-based platform for writing, running, storing, and sharing scientific code. It would be the standard, and journals would (in time) ask that analysis be done using it, or that reasons are provided as to why it could not be used. Maybe one already exists?
- JR1427 2y ago
  One success story is that of the imageJ/Fiji project, which provided/provides a standard way of writing and running image analysis workflows.
- jltsiren 2y ago
  > I've long had in the back of my mind some kind of cloud-based platform for writing, running, storing, and sharing scientific code. And now there are 15 competing standards. Everyone uses the computing resources available at their institution, and each institution makes its own choices. Running someone else's code almost always involves migrating it to a new environment. Even using something as common as Docker is problematic, as it doesn't always interact nicely with the rest of the environment (such as Slurm).
ufmace 2y ago
In a perfect world, studies should not only be required to include the code, any of the following should also be a reason to reject a publication: * The code can't be run outside the original researcher's undocumented special snowflake system * A fellow scientist who understands the domain can't understand what the code is doing and so is unable to verify that it isn't making any mistakes * It'd be really nice if there were some sort of test cases that could verify that the code in question produced results in line with what the scope of the paper in question considers to be established science in that domain before it was applied to the experimental data - probably not always possible, but it'd be good to at least try Yeah I know this doesn't touch the dozens of other reasons the whole institution is in trouble, and for those reasons usually nobody who has the skill and time to properly review the code actually reviews it carefully. For that matter, most scientists aren't very good at writing code that's easy to understand and will run everywhere. Just one more set of things that ought to go on the list of things that really should be changed.