3 ms·

One of my greatest professional worries is publishing a paper with an honest data analysis error and being accused of fabrication. For this reason, I spend an "

One of my greatest professional worries is publishing a paper with an honest data analysis error and being accused of fabrication. For this reason, I spend an "unreasonable" amount of time triple-checking everything (my colleagues are annoyed by that). I wish there was software that would help me spot mistakes by looking at my code and/or write up, but I also recognize that it would be abused by those committing fraud.

karencarits 3y ago
This is also one of my fears, and my strategy has been openness - I share my analysis/code with colleagues/Github/as supplement. Maybe they can spot errors or reuse some of it, and I feel that there is some "protection" in it
- chaxor 3y ago
  It should be a requirement that every paper be completely reproducible, for any software part of the paper. On any OS or systém, it should work. Similar to requiring statistics to be reported a certain way, there should be guarantees that the code should be available and work in all circumstances.
  - analog31 3y ago
    Unfortunately, I don't think this is realistic. Is it provable? Does it require use of an "official" programming language and libraries, that are guaranteed to be maintained? Is Turbo Pascal for MS-DOS still supported? That's what I used for my dissertation, in 1993. It's a far cry beyond what's expected in branches of science that are not mainly about programming.
  - jaredsohn 3y ago
    Perhaps require making a docker container and writing a test script that includes inputs and expected outputs. There could be a website that loads those docker images and shows that the tests pass. Tests could be referenced within the paper. Would still require scrutiny to make sure that code doesn't hard-code the outputs.
- lumost 3y ago
  This should be the way. Folks will forgive sloppy code on a “fast” paper. However it may invite more replication attempts. Many folks have messed up simple algorithms before.
eviks 3y ago
Are there (m)any examples of that? Especially given the fact that exposures of clear fraud when exposed is so often just ignored
crocowhile 3y ago
I do the same. I write and prepare figures for each single paper that comes out from my lab. I double check each single Jupyter notebook and I share every single piece of raw data ( for the last paper we just submitted yesterday we uploaded 316Gb of data on zenodo). All this takes me weeks of course. I would be very skeptical of colleagues who publish more than 4-5 papers a year.

karencarits 3y ago

This is also one of my fears, and my strategy has been openness - I share my analysis/code with colleagues/Github/as supplement. Maybe they can spot errors or reuse some of it, and I feel that there is some "protection" in it

chaxor 3y ago
It should be a requirement that every paper be completely reproducible, for any software part of the paper. On any OS or systém, it should work. Similar to requiring statistics to be reported a certain way, there should be guarantees that the code should be available and work in all circumstances.
- analog31 3y ago
  Unfortunately, I don't think this is realistic. Is it provable? Does it require use of an "official" programming language and libraries, that are guaranteed to be maintained? Is Turbo Pascal for MS-DOS still supported? That's what I used for my dissertation, in 1993. It's a far cry beyond what's expected in branches of science that are not mainly about programming.
- jaredsohn 3y ago
  Perhaps require making a docker container and writing a test script that includes inputs and expected outputs. There could be a website that loads those docker images and shows that the tests pass. Tests could be referenced within the paper. Would still require scrutiny to make sure that code doesn't hard-code the outputs.
lumost 3y ago
This should be the way. Folks will forgive sloppy code on a “fast” paper. However it may invite more replication attempts. Many folks have messed up simple algorithms before.

eviks 3y ago

Are there (m)any examples of that? Especially given the fact that exposures of clear fraud when exposed is so often just ignored

crocowhile 3y ago

I do the same. I write and prepare figures for each single paper that comes out from my lab. I double check each single Jupyter notebook and I share every single piece of raw data ( for the last paper we just submitted yesterday we uploaded 316Gb of data on zenodo). All this takes me weeks of course. I would be very skeptical of colleagues who publish more than 4-5 papers a year.