5 ms·

I’ve never heard of a reviewer writing 15K lines of code. That would be a serious undertaking for someone who is reviewing as an unpaid community service in the

I’ve never heard of a reviewer writing 15K lines of code. That would be a serious undertaking for someone who is reviewing as an unpaid community service in their free time outside of doing their own research/teaching/etc. Maybe a very dedicated reviewer might check a short simulation code that can be written in a few hours max, but even that is exceptionally rare in my experience. Reviewers usually look at the results and validation simulations in the manuscript, and if they look reasonable trust the code doesn’t have major bugs affecting accuracy.

OJFord 6y ago
Exactly, the person I replied to was saying there's no need to open source the code, it's enough for the paper to describe it. My point is that unless it's re-implemented and verified (which obviously it isn't) as part of reviewing the paper, that's not the same at all. It's like reviewing 'Assuming X, Y' and saying 'well I can't verify X, but yep, X->Y checks out. Y.'.
- denzil_correa 6y ago
  > My point is that unless it's re-implemented and verified (which obviously it isn't) as part of reviewing the paper, that's not the same at all. Even if it is re-implemented and verified it won't be the same. There are a million possible conditions for "re-implementation". Where would you draw the line? How many "re-implementations" and "types of conditions" do you need to be sure? You probably need to have expectation mismanagement from a scientific paper. A scientific article will only say "We tried this idea under conditions X,Y,Z and it works with A,B,C metrics and I, J, K assumptions". That is the crux of ANY empirical science - Computer Science or otherwise. That's all that they are paid (and incentivised) for. A computational scientific experiment is not a software product. If you want more, you need to pour in more funding and give them more resources explicitly for those purposes. Either that, or if you want to be 100% verified - you can choose to read theoretical papers where mathematical proofs are "verification".
  - OJFord 6y ago
    Sure, but: > "We tried this idea under conditions X,Y,Z and it works with A,B,C metrics and I, J, K assumptions". Is what I'm calling the assumption (the whole sentence). The implementation is essentially the methodology, and IMO therefore needs to be included. Note I'm arguing against 'academic, paper-supporting, code does or should not need to be open sourced', I'm not saying it needs to be 'software product' quality, written in the same way, to the same expectations, packaged, or anything like that. Just available for someone to say 'wait a second, you didn't try it under conditions, X,Y,Z, because Y gets negated here', or whatever.

OJFord 6y ago

Exactly, the person I replied to was saying there's no need to open source the code, it's enough for the paper to describe it. My point is that unless it's re-implemented and verified (which obviously it isn't) as part of reviewing the paper, that's not the same at all. It's like reviewing 'Assuming X, Y' and saying 'well I can't verify X, but yep, X->Y checks out. Y.'.

denzil_correa 6y ago
> My point is that unless it's re-implemented and verified (which obviously it isn't) as part of reviewing the paper, that's not the same at all. Even if it is re-implemented and verified it won't be the same. There are a million possible conditions for "re-implementation". Where would you draw the line? How many "re-implementations" and "types of conditions" do you need to be sure? You probably need to have expectation mismanagement from a scientific paper. A scientific article will only say "We tried this idea under conditions X,Y,Z and it works with A,B,C metrics and I, J, K assumptions". That is the crux of ANY empirical science - Computer Science or otherwise. That's all that they are paid (and incentivised) for. A computational scientific experiment is not a software product. If you want more, you need to pour in more funding and give them more resources explicitly for those purposes. Either that, or if you want to be 100% verified - you can choose to read theoretical papers where mathematical proofs are "verification".
- OJFord 6y ago
  Sure, but: > "We tried this idea under conditions X,Y,Z and it works with A,B,C metrics and I, J, K assumptions". Is what I'm calling the assumption (the whole sentence). The implementation is essentially the methodology, and IMO therefore needs to be included. Note I'm arguing against 'academic, paper-supporting, code does or should not need to be open sourced', I'm not saying it needs to be 'software product' quality, written in the same way, to the same expectations, packaged, or anything like that. Just available for someone to say 'wait a second, you didn't try it under conditions, X,Y,Z, because Y gets negated here', or whatever.