3 ms·

Considering that a paper is run through a subjective and highly critical peer-review I can see why authors would be reluctant to share their code.

bigiain 8y ago
It's an interesting observation then that the people doing the "highly critical peer review" might promote papers with no source code showing how the calculations were done over papers with messy-but-available source code. I know some fields intentionally/explicitly hide their working ("Law and sausage are two things you do not want to see being made."), but I wouldn't have thought academia would subscribe to that... There's a lot to be said for the IETF's policy of "rough consensus and running code".
- itronitron 8y ago
  The paper should show how the calculations can be made, along with the results of that calculation, preferably on a publicly available dataset. A common strategy is to describe the calculation in highly convoluted or specialized language (mathematical formulas) in order to reduce the number of people that may try to reproduce the work. Having reviewed papers on multiple occasions, I wouldn't spend much if any time looking at the actual source code if it is available. Every paper submission is rushed because it needs to be ahead of what anyone else is doing (at least in CS and related fields).
  - bigiain 8y ago
    Do you see this as an inevitable "that's the way it is" part of how the peer review process works? Or as a flaw in the process? Or something in between. (A deeply cynical and suspicious part of me suspects this is because the prime motivator for "peer reviewed journals" is to make Elsiver richer, and academic knowledge sharing and reputation building is just a side effect of that...)
    - itronitron 8y ago
      Elsevier is like the cocktail lounge of the alcohol industry. They are just running a business by controlling access to the good stuff, that people with money have a taste for, and they will keep prices high for as long as they can. To be fair, some domains really need the good stuff and the cost for access to those journals is probably a drop in the bucket relative to their other operating expenses (or marketing expenses if we are talking about the pharmaceutical industry.) Every peer review process is different from the next, it varies by journal, by conference, and by month of the year. I probably can't emphasize enough how highly subjective every peer review is. Anyone that has a paper rejected should consider the validity of reviewer comments (which may be total BS on occasion), then edit and resubmit the paper to another venue.

bigiain 8y ago

It's an interesting observation then that the people doing the "highly critical peer review" might promote papers with no source code showing how the calculations were done over papers with messy-but-available source code. I know some fields intentionally/explicitly hide their working ("Law and sausage are two things you do not want to see being made."), but I wouldn't have thought academia would subscribe to that... There's a lot to be said for the IETF's policy of "rough consensus and running code".

itronitron 8y ago
The paper should show how the calculations can be made, along with the results of that calculation, preferably on a publicly available dataset. A common strategy is to describe the calculation in highly convoluted or specialized language (mathematical formulas) in order to reduce the number of people that may try to reproduce the work. Having reviewed papers on multiple occasions, I wouldn't spend much if any time looking at the actual source code if it is available. Every paper submission is rushed because it needs to be ahead of what anyone else is doing (at least in CS and related fields).
- bigiain 8y ago
  Do you see this as an inevitable "that's the way it is" part of how the peer review process works? Or as a flaw in the process? Or something in between. (A deeply cynical and suspicious part of me suspects this is because the prime motivator for "peer reviewed journals" is to make Elsiver richer, and academic knowledge sharing and reputation building is just a side effect of that...)
  - itronitron 8y ago
    Elsevier is like the cocktail lounge of the alcohol industry. They are just running a business by controlling access to the good stuff, that people with money have a taste for, and they will keep prices high for as long as they can. To be fair, some domains really need the good stuff and the cost for access to those journals is probably a drop in the bucket relative to their other operating expenses (or marketing expenses if we are talking about the pharmaceutical industry.) Every peer review process is different from the next, it varies by journal, by conference, and by month of the year. I probably can't emphasize enough how highly subjective every peer review is. Anyone that has a paper rejected should consider the validity of reviewer comments (which may be total BS on occasion), then edit and resubmit the paper to another venue.