4 ms·

I think this sets a bad example. In many fields the current standard is that code isn't published at all. If bad coding practices will cause you to get bullied

I think this sets a bad example. In many fields the current standard is that code isn't published at all. If bad coding practices will cause you to get bullied into retracting your paper, that's just another argument for why people won't publish their code. Of course if somebody does find a big that invalidates the results of the paper given the stated assumptions, that's another matter, but that doesn't seem to be what's alleged here.

nradov 6y ago
The standard needs to change. Publication of code and data ought to be required.
- denzil_correa 6y ago
  Why exactly? We could also probably require a website to test the application, a mobile app on all platforms and then a native OS apps on all platforms. I apologise for being facetious but writing code is a serious, specialized task. It should be left to the people who specialise in it, scientists aren't the specialists for it. I think it's important to understand that we only require information to generate algorithms written in the paper. As such, assumptions and limitations should be listed enough to generate the material in the paper. One way to do so is code and data, it is certainly not the only way.
  - toyg 6y ago
    I empathise with what you say, but it's also true that maybe things could be better. We don't expect universities to build their own laboratories and halls of residence from first principles - they will call on property developers, architects, and contractors. It would be good if there were more IT services, built by specialists, that academics could tap when required - so they would spend more time advancing their fields of expertise and less time writing mediocre code. Things like PythonAnywhere / Notebooks, Lambda etc are a step forward, but there is still ample space for advancement.
    - metreo 6y ago
      Getting software engineers interested in your science isn't easy. Though in this particular case there ought to have been enough funding. I'm not knowledgeable about its specific developments.
  - OJFord 6y ago
    Do reviewers of such papers write their own implementations, and verify they get (roughly) the same results?
    - tastroder 6y ago
      Not generally, no. Even if they had access to the code at the time of review, which is hardly ever the case in my experience, there's simply no time. I also don't see a point in doing so, most papers only describe minute changes to some state of the art thing that's well established so you trust in the academic integrity and experience of authors to get that part right and usually focus on parts that matter over things like the ones this issue brings up. Even for real reproductions, that require work just like they do in any other scientific domain, the points in that issue seem pretty irrelevant. There's things like FAIR standards on reproducibility but as far as the status quo goes that repo doesn't look too bad. I could not care less if tests for some project are badly written, at least it's in written in a non-obscure language and shows a somewhat sane structure. What's next? Calling for redactions because somebody didn't follow the same tabs vs spaces paradigm? There's nothing in that issue w.r.t. whatever scientific finding this was used for and this general phenomenon is really fascinating. Instead of a constructive discussion with the technical or scientific folk that put their work out there, or engagement with the politicians that drew conclusions based on those scientific findings, you get github issues and Twitter threads mixing a bunch of unrelated concerns.
      - metreo 6y ago
        I think the matter at hand is much less trivial than tabs v. spaces.
    - krull10 6y ago
      I’ve never heard of a reviewer writing 15K lines of code. That would be a serious undertaking for someone who is reviewing as an unpaid community service in their free time outside of doing their own research/teaching/etc. Maybe a very dedicated reviewer might check a short simulation code that can be written in a few hours max, but even that is exceptionally rare in my experience. Reviewers usually look at the results and validation simulations in the manuscript, and if they look reasonable trust the code doesn’t have major bugs affecting accuracy.
OJFord 6y ago
> Of course if somebody does find a [bug] that invalidates the results of the paper given the stated assumptions, that's another matter And obviously the 'risk' of that happening isn't a reason not to publish it - you don't not publish the paper for risk that peer review reveals an error!

nradov 6y ago

The standard needs to change. Publication of code and data ought to be required.

denzil_correa 6y ago
Why exactly? We could also probably require a website to test the application, a mobile app on all platforms and then a native OS apps on all platforms. I apologise for being facetious but writing code is a serious, specialized task. It should be left to the people who specialise in it, scientists aren't the specialists for it. I think it's important to understand that we only require information to generate algorithms written in the paper. As such, assumptions and limitations should be listed enough to generate the material in the paper. One way to do so is code and data, it is certainly not the only way.
- toyg 6y ago
  I empathise with what you say, but it's also true that maybe things could be better. We don't expect universities to build their own laboratories and halls of residence from first principles - they will call on property developers, architects, and contractors. It would be good if there were more IT services, built by specialists, that academics could tap when required - so they would spend more time advancing their fields of expertise and less time writing mediocre code. Things like PythonAnywhere / Notebooks, Lambda etc are a step forward, but there is still ample space for advancement.
  - metreo 6y ago
    Getting software engineers interested in your science isn't easy. Though in this particular case there ought to have been enough funding. I'm not knowledgeable about its specific developments.
- OJFord 6y ago
  Do reviewers of such papers write their own implementations, and verify they get (roughly) the same results?
  - tastroder 6y ago
    Not generally, no. Even if they had access to the code at the time of review, which is hardly ever the case in my experience, there's simply no time. I also don't see a point in doing so, most papers only describe minute changes to some state of the art thing that's well established so you trust in the academic integrity and experience of authors to get that part right and usually focus on parts that matter over things like the ones this issue brings up. Even for real reproductions, that require work just like they do in any other scientific domain, the points in that issue seem pretty irrelevant. There's things like FAIR standards on reproducibility but as far as the status quo goes that repo doesn't look too bad. I could not care less if tests for some project are badly written, at least it's in written in a non-obscure language and shows a somewhat sane structure. What's next? Calling for redactions because somebody didn't follow the same tabs vs spaces paradigm? There's nothing in that issue w.r.t. whatever scientific finding this was used for and this general phenomenon is really fascinating. Instead of a constructive discussion with the technical or scientific folk that put their work out there, or engagement with the politicians that drew conclusions based on those scientific findings, you get github issues and Twitter threads mixing a bunch of unrelated concerns.
    - metreo 6y ago
      I think the matter at hand is much less trivial than tabs v. spaces.
  - krull10 6y ago
    I’ve never heard of a reviewer writing 15K lines of code. That would be a serious undertaking for someone who is reviewing as an unpaid community service in their free time outside of doing their own research/teaching/etc. Maybe a very dedicated reviewer might check a short simulation code that can be written in a few hours max, but even that is exceptionally rare in my experience. Reviewers usually look at the results and validation simulations in the manuscript, and if they look reasonable trust the code doesn’t have major bugs affecting accuracy.

OJFord 6y ago

> Of course if somebody does find a [bug] that invalidates the results of the paper given the stated assumptions, that's another matter And obviously the 'risk' of that happening isn't a reason not to publish it - you don't not publish the paper for risk that peer review reveals an error!