10 ms·

The workflow for modern scientific research always seems so amateurish from the perspective of basic software development. > We set up a video meeting, and de

• bachmeier 6y ago
  The journal I edit has a policy that the code and data has to be made available to the reviewers. I was surprised at the number of reviewers that dig into the code.
  • semi-extrinsic 6y ago
    TBH, code (even when all you can do is just looking at it and understanding how it works) is probably the most useful thing to the reviewers themselves. Imagine the equivalent for reviewing an experimental paper, if you were given a free instant travel to the place where they did the experiment and you could look at their setup and tinker with it and see how it works. I guarantee you a lot of reviewers would go for that.
    • bachmeier 6y ago
      I've learned that what you say is true. This is particularly the case for papers that use small datasets and not especially complicated empirical analysis.
  • jacobolus 6y ago
    Surprised that the reviewers do or don't look at the code?
    • bachmeier 6y ago
      Surprised that the reviewers do look at the code. I had assumed they wouldn't want to put the time in. In hindsight, I think it's often the case that it's faster to look at the code than to read an unclear methodological description.
      • jbay808 6y ago
        Speaking as a reviewer, reviews do take an immense amount of time, especially just in trying to figure out what was done and how. Code would actually speed things up a lot.
• pbalau 6y ago
  > The proper solution would be to publish the code along with the paper so that others could directly review it. Most ML researchers write code to prove their theory works, not to check if it works.
• marmaduke 6y ago
  > workflow for modern scientific research careful with the word modern, as the latest generation of students and postdocs is using Git, CI/CD etc, sometimes militantly but this amateurish approach reflects the incentive structure, where the goal isn't to produce a robust solution but to achieve notoriety and impact in one's domain. This impact factor plays into getting the next grant which creates a feedback loop that makes any shortcuts to the latest shiny thing obligatory. Even when failures become apparent, scientists can play the feature-not-a-bug card very frequently to stay on top.
• CJefferson 6y ago
  I partially blame software infrastructure here. I've tried to do reproducible research, but just setting something up when I get the exact version of GCC, python and python packages I want, in such a way I can get the same versions 2 years later, is a collosal pain. Just dumping a pile of code people won't be able to run isn't super useful, and becomes a source of continous complaints and requests for fixes.
  • kansface 6y ago
    You could make a container with the pipeline and data set up to go. Also, ideally, you'd publicly host the Dockerfile or w/e build time config, too.
    • ivan888 6y ago
      Downvoters: explain? Maybe this suggestion is aimed too much at a specific piece of technology (Docker). But there are plenty of tools out there to create reproducible builds, which I think is the overall point of this response
      • afarrell 6y ago
        Speculation: many downvoters are people who have repeatedly had a frustrating user experiences and are just exasperated at being repeatedly told how thier tools are good enough.
      • TheGallopedHigh 6y ago
        There is so much on the plate of the academic that to learn yet another piece of software — like docker which can be trying at the best of times — is yet another straw that can break the back.
      • mattkrause 6y ago
        I didn't downvote but... I'm skeptical that specific tools and "formalities" like requiring Docker containers will do a lot to improve the quality of scientific research. Obviously, being able to re-run an old analysis is better than not being able to do so but that wouldn't have helped (much) with this issue The root cause here was a conceptual problem, not a library version mismatch. They used the same data to a) group data points into bins and b) test for differences between those bins. Having a perfectly-engineered, smoothly-running pipeline might have made it easier to confirm the result, but I don't think it would help detect it. The things that would have things like rewarding collaboration (more eyes make bugs shallower) instead of rewarding mostly lead authorship, time and support to carefully mull over findings instead of a helter-skelter rush to publish, and a publication culture that doesn't avoid ambiguity. Those are much harder to achieve though....
• robotresearcher 6y ago
  My lab practice is to publish the repo path and the full hash of the commit that gave the results in the paper. The repo might have subsequent bug fixes, new code or data, etc. But you can get and run the exact code & data that the claims are based on if you wish. The published paper contains the hash printed in the text.
• conformist 6y ago
  Yes, but there needs to be a more complete solution to this. For this to happen, somebody should start funding one (relatively) professional developer per research group. Or, at least, there should be a way to fund "methodology/development" researchers/PhD students that have some domain knowledge, but mainly focus on aspects of publishing code, data, stats methodology, maintaining it, and ensuring that others can use it. Unfortunately, in many fields there does not seem to be a niche for that. But clearly this is a very important problem, possibly more important than funding a larger number of research ideas?
• justinpombrio 6y ago
  > The proper solution would be to publish the code along with the paper so that others could directly review it. This is taking root! More CS conferences should get onboard. https://www.artifact-eval.org/about.html https://www.artifact-eval.org/about.html (certificate seems to be expired...)
• ianbooker 6y ago
  In most fields papers / articles are still the one critical metric crucial for career advancement. You will sit on data and code until you are ready to publish, sometimes for years. Datasets and analytical code must be regarded as a scientific achievement on its on, albeit not as substitute for the "full" substantive analysis in a paper. A social science PhD in Denmark, for example, must write two to four peer reviewed papers at the moment. The option to substitute one of the articles with a data set? Things get better though: People nowadays at least understand my citations better, since I cite at least the explicitly included R-packages used in my work.
• ahi 6y ago
  Universities are trying, but the incentives for researchers just aren't there. https://deepblue.lib.umich.edu/ https://deepblue.lib.umich.edu/
• mhh__ 6y ago
  > The workflow for modern scientific research always seems so amateurish from the perspective of basic software development. It's sad but it's true. Although there are fundamental pressures that come because academia is academia, I pretty firmly believe that this can be fixed culturally too. Ultimately, I think people in some fields enjoy programming but view it to be either beneath them or beneath their paper. This is clearly not the case for Computer Science, but I noticed a few papers in the tire society journal which presented promising results but with absolutely no source code. One slightly ugly truth is that a lot of academics don't really know how to program properly or how computer actually work (Again, outside CS for obvious reasons). I'm reminded of the Imperial Covid Model's bugs as a fairly easy example. As to culture-question - the obvious solution is to start requiring disclosure of almost all materials and data used to construct a paper - for instance if Hendrik Schon had to publish the data behind the curves in his graphs his fraud could've been spotted almost immediately.
  • tehjoker 6y ago
    I've gotta say, I don't read conservative papers, but I follow news closely, and up to this day I didn't know about the bugs. The media landscape is crazy. However, it looks like they've been updating the model (and are at version 10 on github). I have no idea what their results look like now.
  • esyir 6y ago
    I think the catch here is that people in academia who like cs/eng rapidly leave academia for the much better paying sde/data science roles. Then what you have left are the people who care less about the code, at least in aggregate.
• musicale 6y ago
  > The proper solution would be to publish the code along with the paper so that others could directly review it. Ideally, the raw data would also be published, Definitely. Some journals and conferences have badging for papers that also submit their digital artifacts, and also for experiments that have been independently reproduced. Personally I'd like to see entire journals or conference sessions where the code and experimental data for every paper are available for review. Sometimes authors can't submit their code for corporate IP reasons and data for privacy or other reasons, but the more digital artifacts that can be submitted, reviewed, and built upon, the better.