4 ms·

I'm thinking that maybe they're not developers as such, just scientists, and therefore may not know how to do that. Could you open an issue that could show the

• p4wnc6 11y ago
  Unfortunately, I don't have a Bitbucket account because I don't approve of Atlassian, so I can't. But hopefully someone here who does use Atlassian products will do it?
  • nekopa 11y ago
    Looking at the repo, seems like there is only 1 commit. I have a BB account, so when I get back to my PC I'll drop them a message on the repo.
    • p4wnc6 11y ago
      The fact that there is only one commit is a bit disconcerting. It suggests that they edited, debugged, and modified the code "offline" until they felt it was "good enough" to share, so whatever revision history there was when the paper was being created, revised, and submitted, is lost. We can only take the authors' word for it that in between getting results for the paper and releasing the commit, they didn't make any significant alterations that would affect reproducibility. The point is not to be critical of these particular authors. Good on them for sharing their code. The point is to highlight that it's harmful for academics to "hide" their code until they get it all together for the "perfect" commit. It defeats the point of version control and their repo functions instead as basically nothing but a download link. All the reproducibility problems would remain the same as if the code was provided as just a tarball linked from some academic web page.
      • ktRolster 11y ago
        The fact that there is only one commit is a bit disconcerting. It suggests that they edited, debugged, and modified the code "offline" until they felt it was "good enough" to share, so whatever revision history there was when the paper was being created, revised, and submitted, is lost. Obsessions with code history are a little disconcerting. It's interesting to see, but has little relation to the quality of the paper (or the code base, for that matter).
        • p4wnc6 11y ago
          > Obsessions with code history are a little disconcerting. Wanting to see code history is not anywhere close to an "obsession" -- that's just a loaded word that does nothing but distract from the issue. > It's interesting to see, but has little relation to the quality of the paper (or the code base, for that matter). This is just completely wrong. I don't want to see code history because I care about the author's earlier mistakes, previous implementation attempts, refactorings, etc., from any sort of judgemental, coding skill point of view. The reason it's important is that maybe they used version 3 when they ran some part of the study and generated some of their data. They figured it was good enough, but then later they updated some code to version 4. They didn't need to re-run things for that older data, but they needed version 4 for something else. But, version 4 makes a change that causes an error, or an undefined behavior, or a change to some statistical degrees of freedom calculation or something, that would imply the data from version 3 also needs updated. So the authors are using something that came out of version 3, looked good, and they thought never needed revisiting. Then they are also using something from version 4 that, overlooked by them, invalidates or changes something from version 3. All of this gets whitewashed because they go through and manually finalize the code, and upload it to a public repo. Now, you as the reader are not even aware that there ever even was a version 3, let alone that some of the data came from version 3, some came from version 4, and the mixup is based on an oversight or something. The code history really matters. If you cannot reproduce the results immediately with the tarball the authors release as the "good" version, and there is no version history, then, effectively, there is zero reproducibility. You have to push back on the authors, who weren't making their changes under version control when the version 3 to version 4 issue arose, and so good luck ever getting an answer. I've actually experienced this first hand. I had to reproduce some quant finance research on country-based beta calculations. It turned out that in an early draft of the work, the team had used a completely different data set and in the earlier data set, the country code for Sweden had been set to "SW" even though the ISO country code for Sweden is "SE". They got some intermediate statistics out of that data set and saved them for later usage. Then later on they used another data set and someone had the bright idea of using "SW" for Switzerland, even though that country code is "CH". But because all that remained from the first data set was aggregate statistics, and no version history of the code that had loaded the spreadsheet and made these assumptions at that time, there was no way to figure this out. In the final result, Swiss and Swedish data was being aggregated together and throwing numbers off. And those two countries both have very small market caps, so this is hard to spot and track down. After a ton of outrageous debugging, we just happened to get lucky and notice that the difference was almost equal to the largest Swiss company in that time period (Nestle I think), and then began to wonder about the "SW" issue and then tried all permutations of separating the two data sets into Swiss and Swedish data until things matched. If we could have bisected the original source revisions, it would have saved weeks worth of tedious, expensive data drilldown nonsense to replicate a result.