5 ms·

This article seems to cover research software that even can be built. I claim the majority of _code_ written to support research articles is a collection of scr

This article seems to cover research software that even can be built. I claim the majority of _code_ written to support research articles is a collection of scripts written to produce figures to put in the paper. Even when the article is about an algorithm, the script that runs this algorithm is just good enough to produce the theoretically expected results; it is never tested, reproduced, or published, never mind being updated after publication. While others here point out that researchers = bad programmers is a lazy excuse, I think it is important to point out just how steep the learning curve of computer environments can be for the layperson that uses Excel or MATLAB for all their computational work. It can be a huge time investment to get started with tools, such as git or Docker, that we take for granted. I think recognizing this dearth of computer skills is a first step towards training researchers to be computer-competent. Currently, I find the attitude among academics (especially theorists) to be dismissive of the importance of such competencies.

Dumblydorr 6y ago
I am a research scientist published via R, Stata, and Excel analyses. My code documents wouldn't be helpful since the data is all locked up due to HIPAA concerns. We're talking names, health conditions, scrambled SSN, this isn't reproducible because the data is locked to those without security clearance. The code itself is a ton of munging and then some basic stat functions. This information can be gleaned from the methods section of the article anyway. So, really, my field of public health doesn't use GitHub or sharing much, there's simply too little benefit to the researcher to share their code. There's an unwarranted fear of getting your work poached. In modern science, publications are everything, they determine your career. Enabling your direct competitors, those who want the same grants and students and glories, is not common in science.
- asdf_snar 6y ago
  I don't disagree with you on any points. I have some academic friends who mostly do "a ton of munging and then some basic stat functions", as you say (but with less sensitive data). The problem is that their workflow is prone to human error. Even though the stat functions are simple, the proper labeling of inputs and outputs is less reliable. I have some research published for which I wrote MATLAB code years ago. I trust the fundamental results but not the values displayed in the tables. I would have personally benefited from rudimentary version control and unit testing.
- stult 6y ago
  In well designed software, data ingestion should be easily separable from the core logic of the application. Which is the point the parent comment is making. Some basic best practices would allow you to share your core code without implicating HIPAA. Even if it’s just basic stats, sharing the code makes it easier to reproduce your results and to check your logic. Although I agree with your analysis that enabling competitors in science is not common, it really, really should be. That’s kinda the point of publication, at least in theory. Sharing knowledge and methods.
  - jimmyvalmer 6y ago
    > enabling competitors in science ... really, really should be. Said someone whose livelihood doesn't depend on said competition.
- vharuck 6y ago
  As a public health statistician, I am very grateful to all researchers who publish code. More so for those who publish packages that make their techniques easy to use. I am not an expert in the field of statistics, just a grunt applying what you guys devise. It takes a while for me to do enough research and testing to be sure I'm correctly implementing new techniques. Even a basic pseudo-code walkthrough would immensely help. >My code documents wouldn't be helpful since the data is all locked up due to HIPAA concerns. We're talking names, health conditions, scrambled SSN, this isn't reproducible because the data is locked to those without security clearance. Is there a standard format for this kind of data? If so, consider using it. That way, others can easily create artificial datasets to test it. Even if you have no control over your data source, you can convert the raw data to the standard as a "pre-munging" step. >So, really, my field of public health doesn't use GitHub or sharing much, there's simply too little benefit to the researcher to share their code. Sad but true.
Doctor_Fegg 6y ago
> it is never tested, reproduced, or published This never ceases to amaze me. I regularly read recent papers on shortest-path algorithms. Each one is religiously benchmarked down to the level of saying what C++ compiler was used. But the code itself is almost never published.
txdv 6y ago
Reproducibility is a major principle of the scientific method. Yet computer scientists consistently fail to achieve reproducibility with a tool that is the most consistent at following instructions - the computer. Even private business is on the DevOps movement, because they see the positive effects of reproducibility. If the academic world is truly about science, then there is no more excuse, the tools are out there, they need to use them.
- Frost1x 6y ago
  This is really an artifact if unreasonable expectations and modern software ecosystems. When I say unreasonable expectations, the issue is that people assume they can use the latest greatest trendy library and get reproducible results. Good luck on the level of determinism you're looking for. You need to step back and look at more mature, simple codebases and what you can do in those sorts of environments when you want reproducibility. You can't cobble together a bunch of async services in the cloud and hope your Frankenstein tool gives you perfect results. It will give you good enough results for certain aspects if you focus on those specific aspects (banking does a good job of this with transactional processing and making sure values are consistent because it's their entire business, maybe your account or their web interface is skrewy but that's fine, that can fail).

Dumblydorr 6y ago

I am a research scientist published via R, Stata, and Excel analyses. My code documents wouldn't be helpful since the data is all locked up due to HIPAA concerns. We're talking names, health conditions, scrambled SSN, this isn't reproducible because the data is locked to those without security clearance. The code itself is a ton of munging and then some basic stat functions. This information can be gleaned from the methods section of the article anyway. So, really, my field of public health doesn't use GitHub or sharing much, there's simply too little benefit to the researcher to share their code. There's an unwarranted fear of getting your work poached. In modern science, publications are everything, they determine your career. Enabling your direct competitors, those who want the same grants and students and glories, is not common in science.

asdf_snar 6y ago
I don't disagree with you on any points. I have some academic friends who mostly do "a ton of munging and then some basic stat functions", as you say (but with less sensitive data). The problem is that their workflow is prone to human error. Even though the stat functions are simple, the proper labeling of inputs and outputs is less reliable. I have some research published for which I wrote MATLAB code years ago. I trust the fundamental results but not the values displayed in the tables. I would have personally benefited from rudimentary version control and unit testing.
stult 6y ago
In well designed software, data ingestion should be easily separable from the core logic of the application. Which is the point the parent comment is making. Some basic best practices would allow you to share your core code without implicating HIPAA. Even if it’s just basic stats, sharing the code makes it easier to reproduce your results and to check your logic. Although I agree with your analysis that enabling competitors in science is not common, it really, really should be. That’s kinda the point of publication, at least in theory. Sharing knowledge and methods.
- jimmyvalmer 6y ago
  > enabling competitors in science ... really, really should be. Said someone whose livelihood doesn't depend on said competition.
vharuck 6y ago
As a public health statistician, I am very grateful to all researchers who publish code. More so for those who publish packages that make their techniques easy to use. I am not an expert in the field of statistics, just a grunt applying what you guys devise. It takes a while for me to do enough research and testing to be sure I'm correctly implementing new techniques. Even a basic pseudo-code walkthrough would immensely help. >My code documents wouldn't be helpful since the data is all locked up due to HIPAA concerns. We're talking names, health conditions, scrambled SSN, this isn't reproducible because the data is locked to those without security clearance. Is there a standard format for this kind of data? If so, consider using it. That way, others can easily create artificial datasets to test it. Even if you have no control over your data source, you can convert the raw data to the standard as a "pre-munging" step. >So, really, my field of public health doesn't use GitHub or sharing much, there's simply too little benefit to the researcher to share their code. Sad but true.

Doctor_Fegg 6y ago

> it is never tested, reproduced, or published This never ceases to amaze me. I regularly read recent papers on shortest-path algorithms. Each one is religiously benchmarked down to the level of saying what C++ compiler was used. But the code itself is almost never published.

txdv 6y ago

Reproducibility is a major principle of the scientific method. Yet computer scientists consistently fail to achieve reproducibility with a tool that is the most consistent at following instructions - the computer. Even private business is on the DevOps movement, because they see the positive effects of reproducibility. If the academic world is truly about science, then there is no more excuse, the tools are out there, they need to use them.

Frost1x 6y ago
This is really an artifact if unreasonable expectations and modern software ecosystems. When I say unreasonable expectations, the issue is that people assume they can use the latest greatest trendy library and get reproducible results. Good luck on the level of determinism you're looking for. You need to step back and look at more mature, simple codebases and what you can do in those sorts of environments when you want reproducibility. You can't cobble together a bunch of async services in the cloud and hope your Frankenstein tool gives you perfect results. It will give you good enough results for certain aspects if you focus on those specific aspects (banking does a good job of this with transactional processing and making sure values are consistent because it's their entire business, maybe your account or their web interface is skrewy but that's fine, that can fail).