4 ms·

Scientist and programmer here, and my experiences are the opposite. I value keeping things "boringly simple", but I desperately wish there was any kind of engin

Scientist and programmer here, and my experiences are the opposite. I value keeping things "boringly simple", but I desperately wish there was any kind of engineering discipline. First is the reproducibility issue. I think I've spent about as much time simply _trying_ to get the dependencies of research code to run as I have done writing or doing research in my PhD. The simple thing is to write a requirements.txt file! (For Python, at least.) Second, two anecdotes where not following best practices ruined the correctness of research code: - Years ago, I was working on research code which simulated a power-grid. We needed to generate randomized load profiles. I noticed that each time it ran, we got the same results. As a software engineer, I figured I had to re-set the `random` seed, but that didn't work. I dug into the code, talked to the researcher, and found the load-profile algorithm: It was not randomly generated, but a hand-coded string of "1" and "0". - I later had the pleasure of adapting someone's research code. They had essentially hand-engineered IPC. It worked by calling a bash script from Python, which would open other Python processes and generate a random TCP/IP socket, the value of which was saved to an ENV variable. Assuming the socket was open, the Python scripts would then share the socket names of other filenames for the other processes to read and open. To prevent concurrency issues, sleep calls were used throughout the Python and Bash script. This was four Python scripts and two shell scripts, and to this day, I do not understand the reason this wasn't just one Python script.

pennomi 3y ago
Absolutely my experience as well. Scientists write code that works, but is a pain to reproduce in any sort of scalable way. However it’s been getting better over time as programming is becoming a less niche skill.
cauch 3y ago
My problem with this discussion is that a lot of people just say "I'm a scientist (or I'm working with scientists) and I'm observing X so I can say 'scientists blahblahblah'". Different scientific research fields are using widely different computer software environment, and have their own habits and traditions. The way a biologist uses programming has no reason to be similar to the way an astrophysicist does: they have not at all experienced the same software environment. It may even be useless to talk about "scientist" in the same field as two different labs working in the same field may have very different approaches (but it's more difficult if there are shared framework). So, I'm not at all surprised that you observe opposite experience. The same way I'm not surprised to see someone saying they had the opposite experience if someone says "European people are using a lot of 'g' and 'k' in their words" just because they observed what happened in Germany.
- pphysch 3y ago
  I don't think there is much variance in quality of software among (radically different) fields of science. One of the most poorly engineered products I work with was created by a few academic CS guys. The core algorithms are sophisticated and ostensibly implemented well, but the overall product is a horrible mess. The incentives of academia make this obvious. You need to write some code that plausibly works just enough to get a manuscript out of it, but not much else. Reproducibility is not taken that seriously, and "productization"/portability/hardening is out of the question.
  - cauch 3y ago
    That's a strange comment. It looks like the goal of these software are not at all for "productization" and that even if the author wanted to, there will be no point of porting it or hardening it. It feels like Software Developers are "brainwashed": a good software is a software that is good for what software developers need to do. But a good software is a software that is good for what people who needs it need to do. If the academic people don't need "productization" and still code for productzation, then they are doing a bad job. Inversely, software made by software developers may be really bad in academic sector, as it is the subject of the article: they overengineer when not needed, they complicate things just for portability or future use that will never happen, ... It's a bit like if someone says "these cooks who are preparing omelettes are bad cooks: they are mixing white and yolk together, while real cooks who do meringue always remove the yolk. So the proper way to cook eggs is by removing the yolk". (another aspect that I've noticed is that software developers talk about unit test and code review and are offended if they see scientists not using them properly, but they don't even realise that the goal of these is fulfilled in an arguably more efficient way by other processes that exist in academia. For example, there is sometimes no unit test, but the creator of the algo is also the main user and they will notice directly if a new change has broken something. Or as another example, scientific collaboration often implies several teams working independently and writing their own implementation from scratch, so if team A and team B gets different results from the same inputs, they will investigate and find out the problem in the code)
    - 3y ago
BobbyJo 3y ago
The problem I've run into over and over with research code is fragility. We ran it on test A, but when we try test B nothing works and we have no idea why because god forbid there is any error handling, validation, or even just comprehensible function names.

pennomi 3y ago

Absolutely my experience as well. Scientists write code that works, but is a pain to reproduce in any sort of scalable way. However it’s been getting better over time as programming is becoming a less niche skill.

cauch 3y ago

My problem with this discussion is that a lot of people just say "I'm a scientist (or I'm working with scientists) and I'm observing X so I can say 'scientists blahblahblah'". Different scientific research fields are using widely different computer software environment, and have their own habits and traditions. The way a biologist uses programming has no reason to be similar to the way an astrophysicist does: they have not at all experienced the same software environment. It may even be useless to talk about "scientist" in the same field as two different labs working in the same field may have very different approaches (but it's more difficult if there are shared framework). So, I'm not at all surprised that you observe opposite experience. The same way I'm not surprised to see someone saying they had the opposite experience if someone says "European people are using a lot of 'g' and 'k' in their words" just because they observed what happened in Germany.

pphysch 3y ago
I don't think there is much variance in quality of software among (radically different) fields of science. One of the most poorly engineered products I work with was created by a few academic CS guys. The core algorithms are sophisticated and ostensibly implemented well, but the overall product is a horrible mess. The incentives of academia make this obvious. You need to write some code that plausibly works just enough to get a manuscript out of it, but not much else. Reproducibility is not taken that seriously, and "productization"/portability/hardening is out of the question.
- cauch 3y ago
  That's a strange comment. It looks like the goal of these software are not at all for "productization" and that even if the author wanted to, there will be no point of porting it or hardening it. It feels like Software Developers are "brainwashed": a good software is a software that is good for what software developers need to do. But a good software is a software that is good for what people who needs it need to do. If the academic people don't need "productization" and still code for productzation, then they are doing a bad job. Inversely, software made by software developers may be really bad in academic sector, as it is the subject of the article: they overengineer when not needed, they complicate things just for portability or future use that will never happen, ... It's a bit like if someone says "these cooks who are preparing omelettes are bad cooks: they are mixing white and yolk together, while real cooks who do meringue always remove the yolk. So the proper way to cook eggs is by removing the yolk". (another aspect that I've noticed is that software developers talk about unit test and code review and are offended if they see scientists not using them properly, but they don't even realise that the goal of these is fulfilled in an arguably more efficient way by other processes that exist in academia. For example, there is sometimes no unit test, but the creator of the algo is also the main user and they will notice directly if a new change has broken something. Or as another example, scientific collaboration often implies several teams working independently and writing their own implementation from scratch, so if team A and team B gets different results from the same inputs, they will investigate and find out the problem in the code)
  - 3y ago

BobbyJo 3y ago

The problem I've run into over and over with research code is fragility. We ran it on test A, but when we try test B nothing works and we have no idea why because god forbid there is any error handling, validation, or even just comprehensible function names.