3 ms·

I'm an ex-software developer/engineer and current scientist. In my experience TFA makes a good point, even though it's quite strawmanish. Most scientific code

I'm an ex-software developer/engineer and current scientist. In my experience TFA makes a good point, even though it's quite strawmanish. Most scientific code is horrible from any sane software developer perspective. The quality is so bad that I think a huge proportion of published results are plain wrong due to bugs in the analysis. These apply to my much of my code as well. But a lot of "software engineering" code is horrible too. Mostly because most popular technologies and "best practices" are just plain bad. Overenginering is a pandemic and has been a long time. Much of the roots is from the gilded age of enterprise Java. Totally misunderstood OOP. Byzantine layers of pointless abstraction. Rampant premature "web scale". Counterproductive bondage and discipline (yes, including much of static typing). These have become a cargo cult in software development, and these cosmetic features are deemed as "quality code". And this leaks into scientific programming. Even Python, arguably the language of science nowadays, and a "quick and dirty one", forces some of this cargo cult. Modules are needlessly complicated (e.g. relative and absolute imports are quite a mess), let alone the horrible packaging system. And the current trend to push typing. That said, scientific code is getting slowly better, largely due to switch to public/open source code, and away from specialized hacks like MATLAB and R. Especially in more technical fields. In software engineering OTOH things are IMHO getting worse.

collyw 3y ago
I'll add that "code reuse" was pushed as an idea too heavily (at my university at least). I think that leads to a fair bit of the over engineering and unnecessary complexity. I saw a comment later saying not to aim for "code reuse", but rather to "avoid code duplication" which made more sense. I also agree that a lot of best practices are great if you are a massive company with unlimited resources, but they are just overkill and add complexity for smaller projects and teams.
- jampekka 3y ago
  Code reuse was indeed the hype, especially of OOP and inheritance. In practice it just made real code reuse a lot worse. The class hierarchies become so tightly coupled that any "reuse" requires a lot more boilerplate than the actual code to use. I know that the idea is not just not writing more code, but have a "single point of truth". But the boilerplate leads to a situation where it's very inpractical and leads to a tightly coupled mess that's really hard to change. What actually increased code reuse was duck typing (i.e. implicit interfaces), made popular by Python. But that's becoming verboten nowadays. And I don't think current structural typing systems are gonna reach the same level of reuse.
sseagull 3y ago
I basically agree (except I will take static typing over dynamic typing any day). I work in computational chemistry, and scientists here don’t necessarily have problems actually coding (we’ve been doing it for 70+ years). But the “other stuff” is taking more and more time. Before, you wrote Fortran, put the files on disks or whatever, and sent them around. Now, you need to know: C++ and Python (and maybe Fortran too) git and github and github actions, and packaging (pypi/conda/conda-forge I guess is the standard now). Documentation? Sphinx I guess, although mixed codebases are still a pain. And where do I host the docs again? Also better make sure it works on all three OSs. Wait what is docker? Suppose I better learn make images. And they have to be hosted somewhere? Got to set that up too. Oh wait again, Python 3.12 broke something. Have to fix that. Does all of this make “better software”?. I’m coming around to the idea that maybe it doesn’t, at least in aggregate. Either way, scientists don’t have time to really learn all this AND the science they are doing. And it all changes every few years or so.
- dillydogg 3y ago
  > Either way, scientists don’t have time to really learn all this AND the science they are doing. And it all changes every few years or so. This is how I see it as well. I'm an immunologist, and it feels impossible for me to keep on top of my field of research and just about anything else. I don't have to produce quality software, but it seems difficult to keep your research cutting edge while maintaining software. It's hard enough keeping the research up to date!
- jampekka 3y ago
  > (except I will take static typing over dynamic typing any day) Why? What exact typesystem do you prefer over Python's dynamic typing? A lot of its idioms, and probably e.g. NumPy/SciPy infrastructure would be about impossible with current static typesystems (look at the mess that is C++ scientific/ndarray libraries). Ditto for much of the autodiff and GPU stuff like pytorch. Julia could perhaps get there, but the implementation has too many warts for it to take over.
  - sseagull 3y ago

collyw 3y ago

I'll add that "code reuse" was pushed as an idea too heavily (at my university at least). I think that leads to a fair bit of the over engineering and unnecessary complexity. I saw a comment later saying not to aim for "code reuse", but rather to "avoid code duplication" which made more sense. I also agree that a lot of best practices are great if you are a massive company with unlimited resources, but they are just overkill and add complexity for smaller projects and teams.

jampekka 3y ago
Code reuse was indeed the hype, especially of OOP and inheritance. In practice it just made real code reuse a lot worse. The class hierarchies become so tightly coupled that any "reuse" requires a lot more boilerplate than the actual code to use. I know that the idea is not just not writing more code, but have a "single point of truth". But the boilerplate leads to a situation where it's very inpractical and leads to a tightly coupled mess that's really hard to change. What actually increased code reuse was duck typing (i.e. implicit interfaces), made popular by Python. But that's becoming verboten nowadays. And I don't think current structural typing systems are gonna reach the same level of reuse.

sseagull 3y ago

I basically agree (except I will take static typing over dynamic typing any day). I work in computational chemistry, and scientists here don’t necessarily have problems actually coding (we’ve been doing it for 70+ years). But the “other stuff” is taking more and more time. Before, you wrote Fortran, put the files on disks or whatever, and sent them around. Now, you need to know: C++ and Python (and maybe Fortran too) git and github and github actions, and packaging (pypi/conda/conda-forge I guess is the standard now). Documentation? Sphinx I guess, although mixed codebases are still a pain. And where do I host the docs again? Also better make sure it works on all three OSs. Wait what is docker? Suppose I better learn make images. And they have to be hosted somewhere? Got to set that up too. Oh wait again, Python 3.12 broke something. Have to fix that. Does all of this make “better software”?. I’m coming around to the idea that maybe it doesn’t, at least in aggregate. Either way, scientists don’t have time to really learn all this AND the science they are doing. And it all changes every few years or so.

dillydogg 3y ago
> Either way, scientists don’t have time to really learn all this AND the science they are doing. And it all changes every few years or so. This is how I see it as well. I'm an immunologist, and it feels impossible for me to keep on top of my field of research and just about anything else. I don't have to produce quality software, but it seems difficult to keep your research cutting edge while maintaining software. It's hard enough keeping the research up to date!
jampekka 3y ago
> (except I will take static typing over dynamic typing any day) Why? What exact typesystem do you prefer over Python's dynamic typing? A lot of its idioms, and probably e.g. NumPy/SciPy infrastructure would be about impossible with current static typesystems (look at the mess that is C++ scientific/ndarray libraries). Ditto for much of the autodiff and GPU stuff like pytorch. Julia could perhaps get there, but the implementation has too many warts for it to take over.
- sseagull 3y ago