5 ms·

From my time in academia, there are two reasons: 1. It's fast. 2. It's tolerant of poor programmers. Regarding item 2: People who want to specialize in scien

From my time in academia, there are two reasons: 1. It's fast. 2. It's tolerant of poor programmers. Regarding item 2: People who want to specialize in scientific computing intentionally avoid becoming good programmers. Most of them are not in CS, but in some other engineering or in science. They want to focus on their discipline (e.g. climate science as this example gives). They don't care about code quality, or even SW practices (e.g. when I was in grad school a little over a decade ago, I couldn't get a single one of my colleagues to use version control - and some of them had codebases in the tens of thousands of lines). They are not judged by the quality of their code, but by the results their code produces. They don't show their code in their papers, but the plots/tables. So they'll use the language that puts up the least barriers, and is performant. Hence, Fortran. (Incidentally, many of them don't care about reproducibility. They'll modify their large codebase, spit out some results, publish them, and then add new features, and will never check if the modified codebase will continue to give the same results for the same inputs). All the other reasons in the article? Insignificant factors. Edit: Note that I'm not saying that if you code in Fortran you're a poor programmer or use poor practices. And I'm sure there are good Fortran code bases around.

TillE 4y ago
I was toying with the idea of getting into computational chemistry about 15 years ago, as a rare person with a strong background in both CS and chemistry. The main reason I didn't was because grad school didn't sound like much fun, but the prospect of working with a bunch of lousy code wasn't encouraging either.
mrazomor 4y ago
> They are not judged by the quality of their code, but by the results their code produces. How much are software engineers judged by the code quality? And how much by the end results (shipped features/projects)? There's a large gap between good code quality (i.e. keep the soft. engs. happy) and passable maintenance (i.e. features and bugfixes are not terribly late -- keep the execs/business happy). When it comes to Fortran, I wish it long life. It found a great fit and does its job well. Numerical math has many peculiarities. I'm happy we have stable libraries for it. Long live Fortran!
- BeetleB 4y ago
  > How much are software engineers judged by the code quality? And how much by the end results (shipped features/projects)? Many differences: 1. Everyone can tell if the SW is doing what it should. Ensuring correctness in the long run can lead to seeing gains in maintaining code quality. With a lot of scientific output, many people will just trust the output of the program. Crappy code quality is not easy to perceive. 2. In many code bases, backwards compatibility is important. Hopefully you'll have many users. With most scientific computation code bases, the user base is tiny (often 1). So no one cares if you break something when you add a feature. Couple it with bullet 1 above, no one even notices if you broke something. With regular SW projects, the need for backwards compatibility can once again drive good SW practices. I'm not saying all Fortran programmers are bad. Just that it is very tolerant of bad ones, and so they are highly overrepresented. > I'm happy we have stable libraries for it. As someone who couldn't stomach Fortran, as far as numerical work goes, plenty of other languages can utilize those same libraries. Having stable numerical libraries is not unique to Fortran. In fact, a colleague wasted way too much time during his PhD taking his group's 40K Fortran code base and porting it to C++.
  - robocat 4y ago
    > Everyone can tell if the SW is doing what it should I am not sure that would be a majority opinion. Robocat’s Law: software complexity increases until it can’t be understood by anybody. > Fortran code base and porting it to C++ I once tried to counsel a PhD friend to use Mathematica to model their engineering problem, however instead she listened to other people in the civil engineering department to use C++, and she spent most of her time learning C++ and not enough time on her models.
    - lmm 4y ago
      > I am not sure that would be a majority opinion. Robocat’s Law: software complexity increases until it can’t be understood by anybody. Plenty of banks don't understand how their software works. But they can see that at the end of the day it calculates the right amount of money. (If it doesn't, eventually the client complains, or eventually your manager complains, depending on which direction it was wrong in). For a lot of these scientific modelling systems there's no such sanity check.
jononomo 4y ago
Doesn't "tolerant of poor programmers" just mean "even better for strong programmers"?
- sidlls 4y ago
  And usual it depends on the context. “Strong programmer” practices don’t actually bring much to the table with respect to scientific computing. Regression testing (alluded to above), for example, is actually (mostly) implicit, as future results depend on the continued stable output of past results. If the new code caused the old code to produce different outputs for the new result, it would show somewhere. At least it did when I was doing computational astrophysics in academia.
  - BeetleB 4y ago
    > If the new code caused the old code to produce different outputs for the new result, it would show somewhere. At least it did when I was doing computational astrophysics in academia. For libraries others use, sure. For mere code bases that stay in one professor's team and have been there for 20+ years - not really. As an example, I know one computational group where it often took several hours (possibly days) to generate a single data point in the curve they were trying to produce. None of that team's grad students was willing to waste computational resources to see if their code changes would break prior data points.
- BeetleB 4y ago
  > Doesn't "tolerant of poor programmers" just mean "even better for strong programmers"? Not sure. BASIC is very tolerant of poor programmers as well.
- lmm 4y ago
  No. Do you prefer to consider all failure conditions upfront, or only as and when they happen? When two things are similar but not exactly the same, how much do you want to distinguish between them? Programming language design picks positions on these axes, and the answers for poor programmers and strong programmers tend to be different.
RosanaAnaDana 4y ago
This is effectively the same argument for why R is such a powerful language. It's the fastest route from theory to implementation for many scientists.
- Max-Limelihood 4y ago
  Fastest until you include the runtime. (And even then, it falls pretty far behind both Python and Julia in terms of ease-of-use.) The main advantage of R is there's a package for everything, because of how old it is. But it's not especially easy to read or learn. (Or especially hard either--it's definitely easier than most languages, it's just no longer on the cutting edge of that.)
- taeric 4y ago
  Even more so, as it saves them a ton of debates with programmers. :D
akira2501 4y ago
It's maybe the same reason a lot of mechanics drive junk cars. Their expertise blinds them to the risk they're accepting by operating so close to the margin. It's also maybe the reason code is ever written professionally. The coders often don't have this domain expertise for their particular product and add in things like unit tests, data reproduction tests and insane amounts of logging in order to reduce the amount of risk inherent in the code itself.
- debatem1 4y ago
  Are you a mechanic or similar yourself? It seems very bold to me to assume that someone who spends their professional life around cars is "blind" to the reality of the cars they drive, when a much more likely explanation is that experts can more accurately evaluate things in their area of expertise than laypeople.
  - akira2501 4y ago
    No, of course not. They're not that hard to find; though, and they're pretty willing to talk about their trade. I didn't say blind to the reality, but blind to the risk specifically. Are you suggesting that there isn't additional risk to driving a car that's older and has known mechanical issues? The expert assumes they can fix them as they break, which is probably true, but they often don't consider the myriad circumstances they can break in and the additional problems that can impose. This isn't an insult to mechanics, but just an observation of the behavior and of the outcomes. Or are you suggesting experts are impervious to analysis?
    - debatem1 4y ago
      > I didn't say blind to the reality, but blind to the risk specifically. Are you referring to some non-real type of risk, then? I think you may be, and given that this is a subject outside of your wheelhouse you may want to consider that possibility as well. > Are you suggesting that there isn't additional risk to driving a car that's older and has known mechanical issues? I'm suggesting that experts may be able to both predict and price those risks more accurately than non-experts. This should be non-controversial. Consequently, cars which look to you as a nonexpert highly risky may not be. If that is true, as seems likely to me, the rest of your analogy falls apart. > Or are you suggesting experts are impervious to analysis? I'm suggesting that more likely explanations than "the experts are wrong and I am right" exist.
msla 4y ago
> They don't show their code in their papers, but the plots/tables. Which is a reproducibility crisis in and of itself, as you mentioned, if not an outright inducement to dishonesty.
- taeric 4y ago
  Strictly, not really. Consider, we rarely (ever?) see the actual needles that go into taking samples. Or the conditions in the polling place where surveys were given/taken. Worse, reproducibility in the "if you click go on this project, you will see it gives the answers I say it does," is of dubious value. Rather, the reproduction is supposed to be "use your models and change assumptions in such a way, you should see these updates." We are at a time where exact reproduction feels possible, but that is typically not as useful as you'd think. A lot of the robustness of old experiments comes from building them all over again.
- TrackerFF 4y ago
  Tbh, you'd need a lot more than just the code to reproduce results today. Last year I spent a couple of weeks getting some academic code to work, due to dependency issues. Rolling back drivers, rolling back libraries, trying specific combinations of drivers and library versions, and what have you. How fast the code ran also depended on the various combinations. The code was not ancient, but written 12 months prior to me trying to build it. (Deep Learning project)
  - adgjlsfhk1 4y ago
    This is one of the places that IMO Julia really shines. It uses a combination of a Project.toml to track dependencies and a Manifest.toml that tracks all the versions of the entire dependency tree which makes it trivial to install all of the package versions.
    - msla 4y ago
      Even that isn't sufficient for archival purposes, which an academic journal should be aiming for.
karencarits 4y ago
R is also very popular among scientists, and I think that workflowr, Rmarkdown, and Quarto are going to increase transparency and code sharing among scientists. As you mentioned, most researchers just want things to work so that they can focus on their results, and integrating processing and writing is a good way to do that
analog31 4y ago
I'm a scientist, and a programmer, but I don't code in Fortran. I've noticed a "brain drain" of programmers out of science and traditional engineering (mechanical, electrical, etc), simply due to the disparities in pay and mobility. Anybody who's good enough at coding to do it for money is at risk of bailing out of science altogether. The best way to convince someone to learn SW practices is by giving them the hope of finding a better paying job. I think there's actually a shortage of good training for scientific programmers, which probably has to come from people who are immersed in it. On the other hand, is the grass really greener on the other side? One can read plenty of blogs and forum posts from programmers on the commercial side, complaining about the quality of code and coding practices, that they have to work on. I've read about banks running on COBOL and hacked together Excel workbooks.
- lapinot 4y ago
  > Anybody who's good enough at coding to do it for money is at risk of bailing out of science altogether. No. Tons of people in academia are in academia because they believe in free knowledge and because they want to work as public servants outside the realm of for-profit economy. Most of us tolerate being payed little when working conditions are ok and there is freedom. The reason there are so few good programmers (or that they are in small pockets, or that most good software projects from academia are actually non-research side projects), is because software dev and maintenance it is not at all incentivized, chronically underfunded and understaffed. Short-term project-based financing structure makes any sustained software development hazardous. OTOH lets put the ugly project web-pages aside and browse through our distribution's package repository for foundational software. How many of them have been passed from hand to hand through generations of academic workers? Quite a lot.. The problem is not that "we are not good at it" or anything. It's just that nobody's giving a shit, so people do it in their "free time" and so the landscape looks a lot like free software. In fact there is a big intersection.
  - analog31 4y ago
    That's certainly fair, thanks for the comment.
bluepoint 4y ago
Spot on. This certainly applies to much of physics academia at least.

TillE 4y ago

I was toying with the idea of getting into computational chemistry about 15 years ago, as a rare person with a strong background in both CS and chemistry. The main reason I didn't was because grad school didn't sound like much fun, but the prospect of working with a bunch of lousy code wasn't encouraging either.

mrazomor 4y ago

> They are not judged by the quality of their code, but by the results their code produces. How much are software engineers judged by the code quality? And how much by the end results (shipped features/projects)? There's a large gap between good code quality (i.e. keep the soft. engs. happy) and passable maintenance (i.e. features and bugfixes are not terribly late -- keep the execs/business happy). When it comes to Fortran, I wish it long life. It found a great fit and does its job well. Numerical math has many peculiarities. I'm happy we have stable libraries for it. Long live Fortran!

BeetleB 4y ago
> How much are software engineers judged by the code quality? And how much by the end results (shipped features/projects)? Many differences: 1. Everyone can tell if the SW is doing what it should. Ensuring correctness in the long run can lead to seeing gains in maintaining code quality. With a lot of scientific output, many people will just trust the output of the program. Crappy code quality is not easy to perceive. 2. In many code bases, backwards compatibility is important. Hopefully you'll have many users. With most scientific computation code bases, the user base is tiny (often 1). So no one cares if you break something when you add a feature. Couple it with bullet 1 above, no one even notices if you broke something. With regular SW projects, the need for backwards compatibility can once again drive good SW practices. I'm not saying all Fortran programmers are bad. Just that it is very tolerant of bad ones, and so they are highly overrepresented. > I'm happy we have stable libraries for it. As someone who couldn't stomach Fortran, as far as numerical work goes, plenty of other languages can utilize those same libraries. Having stable numerical libraries is not unique to Fortran. In fact, a colleague wasted way too much time during his PhD taking his group's 40K Fortran code base and porting it to C++.
- robocat 4y ago
  > Everyone can tell if the SW is doing what it should I am not sure that would be a majority opinion. Robocat’s Law: software complexity increases until it can’t be understood by anybody. > Fortran code base and porting it to C++ I once tried to counsel a PhD friend to use Mathematica to model their engineering problem, however instead she listened to other people in the civil engineering department to use C++, and she spent most of her time learning C++ and not enough time on her models.
  - lmm 4y ago
    > I am not sure that would be a majority opinion. Robocat’s Law: software complexity increases until it can’t be understood by anybody. Plenty of banks don't understand how their software works. But they can see that at the end of the day it calculates the right amount of money. (If it doesn't, eventually the client complains, or eventually your manager complains, depending on which direction it was wrong in). For a lot of these scientific modelling systems there's no such sanity check.

jononomo 4y ago

Doesn't "tolerant of poor programmers" just mean "even better for strong programmers"?

sidlls 4y ago
And usual it depends on the context. “Strong programmer” practices don’t actually bring much to the table with respect to scientific computing. Regression testing (alluded to above), for example, is actually (mostly) implicit, as future results depend on the continued stable output of past results. If the new code caused the old code to produce different outputs for the new result, it would show somewhere. At least it did when I was doing computational astrophysics in academia.
- BeetleB 4y ago
  > If the new code caused the old code to produce different outputs for the new result, it would show somewhere. At least it did when I was doing computational astrophysics in academia. For libraries others use, sure. For mere code bases that stay in one professor's team and have been there for 20+ years - not really. As an example, I know one computational group where it often took several hours (possibly days) to generate a single data point in the curve they were trying to produce. None of that team's grad students was willing to waste computational resources to see if their code changes would break prior data points.
BeetleB 4y ago
> Doesn't "tolerant of poor programmers" just mean "even better for strong programmers"? Not sure. BASIC is very tolerant of poor programmers as well.
lmm 4y ago
No. Do you prefer to consider all failure conditions upfront, or only as and when they happen? When two things are similar but not exactly the same, how much do you want to distinguish between them? Programming language design picks positions on these axes, and the answers for poor programmers and strong programmers tend to be different.

RosanaAnaDana 4y ago

This is effectively the same argument for why R is such a powerful language. It's the fastest route from theory to implementation for many scientists.

Max-Limelihood 4y ago
Fastest until you include the runtime. (And even then, it falls pretty far behind both Python and Julia in terms of ease-of-use.) The main advantage of R is there's a package for everything, because of how old it is. But it's not especially easy to read or learn. (Or especially hard either--it's definitely easier than most languages, it's just no longer on the cutting edge of that.)
taeric 4y ago
Even more so, as it saves them a ton of debates with programmers. :D

akira2501 4y ago

It's maybe the same reason a lot of mechanics drive junk cars. Their expertise blinds them to the risk they're accepting by operating so close to the margin. It's also maybe the reason code is ever written professionally. The coders often don't have this domain expertise for their particular product and add in things like unit tests, data reproduction tests and insane amounts of logging in order to reduce the amount of risk inherent in the code itself.

debatem1 4y ago
Are you a mechanic or similar yourself? It seems very bold to me to assume that someone who spends their professional life around cars is "blind" to the reality of the cars they drive, when a much more likely explanation is that experts can more accurately evaluate things in their area of expertise than laypeople.
- akira2501 4y ago
  No, of course not. They're not that hard to find; though, and they're pretty willing to talk about their trade. I didn't say blind to the reality, but blind to the risk specifically. Are you suggesting that there isn't additional risk to driving a car that's older and has known mechanical issues? The expert assumes they can fix them as they break, which is probably true, but they often don't consider the myriad circumstances they can break in and the additional problems that can impose. This isn't an insult to mechanics, but just an observation of the behavior and of the outcomes. Or are you suggesting experts are impervious to analysis?
  - debatem1 4y ago
    > I didn't say blind to the reality, but blind to the risk specifically. Are you referring to some non-real type of risk, then? I think you may be, and given that this is a subject outside of your wheelhouse you may want to consider that possibility as well. > Are you suggesting that there isn't additional risk to driving a car that's older and has known mechanical issues? I'm suggesting that experts may be able to both predict and price those risks more accurately than non-experts. This should be non-controversial. Consequently, cars which look to you as a nonexpert highly risky may not be. If that is true, as seems likely to me, the rest of your analogy falls apart. > Or are you suggesting experts are impervious to analysis? I'm suggesting that more likely explanations than "the experts are wrong and I am right" exist.

msla 4y ago

> They don't show their code in their papers, but the plots/tables. Which is a reproducibility crisis in and of itself, as you mentioned, if not an outright inducement to dishonesty.

taeric 4y ago
Strictly, not really. Consider, we rarely (ever?) see the actual needles that go into taking samples. Or the conditions in the polling place where surveys were given/taken. Worse, reproducibility in the "if you click go on this project, you will see it gives the answers I say it does," is of dubious value. Rather, the reproduction is supposed to be "use your models and change assumptions in such a way, you should see these updates." We are at a time where exact reproduction feels possible, but that is typically not as useful as you'd think. A lot of the robustness of old experiments comes from building them all over again.
TrackerFF 4y ago
Tbh, you'd need a lot more than just the code to reproduce results today. Last year I spent a couple of weeks getting some academic code to work, due to dependency issues. Rolling back drivers, rolling back libraries, trying specific combinations of drivers and library versions, and what have you. How fast the code ran also depended on the various combinations. The code was not ancient, but written 12 months prior to me trying to build it. (Deep Learning project)
- adgjlsfhk1 4y ago
  This is one of the places that IMO Julia really shines. It uses a combination of a Project.toml to track dependencies and a Manifest.toml that tracks all the versions of the entire dependency tree which makes it trivial to install all of the package versions.
  - msla 4y ago
    Even that isn't sufficient for archival purposes, which an academic journal should be aiming for.

karencarits 4y ago

R is also very popular among scientists, and I think that workflowr, Rmarkdown, and Quarto are going to increase transparency and code sharing among scientists. As you mentioned, most researchers just want things to work so that they can focus on their results, and integrating processing and writing is a good way to do that

analog31 4y ago

I'm a scientist, and a programmer, but I don't code in Fortran. I've noticed a "brain drain" of programmers out of science and traditional engineering (mechanical, electrical, etc), simply due to the disparities in pay and mobility. Anybody who's good enough at coding to do it for money is at risk of bailing out of science altogether. The best way to convince someone to learn SW practices is by giving them the hope of finding a better paying job. I think there's actually a shortage of good training for scientific programmers, which probably has to come from people who are immersed in it. On the other hand, is the grass really greener on the other side? One can read plenty of blogs and forum posts from programmers on the commercial side, complaining about the quality of code and coding practices, that they have to work on. I've read about banks running on COBOL and hacked together Excel workbooks.

lapinot 4y ago
> Anybody who's good enough at coding to do it for money is at risk of bailing out of science altogether. No. Tons of people in academia are in academia because they believe in free knowledge and because they want to work as public servants outside the realm of for-profit economy. Most of us tolerate being payed little when working conditions are ok and there is freedom. The reason there are so few good programmers (or that they are in small pockets, or that most good software projects from academia are actually non-research side projects), is because software dev and maintenance it is not at all incentivized, chronically underfunded and understaffed. Short-term project-based financing structure makes any sustained software development hazardous. OTOH lets put the ugly project web-pages aside and browse through our distribution's package repository for foundational software. How many of them have been passed from hand to hand through generations of academic workers? Quite a lot.. The problem is not that "we are not good at it" or anything. It's just that nobody's giving a shit, so people do it in their "free time" and so the landscape looks a lot like free software. In fact there is a big intersection.
- analog31 4y ago
  That's certainly fair, thanks for the comment.

bluepoint 4y ago

Spot on. This certainly applies to much of physics academia at least.