6 ms·

Here's the real reason: they are all copying code from each other. Tons of tons of code. Literally all climate global circulation models share a lot of code. I

Here's the real reason: they are all copying code from each other. Tons of tons of code. Literally all climate global circulation models share a lot of code. If the author had actually seen how crap the climate models look, maybe he'll realize the argument is actually backwards and climate models are bad. I've tried code reviewing some of these models just to get an idea of how reliable they are. My mind was blown by the extremely poor quality of the code combined with literally no redundancy between different models in some parts. If some code written in 1990 just made a slight mistake, nearly all climate models would have the same bias. And I did find some distributing surface level inaccuracies, such as constants being taken from very old inaccurate measurements in the 1980s physics books (at least according to the comment), some constants being very low precision compared to the accuracy required (some constants with only 2 decimals of precision), and some things that aren't constant at all and should depend on temperature/pressure/CO2 levels/height. And these are important in a system that actually spends huge efforts to model factors with much weaker impact.

chmaynard 4y ago
> Here's the real reason: they are all copying code from each other. Right, code reuse is dangerous and something to be avoided. From now on, let's write everything from scratch.
- LoganDark 4y ago
  If we reuse code, we risk missing a potential time investment!
- d4mi3n 4y ago
  I don't think this reflects the OP's concern: It's not that code-reuse is bad in and of itself, it's that the code being reused is being reused without maintenance or review.
  - chmaynard 4y ago
    Good point. For example, I'm sure no trained professional would ever copy code from StackOverflow.
    - d4mi3n 4y ago
      This gave me both a good groan and chuckle. You're pointing out a place where I think many software engineers are either untrained, unprofessional, or both. There's already a large body of discourse around the problems that come from these practices—licensing violations, security breaches, and stupendously dumb bugs. Professional rigor is a thing I wish our industry had more of, but in practice you don't see much rigor outside of safety critical applications because there is rarely incentive for it.
- Brentward 4y ago
  I think that's pretty clearly not the point. As someone who has worked with legacy scientific code written in Fortran, the copying is not the issue, it's blindly copying code that was written 50 years ago, probably by a graduate student, and not reviewed or revised in any way since. As the parent comment points out, if everyone reuses that code, the same errors propagate. It's not the type of reuse that comes with a nice GitHub repo and an established mechanism for reporting and fixing bugs. In my case, one of the Fortran files essential to my group's research has 11 (!) versions with only filenames like file.f.goodworking (that's literally one of the extensions) to differentiate them and no version control or history.
kerblang 4y ago
You know more about the subject than I do, but I don't think you're entirely contradicting one of the author's key points: It's incredibly hard to rewrite a system from scratch - and in effect, to just write a system from scratch. But it would make sense to have a "professional scientific fortran programmer" go in and do some housecleaning to make things right, without whining, "Why can't I just rewrite it in Java???" Heck, it sounds like fun to me, but that's because I'm basically a professional janitor, and hey, hurricanes and polar vortexes and all that. So why doesn't that happen? Is testing improvements too expensive? Do scientific programmers just have no respect for the craft whatsoever? Is there nobody with the necessary interdisciplinary skill set?
- hfbff 4y ago
  My experience with academics is that most don't know / don't understand the benefits of coding things in a better way, so they don't even consider hiring someone to help. The other issue is that the incentives aren't necessarily aligned. They don't get payed to share their code or to make their code reusable. The code is used to publish one or a couple of paper and then thrown away. Maybe someone else will reuse it in the future but the original author doesn't necessarily gain much from that
- machina_ex_deus 4y ago
  The way I see it is analogically to writing secure and memory safe code before people knew these things matter. You needed someone with an incentive to go over code, find the bugs, break into computers and for the entire industry to mature from using strcat to having bug bounties and fuzzing and protect zero. It didn't happen over night and there had to be incredible incentives on the attacker / bug finder to exist, to probe and test the parts that aren't tested, and to serve as the devil's advocate. The analogy is that it is hard to find bugs which happen silently. Computation inaccuracies are a huge class of bugs that also happen silently. There are no incentives to look at old code. There's just nothing like it for climate models, and the default behavior of code development is that whatever isn't tested or validated by something will always stagnate with bugs. The code copy just makes everything worse because implementing the same thing in two different ways by different people is actually good at catching bugs. So even if I wouldn't have trusted a single climate model, I might trust several climate models developed independently. The way it works now is that they always prefer to rely on code of others, because academia is driven by citations, and creating new models from scratch means you're missing out mutual citations. It's a systematic problem, the same way security had been a systemic problem: current methodology just doesn't produce code that fits the requirements. Nobody had neither the incentive nor even the idea that these things should work differently. Admitting that this is a systemic problem is just a politicized subject. There are plenty of people saying these models are problematic but getting those working on the models to admit that enough such that the entire system can change its methodology, that won't happen because of human reasons.
bsaul 4y ago
I think the issues you’re mentioning are potentially impacting the world policies (and not a lot of things fall in that category). So if you have some time, it would probably be a great idea to write a detailed post sharing your concerns somewhere.
- BeetleB 4y ago
  A writeup would be nice, but frankly, the issues he's mentioning are "to be expected" for anyone who has spent enough time in academia. They were the same issues I'd see in physics and engineering. I wouldn't expect the climate folks to be any different. Why would they? There's no incentive for them to be.
  - sshine 4y ago
    Right. This appears to be an example of modern software engineering principles, or even the notion of basic automation, not having reached climate researchers. I'm not surprised. Electrical engineers, engineers in general, insurance mathematicians, physicists. They write code. System critical code. Code that puts fundamental research, lives, and trillions of dollars at stake. They're very good at keeping track of the complexity. Not always so good at relaying it. They're sometimes legally required to test. Testing sometimes means doing the same manual thing excessively. That happens. Sometimes not. To this day, I can't believe how many extremely smart people write code for a living, but because of their training, they don't think of themselves as "software developers". So they don't read up on what software developers do to not kill themselves. (Of course, #notall...)
    - BeetleB 4y ago
      Eh - sorry. I worked with both engineering and physics professors and PhD students when I was in grad school. Everything the top comment says about climate science applies equally well to those disciplines (in academia). It's not because climate science folks are less rigorous, or less capable. It's because academia indirectly punishes those who do care about SW quality and correctness.
- 4y ago
mhh__ 4y ago
One thing that is worth keeping in mind is that the end result can end up still being right. Academic Fortran is horrible but it's also (hopefully) maintained by people who know what they're looking for — egregious errors are like errors, things like "this is constant when it shouldn't be" sounds more like a modelling assumption. People got relatively good climate (well, temperature) estimates from the 80s (if you input the right CO2 data for the future), but I'd image the modern ones more subtle predictions could be quite wobbly (i.e. people complain about them having way too many parameters but I'd imagine many of these are for specific "tasks" rather than massively dominating the overall model)
- machina_ex_deus 4y ago
  There are two individual claims, that should be inspected individually: 1. Data from the field showing warming up / other real world observations. I'm not disputing this. This has been corroborated by many different methods without any clear single point of failure. 2. Given 1, what is the probability that the reason for the observed effects are CO2. This is the crucial policy question, and the problem is that the evidence for 2 is actually purely from these models. It's easy to take the conjunction fallacy here and immediately jump to 1&2 being more probable than 1 alone, but it's wrong thinking. The problem is only magnified, in my opinion, by the fact that only few parameters have even had the same kind of detailed analysis that CO2 had. And then the code has clear bias towards the narrative, and the forcing of CO2 in the model is extremely detailed in the parts where the narrative says it's important (radiation forcing) while neglecting many other effects, and taking various things that actually depend on CO2 to be constant. And finally the problem is that the policy decisions are the complete opposite if 2 is wrong: if we're heading towards more chaotic weather and CO2 isn't the culprit, you would want to rely on fossil fuels instead of the weather (a.k.a solar and wind) for energy.
  - mhh__ 4y ago
    The correlation with CO2 output is very very good. The Berkely earth group went in with basically exactly the same question, did everything from scratch and broadly replicated the IPCC curves

chmaynard 4y ago

> Here's the real reason: they are all copying code from each other. Right, code reuse is dangerous and something to be avoided. From now on, let's write everything from scratch.

LoganDark 4y ago
If we reuse code, we risk missing a potential time investment!
d4mi3n 4y ago
I don't think this reflects the OP's concern: It's not that code-reuse is bad in and of itself, it's that the code being reused is being reused without maintenance or review.
- chmaynard 4y ago
  Good point. For example, I'm sure no trained professional would ever copy code from StackOverflow.
  - d4mi3n 4y ago
    This gave me both a good groan and chuckle. You're pointing out a place where I think many software engineers are either untrained, unprofessional, or both. There's already a large body of discourse around the problems that come from these practices—licensing violations, security breaches, and stupendously dumb bugs. Professional rigor is a thing I wish our industry had more of, but in practice you don't see much rigor outside of safety critical applications because there is rarely incentive for it.
Brentward 4y ago
I think that's pretty clearly not the point. As someone who has worked with legacy scientific code written in Fortran, the copying is not the issue, it's blindly copying code that was written 50 years ago, probably by a graduate student, and not reviewed or revised in any way since. As the parent comment points out, if everyone reuses that code, the same errors propagate. It's not the type of reuse that comes with a nice GitHub repo and an established mechanism for reporting and fixing bugs. In my case, one of the Fortran files essential to my group's research has 11 (!) versions with only filenames like file.f.goodworking (that's literally one of the extensions) to differentiate them and no version control or history.

kerblang 4y ago

You know more about the subject than I do, but I don't think you're entirely contradicting one of the author's key points: It's incredibly hard to rewrite a system from scratch - and in effect, to just write a system from scratch. But it would make sense to have a "professional scientific fortran programmer" go in and do some housecleaning to make things right, without whining, "Why can't I just rewrite it in Java???" Heck, it sounds like fun to me, but that's because I'm basically a professional janitor, and hey, hurricanes and polar vortexes and all that. So why doesn't that happen? Is testing improvements too expensive? Do scientific programmers just have no respect for the craft whatsoever? Is there nobody with the necessary interdisciplinary skill set?

hfbff 4y ago
My experience with academics is that most don't know / don't understand the benefits of coding things in a better way, so they don't even consider hiring someone to help. The other issue is that the incentives aren't necessarily aligned. They don't get payed to share their code or to make their code reusable. The code is used to publish one or a couple of paper and then thrown away. Maybe someone else will reuse it in the future but the original author doesn't necessarily gain much from that
machina_ex_deus 4y ago
The way I see it is analogically to writing secure and memory safe code before people knew these things matter. You needed someone with an incentive to go over code, find the bugs, break into computers and for the entire industry to mature from using strcat to having bug bounties and fuzzing and protect zero. It didn't happen over night and there had to be incredible incentives on the attacker / bug finder to exist, to probe and test the parts that aren't tested, and to serve as the devil's advocate. The analogy is that it is hard to find bugs which happen silently. Computation inaccuracies are a huge class of bugs that also happen silently. There are no incentives to look at old code. There's just nothing like it for climate models, and the default behavior of code development is that whatever isn't tested or validated by something will always stagnate with bugs. The code copy just makes everything worse because implementing the same thing in two different ways by different people is actually good at catching bugs. So even if I wouldn't have trusted a single climate model, I might trust several climate models developed independently. The way it works now is that they always prefer to rely on code of others, because academia is driven by citations, and creating new models from scratch means you're missing out mutual citations. It's a systematic problem, the same way security had been a systemic problem: current methodology just doesn't produce code that fits the requirements. Nobody had neither the incentive nor even the idea that these things should work differently. Admitting that this is a systemic problem is just a politicized subject. There are plenty of people saying these models are problematic but getting those working on the models to admit that enough such that the entire system can change its methodology, that won't happen because of human reasons.

bsaul 4y ago

I think the issues you’re mentioning are potentially impacting the world policies (and not a lot of things fall in that category). So if you have some time, it would probably be a great idea to write a detailed post sharing your concerns somewhere.

BeetleB 4y ago
A writeup would be nice, but frankly, the issues he's mentioning are "to be expected" for anyone who has spent enough time in academia. They were the same issues I'd see in physics and engineering. I wouldn't expect the climate folks to be any different. Why would they? There's no incentive for them to be.
- sshine 4y ago
  Right. This appears to be an example of modern software engineering principles, or even the notion of basic automation, not having reached climate researchers. I'm not surprised. Electrical engineers, engineers in general, insurance mathematicians, physicists. They write code. System critical code. Code that puts fundamental research, lives, and trillions of dollars at stake. They're very good at keeping track of the complexity. Not always so good at relaying it. They're sometimes legally required to test. Testing sometimes means doing the same manual thing excessively. That happens. Sometimes not. To this day, I can't believe how many extremely smart people write code for a living, but because of their training, they don't think of themselves as "software developers". So they don't read up on what software developers do to not kill themselves. (Of course, #notall...)
  - BeetleB 4y ago
    Eh - sorry. I worked with both engineering and physics professors and PhD students when I was in grad school. Everything the top comment says about climate science applies equally well to those disciplines (in academia). It's not because climate science folks are less rigorous, or less capable. It's because academia indirectly punishes those who do care about SW quality and correctness.
4y ago

mhh__ 4y ago

One thing that is worth keeping in mind is that the end result can end up still being right. Academic Fortran is horrible but it's also (hopefully) maintained by people who know what they're looking for — egregious errors are like errors, things like "this is constant when it shouldn't be" sounds more like a modelling assumption. People got relatively good climate (well, temperature) estimates from the 80s (if you input the right CO2 data for the future), but I'd image the modern ones more subtle predictions could be quite wobbly (i.e. people complain about them having way too many parameters but I'd imagine many of these are for specific "tasks" rather than massively dominating the overall model)

machina_ex_deus 4y ago
There are two individual claims, that should be inspected individually: 1. Data from the field showing warming up / other real world observations. I'm not disputing this. This has been corroborated by many different methods without any clear single point of failure. 2. Given 1, what is the probability that the reason for the observed effects are CO2. This is the crucial policy question, and the problem is that the evidence for 2 is actually purely from these models. It's easy to take the conjunction fallacy here and immediately jump to 1&2 being more probable than 1 alone, but it's wrong thinking. The problem is only magnified, in my opinion, by the fact that only few parameters have even had the same kind of detailed analysis that CO2 had. And then the code has clear bias towards the narrative, and the forcing of CO2 in the model is extremely detailed in the parts where the narrative says it's important (radiation forcing) while neglecting many other effects, and taking various things that actually depend on CO2 to be constant. And finally the problem is that the policy decisions are the complete opposite if 2 is wrong: if we're heading towards more chaotic weather and CO2 isn't the culprit, you would want to rely on fossil fuels instead of the weather (a.k.a solar and wind) for energy.
- mhh__ 4y ago
  The correlation with CO2 output is very very good. The Berkely earth group went in with basically exactly the same question, did everything from scratch and broadly replicated the IPCC curves