7 ms·

Cool so does this mean climate models, with enormous look-ahead windows, are complete garbage?

gjm11 7y ago
Nope. (Sorry. I take it you would like them to be complete garbage.) There are two main ways in which a model can give bad results. 1. It can be a fundamentally bad model. E.g., Kelvin had a perfectly reasonable model of the long-term evolution of the temperature of the earth from which he deduced that it couldn't be more than a few hundred million years old, probably less. The problem was that he didn't know about radioactivity, which turns out to make a tremendous difference. 2. It can be a reasonable model but the parameters can be hard to estimate accurately, and this can lead to inaccurate predictions. In some cases, small errors in the parameters lead to large errors in the predictions. For instance, if you try to forecast the weather a year ahead then you will get all the details wrong. This article is about the second kind of problem, and that's the one that shows the most dramatic degradation as you look at longer and longer intervals. Different properties of the model output can show very different sensitivities to the inputs. For instance, suppose you make a model of the solar system and try to predict where the planets will be ten million years from now. I expect (though I haven't actually done the relevant calculations) you will find that tiny errors in your measurements of the planets' present positions and momenta will lead to very large errors in estimating how far around their orbits the planets are, and probably also in estimating the precession of the orbits. But I bet that almost every simulation run will put the planets' distances from the sun at about the same values, which will be about the same as they are now, and I would bet very heavily that those distances will be a pretty good match for reality. (Barring utterly out-of-model events like the human race getting technologically advanced enough to move planets around and finding it useful to do so.) Or suppose you make a model of the weather and try to predict things five years from now. The model will do very poorly at predicting which days will be rainy in (say) New York, and if you run it many times with tiny changes in its parameters it will give completely different predictions. But it will very consistently tell you that in New York it'll be warmer in July than in January, and it will almost certainly be right; it will do pretty well at predicting rainfall in the Sahara Desert, and it will almost certainly be right about that. Some things behave chaotically, some things not so much. So, what about those climate models? Weather is chaotic: small variations quickly lead to huge changes. Climate is much less so: to whatever extent the models are fundamentally accurate, small changes to the parameters tend to produce small changes to outputs like global mean temperature ten years later. How fundamentally accurate are they? They seem to be pretty decent; when you go back and look at the predictions of old models for what are now past years, they do quite well. (To predict the climate you need to predict human behaviour, because human behaviour affects the climate. Some of the predictions of human behaviour that went into the models were wrong. So you can get a sort of halfway house by running an old model but using now-historical information about things like how much CO2 we would put into the atmosphere. Unsurprisingly, this improves their performance relative to just using the old models as they stand.) See e.g. https://agupubs.onlinelibrary.wiley.com/doi/abs/10.1029/2019GL085378 https://agupubs.onlinelibrary.wiley.com/doi/abs/10.1029/2019... for some analysis. So: no, the sort of effects described in the article don't mean that climate models are complete garbage. The other sort of problem (fundamental errors in the model itself as opposed to its parameters) could make them garbage, but the evidence available so far suggests that even decades-old models were pretty good in this respect.
- zaroth 7y ago
  You wrote so much, it’s a shame you insist on claiming the models are not wrong. The right answer is that, just like COVID models, of course the climate models will turn out to be completely wrong in the exact details, in the scale, in the timeframe, and possibly even in the endpoints. COVID is actually supremely easier to predict than climate. The two fundamental problems with most COVID models are that 1) the positive test counts don’t mean what you think they do, and 2) because of that, everything that is based on positive test counts being relatively meaningful is therefore totally wrong. The climate models, since they were created by humans and not God, will likewise have almost innumerable errors which will cause all sorts of problems both minor and major with their predictions. This is universally true with all models of complex systems, and therefore should be totally uncontroversial, unfortunately the models have become like sacred cows. Personally, just like I was with COVID, I am confident that paradigm-shifting events will intercede that will totally invalidate the model one way or another. For COVID the paradigm shift was that the majority of cases are asymptomatic. Most people are still coming to terms with this. I heard it recently best described to think of SARS-CoV-2 as actually two viruses. One, much less prevalent, is serious and deadly. The other, much more prevalent, is mild to the point of being unremarkable. Getting one confers immunity to both. This is, I stress, a mental model, not a scientific explanation of what’s actually happening (which does not presently exist). The point is that the things that make models wrong are often “external” to the model, the unknowables. For Kelvin it was radioactivity. His model was total “sound” just like the climate models are “sound” in terms of everything that is presently knowable. But complex systems, particularly ones with humans in the loop, have a way of surfacing curveballs the model did not, and could not, anticipate which have outsized effects in the endpoints. The lesson with climate, just as it is being shown with COVID, is that dramatic action to address the problem (“flatten the curve”) with almost incomprehensible direct costs does not pay off. With climate, just like with COVID, a less blunt instrument will achieve a better outcome for orders of magnitude less cost. The sad part is that the top 2% couldn’t give a fuck about the cost either way. It’s the bottom 75% which will suffer from terrible policy.
  - nl 7y ago
    The lesson with climate, just as it is being shown with COVID, is that dramatic action to address the problem (“flatten the curve”) with almost incomprehensible direct costs does not pay off. Surely the lesson from Covid-19 is "early intervention works". Compare for eg the number of deaths in Japan/Australia/South Korea with Italy/US/UK.
2019-nCoV 7y ago
Yes, they are. So what would the correct policy be if we have no reliable model? Well as all existential threats should be treated, with extreme risk aversion.

Nope. (Sorry. I take it you would like them to be complete garbage.) There are two main ways in which a model can give bad results. 1. It can be a fundamentally bad model. E.g., Kelvin had a perfectly reasonable model of the long-term evolution of the temperature of the earth from which he deduced that it couldn't be more than a few hundred million years old, probably less. The problem was that he didn't know about radioactivity, which turns out to make a tremendous difference. 2. It can be a reasonable model but the parameters can be hard to estimate accurately, and this can lead to inaccurate predictions. In some cases, small errors in the parameters lead to large errors in the predictions. For instance, if you try to forecast the weather a year ahead then you will get all the details wrong. This article is about the second kind of problem, and that's the one that shows the most dramatic degradation as you look at longer and longer intervals. Different properties of the model output can show very different sensitivities to the inputs. For instance, suppose you make a model of the solar system and try to predict where the planets will be ten million years from now. I expect (though I haven't actually done the relevant calculations) you will find that tiny errors in your measurements of the planets' present positions and momenta will lead to very large errors in estimating how far around their orbits the planets are, and probably also in estimating the precession of the orbits. But I bet that almost every simulation run will put the planets' distances from the sun at about the same values, which will be about the same as they are now, and I would bet very heavily that those distances will be a pretty good match for reality. (Barring utterly out-of-model events like the human race getting technologically advanced enough to move planets around and finding it useful to do so.) Or suppose you make a model of the weather and try to predict things five years from now. The model will do very poorly at predicting which days will be rainy in (say) New York, and if you run it many times with tiny changes in its parameters it will give completely different predictions. But it will very consistently tell you that in New York it'll be warmer in July than in January, and it will almost certainly be right; it will do pretty well at predicting rainfall in the Sahara Desert, and it will almost certainly be right about that. Some things behave chaotically, some things not so much. So, what about those climate models? Weather is chaotic: small variations quickly lead to huge changes. Climate is much less so: to whatever extent the models are fundamentally accurate, small changes to the parameters tend to produce small changes to outputs like global mean temperature ten years later. How fundamentally accurate are they? They seem to be pretty decent; when you go back and look at the predictions of old models for what are now past years, they do quite well. (To predict the climate you need to predict human behaviour, because human behaviour affects the climate. Some of the predictions of human behaviour that went into the models were wrong. So you can get a sort of halfway house by running an old model but using now-historical information about things like how much CO2 we would put into the atmosphere. Unsurprisingly, this improves their performance relative to just using the old models as they stand.) See e.g. https://agupubs.onlinelibrary.wiley.com/doi/abs/10.1029/2019GL085378 https://agupubs.onlinelibrary.wiley.com/doi/abs/10.1029/2019... for some analysis. So: no, the sort of effects described in the article don't mean that climate models are complete garbage. The other sort of problem (fundamental errors in the model itself as opposed to its parameters) could make them garbage, but the evidence available so far suggests that even decades-old models were pretty good in this respect.

zaroth 7y ago
You wrote so much, it’s a shame you insist on claiming the models are not wrong. The right answer is that, just like COVID models, of course the climate models will turn out to be completely wrong in the exact details, in the scale, in the timeframe, and possibly even in the endpoints. COVID is actually supremely easier to predict than climate. The two fundamental problems with most COVID models are that 1) the positive test counts don’t mean what you think they do, and 2) because of that, everything that is based on positive test counts being relatively meaningful is therefore totally wrong. The climate models, since they were created by humans and not God, will likewise have almost innumerable errors which will cause all sorts of problems both minor and major with their predictions. This is universally true with all models of complex systems, and therefore should be totally uncontroversial, unfortunately the models have become like sacred cows. Personally, just like I was with COVID, I am confident that paradigm-shifting events will intercede that will totally invalidate the model one way or another. For COVID the paradigm shift was that the majority of cases are asymptomatic. Most people are still coming to terms with this. I heard it recently best described to think of SARS-CoV-2 as actually two viruses. One, much less prevalent, is serious and deadly. The other, much more prevalent, is mild to the point of being unremarkable. Getting one confers immunity to both. This is, I stress, a mental model, not a scientific explanation of what’s actually happening (which does not presently exist). The point is that the things that make models wrong are often “external” to the model, the unknowables. For Kelvin it was radioactivity. His model was total “sound” just like the climate models are “sound” in terms of everything that is presently knowable. But complex systems, particularly ones with humans in the loop, have a way of surfacing curveballs the model did not, and could not, anticipate which have outsized effects in the endpoints. The lesson with climate, just as it is being shown with COVID, is that dramatic action to address the problem (“flatten the curve”) with almost incomprehensible direct costs does not pay off. With climate, just like with COVID, a less blunt instrument will achieve a better outcome for orders of magnitude less cost. The sad part is that the top 2% couldn’t give a fuck about the cost either way. It’s the bottom 75% which will suffer from terrible policy.
- nl 7y ago
  The lesson with climate, just as it is being shown with COVID, is that dramatic action to address the problem (“flatten the curve”) with almost incomprehensible direct costs does not pay off. Surely the lesson from Covid-19 is "early intervention works". Compare for eg the number of deaths in Japan/Australia/South Korea with Italy/US/UK.

2019-nCoV 7y ago

Yes, they are. So what would the correct policy be if we have no reliable model? Well as all existential threats should be treated, with extreme risk aversion.