4 ms·

How robust will ML models be in the world of rapid climate change, where the past no longer predicts the future?

zdragnar 2y ago
Climate is not weather. Weather predictions are for specific events and areas, made on the order of days- typically no more than 2-3 weeks into the future. Climate models predict future averages over large regions on the order of decades. "rapid climate change" is on the order of "within this century". Whether the climate changes or not doesn't really impact the weather models at all, because the climate's not changing on the same time scale.
- labster 2y ago
  While I really want to agree with your sound argument, I have this suspicion that climate change significantly affected forecast skill on Hurricane Otis.
  - melagonster 2y ago
    Climate change: more typhoons in the future. Weather prediction: which path typhoons will follow?
  - zdragnar 2y ago
    The climate didn't change in the time from Otis forming to landing. Weather models look at current conditions and extrapolate what the next set of conditions will be. Another way to look at it is that climate change models reflect what we will experience. Weather models reflect the mechanics of humidity, temperature and air flow- given a current state, what is the next likely state. Climate change doesn't change how those mechanics interact. It only guesses what "current state" will look like in the far future.
    - avianlyric 2y ago
      Climate change doesn’t impact the physics of weather, but our understanding of the inputs to a weather system is still very incomplete. Climate change does impact the relative importance of different physical inputs into a weather system, and inherently pushes weather systems away from the most studied and well understood aspects of weather physics. That in turn degrades the accuracy of our models, as it takes time to research and understand whats changed in a weather system, and what previously unimportant inputs have become much more important.
      - zdragnar 2y ago
        Let's say I live in the hypothetical state of Midwestia. It gets up to 100 degrees F in the summer and down to -40 in the winter months. Over the course of ten years, the average daily temperature goes up by 1 degree F (which is many times faster than reality). That means that there's basically one day in the entire year that doesn't fit within the existing model, or maybe a handful if some other part of the year is cooler than normal. We see more variation between normal years than we expect to see average increase in both the short and medium terms. Our weather models predict no more than two or three weeks into the future. There's literally nothing about climate change that will change how weather systems interact that will occur faster than our weather models will adapt to. Our weather patterns may change drastically, but that's a fundamentally different problem.
        avianlyric 2y ago
        How quickly do you think our weather models adapt? Our current models are built on top of decades of detailed weather research by tens of thousands of researchers around the world, and has involved the launch of some the largest satellites in space. All to help us try and understand how weather works. Those models don't just "adapt", they're carefully and manually refined over time, slowly incorporating better physical models as they're developed, and better methods of collecting data as it become available. Climate change absolutely changes how weather systems interact faster than our models can adapt, primarily because we can only adapt models to changes after their accuracy degrades, and it becomes possible to start identifying potential weaknesses in the existing approaches. You're example of seasonal temperature changes vs global average temperature increases, vastly underestimates how complicated weather models are, and how much impact changes in global climate systems (like the North Atlantic Current) can drastically impact local weather conditions. Weather models, like all models, will make assumptions about the behaviours of extremely large systems which aren't expected to change very much, or are very hard to measure in real-time. As those large systems change due to climate change, the assumptions made by weather models will grow increasingly incorrect. But that hard part is figuring our which assumption is now incorrect, not always easy to identity exactly what assumptions have been made, or accurately measure the difference between the assumption and reality.
- danaris 2y ago
  But when climate changes even by what appear, to us, to be very small amounts, that can have huge effects on how weather systems behave. Thus, if you want your AI model that predicts weather, which was trained on data before that change, to predict weather after that change, you may very well find that it rapidly loses accuracy.
  - jvanderbot 2y ago
    I think the naive answer is that physics doesn't change when the climate does, just the initial conditions for the day's simulation of the next week's weather. And every year over year change is within error bars of the prior years results, once you account for those initial conditions.
    - danaris 2y ago
      And if your model is based on physics, then that's fine. But if your model is an LLM (or close cousin) purely based on previous weather patterns, then it may struggle to accurately predict once the underlying conditions change.
    - geysersam 2y ago
      The less naive answer is that a typical weather model has hundreds of parameters that have been tuned manually over decades of use.
- abdullahkhalids 2y ago
  One of the characteristics of the present climate change on Earth is that weather events (storms, heat waves, maximum and minimum temperatures, start of seasons) display a lot more variability, and consequently also a lot more extremes than in the recent past. This is already happening. If you train a weather model ML on past weather data, which has reduced variability, and use it to predict future weather, it is possible that it under predicts variability of weather variables (amount of expected rainfall, maximum temperature tomorrow etc). Variance not mean. I say "it is possible" because only rigorous modeling can tell us the truth. Which is why I asked the question originally. I don't think it can be denied by abstract arguments.
labster 2y ago
AI is not just pattern matching. Assuming they have some idea of initial and boundary conditions, they should be fine.
crm9125 2y ago
"Tomorrow will be partly cloudy, with a chance of hurricane."
KennyBlanken 2y ago
I wish HNers would ask themselves "If I, someone with nothing more than layman's knowledge, thought of this - would someone who has spent years studying in their field, possibly the head of their lab, the people they collaborated with, the people who approved their grant, and the people who reviewed the paper for publication - think of this?" I can hear the keyboard mashing already. YES, I am aware there are problems in both research and publishing. That does not mean that a HN'ers "did they think of X?" is any more valid. It's like saying that because bridges collapse, we should be allowed people with no relevant experience, knowledge, or training to look at new bridges and say "that bit there doesn't look strong enough to me!"
- throwup238 2y ago
  I wish HNers would stop and ask themselves “Do HNers need to read yet another misguided Dunning-Kruger rant from someone who clearly misread the parent comment?” The GP asked _how_ robust the ML models will be, which is a perfectly good question to ask. Maybe a climate scientist specializing in ML can answer that question.
- RicoElectrico 2y ago
  If you know about a particular field and read HN comments it's even worse.
  - genewitch 2y ago
    Murray Gell-Mann says "et tu?"
- graemep 2y ago
  On the other hands experts do need to be questions and provide explanations. "It's like saying that because bridges collapse, we should be allowed people with no relevant experience, knowledge, or training to look at new bridges and say "that bit there doesn't look strong enough to me!"" Not a valid comparison. Civil engineering is a field with a lot of known answers. bridge designs rarely rely on cutting edge research.
VoodooJuJu 2y ago
[dead]
randoKalrisian 2y ago
It remains to be seen but right now models are constantly evolving and being trained in on recent data, I.e training data including an already changing climate. I have heard the opinion that the models would not learn weather patterns but rather “weather physics”. I am not knowledgeable enough in ML to comment on that though.
interludead 2y ago
Honestly one of the biggest open questions in applying ML to weather and climate

Climate is not weather. Weather predictions are for specific events and areas, made on the order of days- typically no more than 2-3 weeks into the future. Climate models predict future averages over large regions on the order of decades. "rapid climate change" is on the order of "within this century". Whether the climate changes or not doesn't really impact the weather models at all, because the climate's not changing on the same time scale.

labster 2y ago
While I really want to agree with your sound argument, I have this suspicion that climate change significantly affected forecast skill on Hurricane Otis.
- melagonster 2y ago
  Climate change: more typhoons in the future. Weather prediction: which path typhoons will follow?
- zdragnar 2y ago
  The climate didn't change in the time from Otis forming to landing. Weather models look at current conditions and extrapolate what the next set of conditions will be. Another way to look at it is that climate change models reflect what we will experience. Weather models reflect the mechanics of humidity, temperature and air flow- given a current state, what is the next likely state. Climate change doesn't change how those mechanics interact. It only guesses what "current state" will look like in the far future.
  - avianlyric 2y ago
    Climate change doesn’t impact the physics of weather, but our understanding of the inputs to a weather system is still very incomplete. Climate change does impact the relative importance of different physical inputs into a weather system, and inherently pushes weather systems away from the most studied and well understood aspects of weather physics. That in turn degrades the accuracy of our models, as it takes time to research and understand whats changed in a weather system, and what previously unimportant inputs have become much more important.
    - zdragnar 2y ago
      Let's say I live in the hypothetical state of Midwestia. It gets up to 100 degrees F in the summer and down to -40 in the winter months. Over the course of ten years, the average daily temperature goes up by 1 degree F (which is many times faster than reality). That means that there's basically one day in the entire year that doesn't fit within the existing model, or maybe a handful if some other part of the year is cooler than normal. We see more variation between normal years than we expect to see average increase in both the short and medium terms. Our weather models predict no more than two or three weeks into the future. There's literally nothing about climate change that will change how weather systems interact that will occur faster than our weather models will adapt to. Our weather patterns may change drastically, but that's a fundamentally different problem.
      - avianlyric 2y ago
        How quickly do you think our weather models adapt? Our current models are built on top of decades of detailed weather research by tens of thousands of researchers around the world, and has involved the launch of some the largest satellites in space. All to help us try and understand how weather works. Those models don't just "adapt", they're carefully and manually refined over time, slowly incorporating better physical models as they're developed, and better methods of collecting data as it become available. Climate change absolutely changes how weather systems interact faster than our models can adapt, primarily because we can only adapt models to changes after their accuracy degrades, and it becomes possible to start identifying potential weaknesses in the existing approaches. You're example of seasonal temperature changes vs global average temperature increases, vastly underestimates how complicated weather models are, and how much impact changes in global climate systems (like the North Atlantic Current) can drastically impact local weather conditions. Weather models, like all models, will make assumptions about the behaviours of extremely large systems which aren't expected to change very much, or are very hard to measure in real-time. As those large systems change due to climate change, the assumptions made by weather models will grow increasingly incorrect. But that hard part is figuring our which assumption is now incorrect, not always easy to identity exactly what assumptions have been made, or accurately measure the difference between the assumption and reality.
danaris 2y ago
But when climate changes even by what appear, to us, to be very small amounts, that can have huge effects on how weather systems behave. Thus, if you want your AI model that predicts weather, which was trained on data before that change, to predict weather after that change, you may very well find that it rapidly loses accuracy.
- jvanderbot 2y ago
  I think the naive answer is that physics doesn't change when the climate does, just the initial conditions for the day's simulation of the next week's weather. And every year over year change is within error bars of the prior years results, once you account for those initial conditions.
  - danaris 2y ago
    And if your model is based on physics, then that's fine. But if your model is an LLM (or close cousin) purely based on previous weather patterns, then it may struggle to accurately predict once the underlying conditions change.
  - geysersam 2y ago
    The less naive answer is that a typical weather model has hundreds of parameters that have been tuned manually over decades of use.
abdullahkhalids 2y ago
One of the characteristics of the present climate change on Earth is that weather events (storms, heat waves, maximum and minimum temperatures, start of seasons) display a lot more variability, and consequently also a lot more extremes than in the recent past. This is already happening. If you train a weather model ML on past weather data, which has reduced variability, and use it to predict future weather, it is possible that it under predicts variability of weather variables (amount of expected rainfall, maximum temperature tomorrow etc). Variance not mean. I say "it is possible" because only rigorous modeling can tell us the truth. Which is why I asked the question originally. I don't think it can be denied by abstract arguments.

labster 2y ago

AI is not just pattern matching. Assuming they have some idea of initial and boundary conditions, they should be fine.

crm9125 2y ago

"Tomorrow will be partly cloudy, with a chance of hurricane."

KennyBlanken 2y ago

I wish HNers would ask themselves "If I, someone with nothing more than layman's knowledge, thought of this - would someone who has spent years studying in their field, possibly the head of their lab, the people they collaborated with, the people who approved their grant, and the people who reviewed the paper for publication - think of this?" I can hear the keyboard mashing already. YES, I am aware there are problems in both research and publishing. That does not mean that a HN'ers "did they think of X?" is any more valid. It's like saying that because bridges collapse, we should be allowed people with no relevant experience, knowledge, or training to look at new bridges and say "that bit there doesn't look strong enough to me!"

throwup238 2y ago
I wish HNers would stop and ask themselves “Do HNers need to read yet another misguided Dunning-Kruger rant from someone who clearly misread the parent comment?” The GP asked _how_ robust the ML models will be, which is a perfectly good question to ask. Maybe a climate scientist specializing in ML can answer that question.
RicoElectrico 2y ago
If you know about a particular field and read HN comments it's even worse.
- genewitch 2y ago
  Murray Gell-Mann says "et tu?"
graemep 2y ago
On the other hands experts do need to be questions and provide explanations. "It's like saying that because bridges collapse, we should be allowed people with no relevant experience, knowledge, or training to look at new bridges and say "that bit there doesn't look strong enough to me!"" Not a valid comparison. Civil engineering is a field with a lot of known answers. bridge designs rarely rely on cutting edge research.

VoodooJuJu 2y ago

[dead]

randoKalrisian 2y ago

It remains to be seen but right now models are constantly evolving and being trained in on recent data, I.e training data including an already changing climate. I have heard the opinion that the models would not learn weather patterns but rather “weather physics”. I am not knowledgeable enough in ML to comment on that though.

interludead 2y ago

Honestly one of the biggest open questions in applying ML to weather and climate