3 ms·

Seems to be a "pattern" in AI for weather forecasting -- 1. Due to AI hype and funding (even predecessor hype cycles like "big data", ML, even GOF time-serie

by npalli 2y ago

Seems to be a "pattern" in AI for weather forecasting -- 1. Due to AI hype and funding (even predecessor hype cycles like "big data", ML, even GOF time-series statistics), generate 100s of AI models. 2. Of these, a few perform better than current models and are seen as the "future". 3. One year later models in 2. fail catastrophically. 4. Go back to step 1.

darby_nine 2y ago
At first blush, AI seems to be fantastic for weather prediction: 1. Hugely chaotic model, which means deterministic prediction is extremely expensive, and accurate prediction more so. 2. Certain trends seem to dominate the observed time period, hence the field of meteorology.> 3. AI can clue you in as to which trends seem to be most likely to dominate, or if uncertainty is too high to predict, without the investment of major labor costs. While I'm not sure what the market offers, I can say that if the demand-side seems unsatisfied this is the perfect place for a state-operated solution to enter: a zero-margin service where quality in the long-term is demanded over short-term profits. EDIT: just clarifying terms EDIT2: Meteorology -> meteorology
- mjburgess 2y ago
  Well with "AI" all we're talking about is theory-free frequentist modelling. We can already do that for the weather: just take the mode of the last 10 years of the weather on this day, and some sort of weighted mean for temp/humidity/etc. All "AI" is doing here is overfitting this process. On the overfit range, yes, you'll get better peformance. But the baseline performance here, from taking averages, is already pretty good. What we need for the "future" of weather is long-tail events which cannot be modelled by curve-fitting to historical data.
  - partitioned 2y ago
    This is simply not true and an uninformed opinion of what modeling physical events with AI looks like. You seem to assume it’s a purely data-driven approach, it is not. You could use a physics informed neural network (PINN) that utilizes both approaches. That is, it uses both historical data (and likely synthetic data from physics models), as well as physical equations in the loss function (in this case atmospheric as fluid equations) as part of the training. It can truly be the best of both worlds if approached correctly. That being said, 99% of AI out there is just masters thesis level data in -> prediction out, but that is far from what the useful AI models that are currently being developed to predict and forecast dynamical physical systems. Additionally, you can generate synthetic data with the physical models of “edge” and “tail” events to train the model on. This by itself allows the ML model to be able to model almost all events that we can physically model, so at its base it’s at least as useful as the big O order models we use while being orders of magnitude faster. This doesn’t even account for using the physical equations to assist in training the model directly (through architecture tricks or in the loss function). Source: I work on AI models that merge data and physics for dynamical physical systems
    - Majromax 2y ago
      > That being said, 99% of AI out there is just masters thesis level data in -> prediction out, but that is far from what the useful AI models that are currently being developed to predict and forecast dynamical physical systems. I think one surprising outcome of the recent wave of ML-based weather forecasting models is just how accurate the "dumb" approaches are. Pangu-Weather and a couple of its peers are essentially vision transformers without much explicit physics-informed regularization.
    - mjburgess 2y ago
      If you have explanatory models constraining the space of possible function fits, etc. etc. then I concede the point -- though, I rather regard it as my point. The comment I replied to used "AI" in its generic sense which I take to name the theory-free frequentist stats currently in vogue. I don't regard theories as AI -- so adding physics to a NN is, in large part, computational physics. You can call it "AI", but then so-goes any use of a computer model of any kind.
      - partitioned 2y ago
        Well, the difference is the data-driven aspect of parts of the model. While its constrained by physics during the learning process it isn't just running a forward physics model to get the solution. The upfront computational load and extremely fast inference times through parameterization IS what makes it AI, and what makes it useful versus a normal numerical computer model.
        mjburgess 2y ago
        Physics has used "empirical/phenomenological models" where curve-fitting to data has served to preclude the need for simulation, or if it's computationally intractable, etc. I'd agree that it had been underused, since I'd say such modelling is held somewhat in contempt as giving up on doing physics. Do you have a paper that discusses any of this work in these terms? I'm presently writing a larger survey on XAI towards a theory-informed approach, and it seems these mixed models might have some novel explanatory upside/needs. At the moment i'm inclined to partition the world into theory-based and theory-free.
  - jebarker 2y ago
    > What we need for the "future" of weather is long-tail events which cannot be modelled by curve-fitting to historical data. Yes, it seems like weather forecasting is a simulation problem not a low-shot prediction problem. I assume it's one of the computationally irreducible problems Stephen Wolfram talks about
  - Majromax 2y ago
    Although it's obviously difficult to crack open an ML model, they do perform enough computation to have potentially learned something like the dynamical equations for the atmosphere. At the same time, some ML models are surprisingly parsimonious. Graphcast has about 37 million trainable parameters, but its output is a forecast (increment) of six variables on a 37-level, quarter-degree lat/lon grid. That's about 235 million outputs for a single forecast date, so it's safe to conclude that Graphcast cannot memorize its training set. Researchers are also explicitly interested in probing the out-of-sample behaviour of ML models for plausibility. A paper last year by Hakim and Masanam (https://arxiv.org/abs/2309.10867 https://arxiv.org/abs/2309.10867) put Pangu-Weather through some simplified but out-of-sample test cases and saw physically plausible outputs, so the ML models have at least not fallen at the first hurdle. Meanwhile, it's also not quite correct to give traditional models an automatic pass for out-of-sample behaviour. The large-scale dyanmical equations of the atmosphere are well-understood, but so much of the chaos comes from poorly-resolved, poorly-modeled, or poorly-constrained processes near and below the grid scale. The microstructure of clouds, for example, is completely invisible to models that must necessarily run at kilometer or tens-of-kilometer scales. Operational weather models rely on parameterizations to close the system and statistical correlations to assimilate observational data. As far as I'm aware, all of the operational weather models missed the rapid intensification of hurricane Otis last year, an out-of-sample event with deadly consequences.
    - Kon-Peki 2y ago
      > weather models missed the rapid intensification of hurricane Otis last year Which happened because there was very little data to feed into the models. AI isn't going to help with this. The Atlantic Ocean and Gulf of Mexico have tons of data-collecting bouys and the Hurricane Hunter aircraft fly from the eastern US. Hurricane Hunters that go to the Pacific fly out of Mississippi, which adds quite a lot of latency to the data collection probes. We should be adding more bouys to the Pacific, and need to add a Hurricane Hunter crew in San Diego (or perhaps the government of Mexico would like to host and pay for them). Then we can start seeing what the models and AI will do.
      - Majromax 2y ago
mandibeet 2y ago
Initial successes can lead to failure but I think this is the pattern to future development
- zuckerma 2y ago
  So true.
prasoonds 2y ago
This has not really been my experience in my information bubble. But I admit I haven't kept up too well with the latest models and their failure modes. Can you have provide any examples of the scenario you described?

by npalli 2y ago

darby_nine 2y ago

At first blush, AI seems to be fantastic for weather prediction: 1. Hugely chaotic model, which means deterministic prediction is extremely expensive, and accurate prediction more so. 2. Certain trends seem to dominate the observed time period, hence the field of meteorology.> 3. AI can clue you in as to which trends seem to be most likely to dominate, or if uncertainty is too high to predict, without the investment of major labor costs. While I'm not sure what the market offers, I can say that if the demand-side seems unsatisfied this is the perfect place for a state-operated solution to enter: a zero-margin service where quality in the long-term is demanded over short-term profits. EDIT: just clarifying terms EDIT2: Meteorology -> meteorology

mjburgess 2y ago
Well with "AI" all we're talking about is theory-free frequentist modelling. We can already do that for the weather: just take the mode of the last 10 years of the weather on this day, and some sort of weighted mean for temp/humidity/etc. All "AI" is doing here is overfitting this process. On the overfit range, yes, you'll get better peformance. But the baseline performance here, from taking averages, is already pretty good. What we need for the "future" of weather is long-tail events which cannot be modelled by curve-fitting to historical data.
- partitioned 2y ago
  This is simply not true and an uninformed opinion of what modeling physical events with AI looks like. You seem to assume it’s a purely data-driven approach, it is not. You could use a physics informed neural network (PINN) that utilizes both approaches. That is, it uses both historical data (and likely synthetic data from physics models), as well as physical equations in the loss function (in this case atmospheric as fluid equations) as part of the training. It can truly be the best of both worlds if approached correctly. That being said, 99% of AI out there is just masters thesis level data in -> prediction out, but that is far from what the useful AI models that are currently being developed to predict and forecast dynamical physical systems. Additionally, you can generate synthetic data with the physical models of “edge” and “tail” events to train the model on. This by itself allows the ML model to be able to model almost all events that we can physically model, so at its base it’s at least as useful as the big O order models we use while being orders of magnitude faster. This doesn’t even account for using the physical equations to assist in training the model directly (through architecture tricks or in the loss function). Source: I work on AI models that merge data and physics for dynamical physical systems
  - Majromax 2y ago
    > That being said, 99% of AI out there is just masters thesis level data in -> prediction out, but that is far from what the useful AI models that are currently being developed to predict and forecast dynamical physical systems. I think one surprising outcome of the recent wave of ML-based weather forecasting models is just how accurate the "dumb" approaches are. Pangu-Weather and a couple of its peers are essentially vision transformers without much explicit physics-informed regularization.
  - mjburgess 2y ago
    If you have explanatory models constraining the space of possible function fits, etc. etc. then I concede the point -- though, I rather regard it as my point. The comment I replied to used "AI" in its generic sense which I take to name the theory-free frequentist stats currently in vogue. I don't regard theories as AI -- so adding physics to a NN is, in large part, computational physics. You can call it "AI", but then so-goes any use of a computer model of any kind.
    - partitioned 2y ago
      Well, the difference is the data-driven aspect of parts of the model. While its constrained by physics during the learning process it isn't just running a forward physics model to get the solution. The upfront computational load and extremely fast inference times through parameterization IS what makes it AI, and what makes it useful versus a normal numerical computer model.
      - mjburgess 2y ago
        Physics has used "empirical/phenomenological models" where curve-fitting to data has served to preclude the need for simulation, or if it's computationally intractable, etc. I'd agree that it had been underused, since I'd say such modelling is held somewhat in contempt as giving up on doing physics. Do you have a paper that discusses any of this work in these terms? I'm presently writing a larger survey on XAI towards a theory-informed approach, and it seems these mixed models might have some novel explanatory upside/needs. At the moment i'm inclined to partition the world into theory-based and theory-free.
- jebarker 2y ago
  > What we need for the "future" of weather is long-tail events which cannot be modelled by curve-fitting to historical data. Yes, it seems like weather forecasting is a simulation problem not a low-shot prediction problem. I assume it's one of the computationally irreducible problems Stephen Wolfram talks about
- Majromax 2y ago
  Although it's obviously difficult to crack open an ML model, they do perform enough computation to have potentially learned something like the dynamical equations for the atmosphere. At the same time, some ML models are surprisingly parsimonious. Graphcast has about 37 million trainable parameters, but its output is a forecast (increment) of six variables on a 37-level, quarter-degree lat/lon grid. That's about 235 million outputs for a single forecast date, so it's safe to conclude that Graphcast cannot memorize its training set. Researchers are also explicitly interested in probing the out-of-sample behaviour of ML models for plausibility. A paper last year by Hakim and Masanam (https://arxiv.org/abs/2309.10867 https://arxiv.org/abs/2309.10867) put Pangu-Weather through some simplified but out-of-sample test cases and saw physically plausible outputs, so the ML models have at least not fallen at the first hurdle. Meanwhile, it's also not quite correct to give traditional models an automatic pass for out-of-sample behaviour. The large-scale dyanmical equations of the atmosphere are well-understood, but so much of the chaos comes from poorly-resolved, poorly-modeled, or poorly-constrained processes near and below the grid scale. The microstructure of clouds, for example, is completely invisible to models that must necessarily run at kilometer or tens-of-kilometer scales. Operational weather models rely on parameterizations to close the system and statistical correlations to assimilate observational data. As far as I'm aware, all of the operational weather models missed the rapid intensification of hurricane Otis last year, an out-of-sample event with deadly consequences.
  - Kon-Peki 2y ago
    > weather models missed the rapid intensification of hurricane Otis last year Which happened because there was very little data to feed into the models. AI isn't going to help with this. The Atlantic Ocean and Gulf of Mexico have tons of data-collecting bouys and the Hurricane Hunter aircraft fly from the eastern US. Hurricane Hunters that go to the Pacific fly out of Mississippi, which adds quite a lot of latency to the data collection probes. We should be adding more bouys to the Pacific, and need to add a Hurricane Hunter crew in San Diego (or perhaps the government of Mexico would like to host and pay for them). Then we can start seeing what the models and AI will do.
    - Majromax 2y ago

mandibeet 2y ago

Initial successes can lead to failure but I think this is the pattern to future development

zuckerma 2y ago
So true.

prasoonds 2y ago

This has not really been my experience in my information bubble. But I admit I haven't kept up too well with the latest models and their failure modes. Can you have provide any examples of the scenario you described?