4 ms·

I am dearly hoping that they are using the current "AI" craze to talk up the machine learning methods they have presumably been using for a decade at this point

I am dearly hoping that they are using the current "AI" craze to talk up the machine learning methods they have presumably been using for a decade at this point, and not that they have actually integrated an LLM into a weather model.

idontwantthis 11mo ago
Hopefully they weren’t all forced out this year. The NOAA had massive cuts.
- trueismywork 11mo ago
  NCAR is being dismantled as we speak.
  - derbOac 11mo ago
    I suspect the names of those perpetrating this kind of destruction will become synonymous with ignorance and intellectual cowardice.
username223 11mo ago
Same. I hope this was written by hardened greybeards who have dedicated their lives to weather prediction and atmospheric modeling, and have "weathered" a few funding cycles.
- akdev1l 11mo ago
  inb4 it’s actually an intern maintaining a 3000+ line markdown file
  - RHSeeger 11mo ago
    I can see it now The following snippet highlights the algorithm used to determine <thing> ```fortran .....
sigmar 11mo ago
Graphcast (the model this is based on) has been validated in weather models for a while[1]. It uses transformers, much like LLMs. Transformers are really impressive at modeling a variety of things and have become very common throughout a lot of ML models, there's no reason to besmirch these methods as "integrating an LLM into a weather model" [1] https://github.com/google-deepmind/graphcast https://github.com/google-deepmind/graphcast
- lynndotpy 11mo ago
  A lot of shiny new "AI" features being shipped are language models being placed where they don't belong. It's reasonable to be skeptical here, not just because of the AI label, but especially for the troubled history of neural-network based ML methods for weather prediction. Even before LLMs got big, a lot of machine learning research being published were models which underperformed SOTA (which was the case for weather modeling for a long time!) or models which are far far larger than they need to be (e.g. this [1] Nature paper using 'deep learning' for aftershock prediction being bested by this [2] Nature paper using one neuron. [1] https://www.nature.com/articles/s41586-018-0438-y https://www.nature.com/articles/s41586-018-0438-y [2] https://www.nature.com/articles/s41586-019-1582-8 https://www.nature.com/articles/s41586-019-1582-8
  - sdenton4 11mo ago
    Not all transformers are LLMs.
    - lynndotpy 11mo ago
      Yes, that is not in contention. Not all transformers are LLMs, not all neural networks are transformers, not all machine learning methods are neural networks, not all statistical methods are machine learning. I'm not saying this is an LLM, margalabargala is not saying this is an LLM. They only said they hoped that they did not integrate an LLM into the weather model, which is a reasonable and informed concern to have. Sigmar is correctly pointing out that they're using a transformer model, and that transformers are effective for modeling things other than language. (And, implicitly, that this _isn't_ adding a step where they ask ChatGPT to vibe check the forecast.)
Legend2440 11mo ago
It’s not an LLM, but it is genAI. It’s based on the same idea of predict-the-next-thing, but instead of predicting words it predicts the next state of the atmosphere from the current state.
- adamweld 11mo ago
  It is in fact one of the least generalized forms of "AI" out there. A model focused solely on predicting weather.
  - astrange 11mo ago
    "gen" stands for "generative". If you read the GenCast paper they call it a generative AI - IIRC it's an autoregressive GNN plus a diffusion model. Which is surprising to me because I didn't think it would work for this; they're bad at estimating uncertainty for instance.
    - Majromax 11mo ago
      > Which is surprising to me because I didn't think it would work for this; they're bad at estimating uncertainty for instance. FGN (the model that is 'WeatherNext 2'), FourCastNet 3 (NVIDIA's offering), and AIFS-CRPS (the model from ECMWF) have all moved to train on whole ensembles, using a cumulative ranked probability score (CRPS) loss function. Minimizing the CRPS minimizes the integrated square differences of the cumulative density function between the prediction and truth, so it's effectively teaching the model to have uncertainty proportional to its expected error. GenCast is a more classic diffusion-based model trained on a mean-squared-error-type loss function, much like any of the image diffusion models. Nonetheless it performed well.
lukeschlather 11mo ago
The GraphCast paper says "GraphCast is implemented using GNNs" without explaining that the acronym stands for Graph Neural Networks. It contrasts GNNs to the " convolutional neural network (CNN)" and "graph attention network." (GAN?) It doesn't really explain the difference between GAN and a GNN. I think LLMs are GANs. So no, it's not an LLM in a weather model, but it's very similar to an LLM in terms of how it is trained.
- astrange 11mo ago
  > I think LLMs are GANs. They aren't, but both of them are transformer models. nb GAN usually means something else (Generative Adversarial Network).
  - lukeschlather 11mo ago
    I used GAN to mean graph attention network in my comment, which is how the GraphCast paper defines transformers. https://arxiv.org/pdf/2212.12794 https://arxiv.org/pdf/2212.12794 I was looking at this part in particular: > And while Transformers [48] can also compute arbitrarily long-range computations, they do not scale well with very large inputs (e.g., the 1 million-plus grid points in GraphCast’s global inputs) because of the quadratic memory complexity induced by computing all-to-all interactions. Contemporary extensions of Transformers often sparsify possible interactions to reduce the complexity, which in effect makes them analogous to GNNs (e.g., graph attention networks [49]). Which kind of makes a soup of the whole thing and suggests that LLMs/Graph Attention Networks are "extensions to transformers" and not exactly transformers themselves.
    - astrange 11mo ago
      Oh yeah, GNN (graph neural network) is the common term, "graph attention network" is pretty confusing because a GAN is a totally different architecture. (Well, not necessarily architecture. Training method?)
optimalsolver 11mo ago
You're absolutely right! That was a category 5. Thanks for pointing that out.

idontwantthis 11mo ago

Hopefully they weren’t all forced out this year. The NOAA had massive cuts.

trueismywork 11mo ago
NCAR is being dismantled as we speak.
- derbOac 11mo ago
  I suspect the names of those perpetrating this kind of destruction will become synonymous with ignorance and intellectual cowardice.

username223 11mo ago

Same. I hope this was written by hardened greybeards who have dedicated their lives to weather prediction and atmospheric modeling, and have "weathered" a few funding cycles.

akdev1l 11mo ago
inb4 it’s actually an intern maintaining a 3000+ line markdown file
- RHSeeger 11mo ago
  I can see it now The following snippet highlights the algorithm used to determine <thing> ```fortran .....

sigmar 11mo ago

Graphcast (the model this is based on) has been validated in weather models for a while[1]. It uses transformers, much like LLMs. Transformers are really impressive at modeling a variety of things and have become very common throughout a lot of ML models, there's no reason to besmirch these methods as "integrating an LLM into a weather model" [1] https://github.com/google-deepmind/graphcast https://github.com/google-deepmind/graphcast

lynndotpy 11mo ago
A lot of shiny new "AI" features being shipped are language models being placed where they don't belong. It's reasonable to be skeptical here, not just because of the AI label, but especially for the troubled history of neural-network based ML methods for weather prediction. Even before LLMs got big, a lot of machine learning research being published were models which underperformed SOTA (which was the case for weather modeling for a long time!) or models which are far far larger than they need to be (e.g. this [1] Nature paper using 'deep learning' for aftershock prediction being bested by this [2] Nature paper using one neuron. [1] https://www.nature.com/articles/s41586-018-0438-y https://www.nature.com/articles/s41586-018-0438-y [2] https://www.nature.com/articles/s41586-019-1582-8 https://www.nature.com/articles/s41586-019-1582-8
- sdenton4 11mo ago
  Not all transformers are LLMs.
  - lynndotpy 11mo ago
    Yes, that is not in contention. Not all transformers are LLMs, not all neural networks are transformers, not all machine learning methods are neural networks, not all statistical methods are machine learning. I'm not saying this is an LLM, margalabargala is not saying this is an LLM. They only said they hoped that they did not integrate an LLM into the weather model, which is a reasonable and informed concern to have. Sigmar is correctly pointing out that they're using a transformer model, and that transformers are effective for modeling things other than language. (And, implicitly, that this _isn't_ adding a step where they ask ChatGPT to vibe check the forecast.)

Legend2440 11mo ago

It’s not an LLM, but it is genAI. It’s based on the same idea of predict-the-next-thing, but instead of predicting words it predicts the next state of the atmosphere from the current state.

adamweld 11mo ago
It is in fact one of the least generalized forms of "AI" out there. A model focused solely on predicting weather.
- astrange 11mo ago
  "gen" stands for "generative". If you read the GenCast paper they call it a generative AI - IIRC it's an autoregressive GNN plus a diffusion model. Which is surprising to me because I didn't think it would work for this; they're bad at estimating uncertainty for instance.
  - Majromax 11mo ago
    > Which is surprising to me because I didn't think it would work for this; they're bad at estimating uncertainty for instance. FGN (the model that is 'WeatherNext 2'), FourCastNet 3 (NVIDIA's offering), and AIFS-CRPS (the model from ECMWF) have all moved to train on whole ensembles, using a cumulative ranked probability score (CRPS) loss function. Minimizing the CRPS minimizes the integrated square differences of the cumulative density function between the prediction and truth, so it's effectively teaching the model to have uncertainty proportional to its expected error. GenCast is a more classic diffusion-based model trained on a mean-squared-error-type loss function, much like any of the image diffusion models. Nonetheless it performed well.

lukeschlather 11mo ago

The GraphCast paper says "GraphCast is implemented using GNNs" without explaining that the acronym stands for Graph Neural Networks. It contrasts GNNs to the " convolutional neural network (CNN)" and "graph attention network." (GAN?) It doesn't really explain the difference between GAN and a GNN. I think LLMs are GANs. So no, it's not an LLM in a weather model, but it's very similar to an LLM in terms of how it is trained.

astrange 11mo ago
> I think LLMs are GANs. They aren't, but both of them are transformer models. nb GAN usually means something else (Generative Adversarial Network).
- lukeschlather 11mo ago
  I used GAN to mean graph attention network in my comment, which is how the GraphCast paper defines transformers. https://arxiv.org/pdf/2212.12794 https://arxiv.org/pdf/2212.12794 I was looking at this part in particular: > And while Transformers [48] can also compute arbitrarily long-range computations, they do not scale well with very large inputs (e.g., the 1 million-plus grid points in GraphCast’s global inputs) because of the quadratic memory complexity induced by computing all-to-all interactions. Contemporary extensions of Transformers often sparsify possible interactions to reduce the complexity, which in effect makes them analogous to GNNs (e.g., graph attention networks [49]). Which kind of makes a soup of the whole thing and suggests that LLMs/Graph Attention Networks are "extensions to transformers" and not exactly transformers themselves.
  - astrange 11mo ago
    Oh yeah, GNN (graph neural network) is the common term, "graph attention network" is pretty confusing because a GAN is a totally different architecture. (Well, not necessarily architecture. Training method?)

optimalsolver 11mo ago

You're absolutely right! That was a category 5. Thanks for pointing that out.