6 ms·

Google's AI weather prediction model is pretty darn good

79 points by Garbage 2y ago

rvnx 2y ago
GenCast was trained on data from 1959-2023, so no surprise it can "predict" back 2019. It's like the super trading algorithms who achieve perfect scores during backtest. The question is, how does it perform on unknown events.
- 1727706962 2y ago
  Looks like it was a forward prediction. From the linked article: > GenCast is a machine learning weather prediction model trained on weather data from 1979 to 2018 and a google blog https://deepmind.google/discover/blog/gencast-predicts-weather-and-the-risks-of-extreme-conditions-with-sota-accuracy/ https://deepmind.google/discover/blog/gencast-predicts-weath... > To rigorously evaluate GenCast's performance, we trained it on historical weather data up to 2018, and tested it on data from 2019
  - pfisherman 2y ago
    This is still on retrospective data. The machine learning graveyard is filled with models that worked well on retrospective data, but did not hold up in a live inference setting. Just ask Zillow. The real test is whether they can predict the weather 14 days out in 2025. I am guessing they did not want to set up the data pipeline to run inference in a live setting. But that is what I would need to see to be a true believer. Still a cool result and article though.
    - scellus 2y ago
      ECMWF runs many such models at their site, a run two or four times per day, and they have verification statistics too, no need to doubt the accuracy. The Google model is probably the best so far but ECMWF's own diffusion model was already on par with ENS and many point-forecast models (graph transformers, not diffusion) outperform state-of-the-art physical models. What is missing is initialization directly from observations. All the best-performing models initialize from ERA5 or other reconstruction.
- akira2501 2y ago
  > One caveat is that GenCast tested itself against an older version of ENS, which now operates at a higher resolution. The peer-reviewed research compares GenCast predictions to ENS forecasts for 2019, seeing how close each model got to real-world conditions that year. And GenCast was tested against and older model which performs worse. > The ENS system has improved significantly since 2019, according to ECMWF machine learning coordinator Matt Chantry. That makes it difficult to say how well GenCast might perform against ENS today. And the testing makes it "difficult to say." The obvious conclusion is "run a new set of tests" but they'd rather pay of the verge to publish half truths instead.
  - scellus 2y ago
    But ECMWF itself runs a diffusion model that is practically on par with ENS in accuracy. They also seem to collaborate closely.
amelius 2y ago
Too bad climate change is happening now, so the model will have to extrapolate.
- akira2501 2y ago
  Forecast models predict 7 to 14 days ahead. They have not and never will be able to predict further than that. We don't even need them to.
  - zsims 2y ago
    We do, I want to know if it's going to rain on my birthday
  - unsupp0rted 2y ago
    Never will? I wouldn't be surprised if predicting climate + weather 12 months out is a simpler problem than most medical problems at which AI is currently being pointed.
    - JumpCrisscross 2y ago
      > wouldn't be surprised if predicting climate + weather 12 months out is a simpler problem than most medical problems at which AI is currently being pointed Simple systems can be famously unpredictable [1]. Our bodies manage entropy; that should make them complex but predictable. The weather, on the other hand, has no governors or raison d'être. [1] https://en.wikipedia.org/wiki/Three-body_problem https://en.wikipedia.org/wiki/Three-body_problem
      - tomjakubowski 2y ago
        The three body problem lacks a closed form solution. How does that mean it's unpredictable, though? I thought that numerical methods can be used to make n-body predictions to arbitrary precision. Are these simulations less accurate than I am thinking? How do engineers and scientists working on space probes plan their trajectories and such?
dehrmann 2y ago
I'm not surprised. This is the sort of problem machine learning is really good at solving. There's a lot of quality training data, and the results are governed by physics.
- singhrac 2y ago
  That's not really 100% true. A lot of the data this is trained on is ERA5, which appears to be highly dense in both time and space, but is assimilation data inferred from much sparser observations. I wouldn't say it's inaccurate, but I see pretty large deviations between assimilation datasets and private weather observations (I work on this problem). The results are governed by physics up to some level, but we can't simulate at that fine a level, so there's some inherent aleatoric uncertainty (i.e. noise). And I would generally say that physics-simulation-ML is not moving as fast as say, inference on images or text. For example, if you see a picture of a car, there's very little inherent uncertainty on what the answer is. If you see the world simulator state there's a lot of uncertainty on what happens next. That all being said, I think is basically the best model out there , and almost certainly the best open model. This is really the culmination of many years of effort getting data and software in place to run such a large scale training job. Very impressive!
  - qeternity 2y ago
    > And I would generally say that physics-simulation-ML is not moving as fast as say, inference on images or text. I’m not sure that a blanket statement like this is a valid argument in an article that perhaps suggests the opposite is true.
  - dehrmann 2y ago
    > if you see a picture of a car, there's very little inherent uncertainty on what the answer is Unless its a captcha.
  - wenc 2y ago
    > For example, if you see a picture of a car, there's very little inherent uncertainty on what the answer is. If you see the world simulator state there's a lot of uncertainty on what happens next. I've been thinking about this a lot. Many ML people work with what is "closed-domain" data -- the data is essentially complete (image, sound, words, or any kind of embeddings) with no unmeasured variables, so the ML algorithm is essentially trying to learn a function that can predict this. Unfortunately a lot of "open-domain" data has tons of unmeasured variables that are contextual. Suppose I were to try to predict how a full a parking lot would be over the course of a week. You can gather lots and lots of data, but still never get to a near-perfect level of accuracy because the co-variates that drive how full a parking lot is (unexpected influencer effects on the demand, competitive forces that happen to shift one day, power outages in the other part of town, other irreducible randomness = "aleatoric uncertainty" in technical parlance) aren't in the data (or at least not completely). Fortunately this isn't a problem in real life because many effects cancel each other out, so we are able to arrive at a good-enough aggregate prediction. But "open-domain" ML problems will never achieve the kind of accuracy that "closed-domain" ML can achieve, even with tons of data. Closed domain ML can assume a degree of regularity that open domain ML can never assume.
Reubend 2y ago
Although it's great to see these advancements, I would like to see it integrated with the Google Weather results that show up in search and on Android devices before I get excited. Spinning up the model on my own hardware and feeding it data manually is a decent amount of work, and I'm too lazy to do that.
- scellus 2y ago
  It's a medium-range global model, while Google Weather (which I don't have) is mostly about local short-range weather? But Google Weather is already based on an AI prediction on most cities: https://research.google/blog/metnet-3-a-state-of-the-art-neural-weather-model-available-in-google-products/ https://research.google/blog/metnet-3-a-state-of-the-art-neu... Google says GenCast forecasts will later be available from them too. Also ECMWF runs a very similar diffusion model, it's not operational but run a couple of times a day with results available on their graph site (and as data files too I guess): https://charts.ecmwf.int/ https://charts.ecmwf.int/
smartmic 2y ago
I wonder if GenCast's 15 day forecast is really the right indicator for forecasting? I could imagine that such long term ML forecasts tend to get closer to yearly averages, which are kind of "washed out", but of course look good for benchmarking and marketing reasons. But they are not so practical for the majority of weather forecast users. In short, it still smells a bit like AI snake oil to me. [1] [1] more about this: https://press.princeton.edu/books/hardcover/9780691249131/ai-snake-oil https://press.princeton.edu/books/hardcover/9780691249131/ai...
- aardvarkr 2y ago
  If that was the only thing it was trained on that could be the case but they’re predicting everything and being graded against more than just their success 15 days out. I think that’s just one of the flashier value adds for weather-dependent businesses like wind production that they can have vs traditional models
hnburnsy 2y ago
You can see the weather forecast maps of Deep Mind here... https://charts.ecmwf.int/products/graphcast_medium-mslp-wind850 https://charts.ecmwf.int/products/graphcast_medium-mslp-wind... I was watching it during the recent hurricane season, and it did not seem to perform much differently than other models.
deleted 2y ago
[deleted]

79 points by Garbage 2y ago

rvnx 2y ago

GenCast was trained on data from 1959-2023, so no surprise it can "predict" back 2019. It's like the super trading algorithms who achieve perfect scores during backtest. The question is, how does it perform on unknown events.

1727706962 2y ago
Looks like it was a forward prediction. From the linked article: > GenCast is a machine learning weather prediction model trained on weather data from 1979 to 2018 and a google blog https://deepmind.google/discover/blog/gencast-predicts-weather-and-the-risks-of-extreme-conditions-with-sota-accuracy/ https://deepmind.google/discover/blog/gencast-predicts-weath... > To rigorously evaluate GenCast's performance, we trained it on historical weather data up to 2018, and tested it on data from 2019
- pfisherman 2y ago
  This is still on retrospective data. The machine learning graveyard is filled with models that worked well on retrospective data, but did not hold up in a live inference setting. Just ask Zillow. The real test is whether they can predict the weather 14 days out in 2025. I am guessing they did not want to set up the data pipeline to run inference in a live setting. But that is what I would need to see to be a true believer. Still a cool result and article though.
  - scellus 2y ago
    ECMWF runs many such models at their site, a run two or four times per day, and they have verification statistics too, no need to doubt the accuracy. The Google model is probably the best so far but ECMWF's own diffusion model was already on par with ENS and many point-forecast models (graph transformers, not diffusion) outperform state-of-the-art physical models. What is missing is initialization directly from observations. All the best-performing models initialize from ERA5 or other reconstruction.
akira2501 2y ago
> One caveat is that GenCast tested itself against an older version of ENS, which now operates at a higher resolution. The peer-reviewed research compares GenCast predictions to ENS forecasts for 2019, seeing how close each model got to real-world conditions that year. And GenCast was tested against and older model which performs worse. > The ENS system has improved significantly since 2019, according to ECMWF machine learning coordinator Matt Chantry. That makes it difficult to say how well GenCast might perform against ENS today. And the testing makes it "difficult to say." The obvious conclusion is "run a new set of tests" but they'd rather pay of the verge to publish half truths instead.
- scellus 2y ago
  But ECMWF itself runs a diffusion model that is practically on par with ENS in accuracy. They also seem to collaborate closely.

amelius 2y ago

Too bad climate change is happening now, so the model will have to extrapolate.

akira2501 2y ago
Forecast models predict 7 to 14 days ahead. They have not and never will be able to predict further than that. We don't even need them to.
- zsims 2y ago
  We do, I want to know if it's going to rain on my birthday
- unsupp0rted 2y ago
  Never will? I wouldn't be surprised if predicting climate + weather 12 months out is a simpler problem than most medical problems at which AI is currently being pointed.
  - JumpCrisscross 2y ago
    > wouldn't be surprised if predicting climate + weather 12 months out is a simpler problem than most medical problems at which AI is currently being pointed Simple systems can be famously unpredictable [1]. Our bodies manage entropy; that should make them complex but predictable. The weather, on the other hand, has no governors or raison d'être. [1] https://en.wikipedia.org/wiki/Three-body_problem https://en.wikipedia.org/wiki/Three-body_problem
    - tomjakubowski 2y ago
      The three body problem lacks a closed form solution. How does that mean it's unpredictable, though? I thought that numerical methods can be used to make n-body predictions to arbitrary precision. Are these simulations less accurate than I am thinking? How do engineers and scientists working on space probes plan their trajectories and such?

dehrmann 2y ago

I'm not surprised. This is the sort of problem machine learning is really good at solving. There's a lot of quality training data, and the results are governed by physics.

singhrac 2y ago
That's not really 100% true. A lot of the data this is trained on is ERA5, which appears to be highly dense in both time and space, but is assimilation data inferred from much sparser observations. I wouldn't say it's inaccurate, but I see pretty large deviations between assimilation datasets and private weather observations (I work on this problem). The results are governed by physics up to some level, but we can't simulate at that fine a level, so there's some inherent aleatoric uncertainty (i.e. noise). And I would generally say that physics-simulation-ML is not moving as fast as say, inference on images or text. For example, if you see a picture of a car, there's very little inherent uncertainty on what the answer is. If you see the world simulator state there's a lot of uncertainty on what happens next. That all being said, I think is basically the best model out there , and almost certainly the best open model. This is really the culmination of many years of effort getting data and software in place to run such a large scale training job. Very impressive!
- qeternity 2y ago
  > And I would generally say that physics-simulation-ML is not moving as fast as say, inference on images or text. I’m not sure that a blanket statement like this is a valid argument in an article that perhaps suggests the opposite is true.
- dehrmann 2y ago
  > if you see a picture of a car, there's very little inherent uncertainty on what the answer is Unless its a captcha.
- wenc 2y ago
  > For example, if you see a picture of a car, there's very little inherent uncertainty on what the answer is. If you see the world simulator state there's a lot of uncertainty on what happens next. I've been thinking about this a lot. Many ML people work with what is "closed-domain" data -- the data is essentially complete (image, sound, words, or any kind of embeddings) with no unmeasured variables, so the ML algorithm is essentially trying to learn a function that can predict this. Unfortunately a lot of "open-domain" data has tons of unmeasured variables that are contextual. Suppose I were to try to predict how a full a parking lot would be over the course of a week. You can gather lots and lots of data, but still never get to a near-perfect level of accuracy because the co-variates that drive how full a parking lot is (unexpected influencer effects on the demand, competitive forces that happen to shift one day, power outages in the other part of town, other irreducible randomness = "aleatoric uncertainty" in technical parlance) aren't in the data (or at least not completely). Fortunately this isn't a problem in real life because many effects cancel each other out, so we are able to arrive at a good-enough aggregate prediction. But "open-domain" ML problems will never achieve the kind of accuracy that "closed-domain" ML can achieve, even with tons of data. Closed domain ML can assume a degree of regularity that open domain ML can never assume.

Reubend 2y ago

Although it's great to see these advancements, I would like to see it integrated with the Google Weather results that show up in search and on Android devices before I get excited. Spinning up the model on my own hardware and feeding it data manually is a decent amount of work, and I'm too lazy to do that.

scellus 2y ago
It's a medium-range global model, while Google Weather (which I don't have) is mostly about local short-range weather? But Google Weather is already based on an AI prediction on most cities: https://research.google/blog/metnet-3-a-state-of-the-art-neural-weather-model-available-in-google-products/ https://research.google/blog/metnet-3-a-state-of-the-art-neu... Google says GenCast forecasts will later be available from them too. Also ECMWF runs a very similar diffusion model, it's not operational but run a couple of times a day with results available on their graph site (and as data files too I guess): https://charts.ecmwf.int/ https://charts.ecmwf.int/

smartmic 2y ago

I wonder if GenCast's 15 day forecast is really the right indicator for forecasting? I could imagine that such long term ML forecasts tend to get closer to yearly averages, which are kind of "washed out", but of course look good for benchmarking and marketing reasons. But they are not so practical for the majority of weather forecast users. In short, it still smells a bit like AI snake oil to me. [1] [1] more about this: https://press.princeton.edu/books/hardcover/9780691249131/ai-snake-oil https://press.princeton.edu/books/hardcover/9780691249131/ai...

aardvarkr 2y ago
If that was the only thing it was trained on that could be the case but they’re predicting everything and being graded against more than just their success 15 days out. I think that’s just one of the flashier value adds for weather-dependent businesses like wind production that they can have vs traditional models

hnburnsy 2y ago

You can see the weather forecast maps of Deep Mind here... https://charts.ecmwf.int/products/graphcast_medium-mslp-wind850 https://charts.ecmwf.int/products/graphcast_medium-mslp-wind... I was watching it during the recent hurricane season, and it did not seem to perform much differently than other models.

deleted 2y ago

[deleted]