3 ms·

GenCast was trained on data from 1959-2023, so no surprise it can "predict" back 2019. It's like the super trading algorithms who achieve perfect scores during

by rvnx 2y ago

GenCast was trained on data from 1959-2023, so no surprise it can "predict" back 2019. It's like the super trading algorithms who achieve perfect scores during backtest. The question is, how does it perform on unknown events.

1727706962 2y ago
Looks like it was a forward prediction. From the linked article: > GenCast is a machine learning weather prediction model trained on weather data from 1979 to 2018 and a google blog https://deepmind.google/discover/blog/gencast-predicts-weather-and-the-risks-of-extreme-conditions-with-sota-accuracy/ https://deepmind.google/discover/blog/gencast-predicts-weath... > To rigorously evaluate GenCast's performance, we trained it on historical weather data up to 2018, and tested it on data from 2019
- pfisherman 2y ago
  This is still on retrospective data. The machine learning graveyard is filled with models that worked well on retrospective data, but did not hold up in a live inference setting. Just ask Zillow. The real test is whether they can predict the weather 14 days out in 2025. I am guessing they did not want to set up the data pipeline to run inference in a live setting. But that is what I would need to see to be a true believer. Still a cool result and article though.
  - scellus 2y ago
    ECMWF runs many such models at their site, a run two or four times per day, and they have verification statistics too, no need to doubt the accuracy. The Google model is probably the best so far but ECMWF's own diffusion model was already on par with ENS and many point-forecast models (graph transformers, not diffusion) outperform state-of-the-art physical models. What is missing is initialization directly from observations. All the best-performing models initialize from ERA5 or other reconstruction.
akira2501 2y ago
> One caveat is that GenCast tested itself against an older version of ENS, which now operates at a higher resolution. The peer-reviewed research compares GenCast predictions to ENS forecasts for 2019, seeing how close each model got to real-world conditions that year. And GenCast was tested against and older model which performs worse. > The ENS system has improved significantly since 2019, according to ECMWF machine learning coordinator Matt Chantry. That makes it difficult to say how well GenCast might perform against ENS today. And the testing makes it "difficult to say." The obvious conclusion is "run a new set of tests" but they'd rather pay of the verge to publish half truths instead.
- scellus 2y ago
  But ECMWF itself runs a diffusion model that is practically on par with ENS in accuracy. They also seem to collaborate closely.

by rvnx 2y ago

1727706962 2y ago

Looks like it was a forward prediction. From the linked article: > GenCast is a machine learning weather prediction model trained on weather data from 1979 to 2018 and a google blog https://deepmind.google/discover/blog/gencast-predicts-weather-and-the-risks-of-extreme-conditions-with-sota-accuracy/ https://deepmind.google/discover/blog/gencast-predicts-weath... > To rigorously evaluate GenCast's performance, we trained it on historical weather data up to 2018, and tested it on data from 2019

pfisherman 2y ago
This is still on retrospective data. The machine learning graveyard is filled with models that worked well on retrospective data, but did not hold up in a live inference setting. Just ask Zillow. The real test is whether they can predict the weather 14 days out in 2025. I am guessing they did not want to set up the data pipeline to run inference in a live setting. But that is what I would need to see to be a true believer. Still a cool result and article though.
- scellus 2y ago
  ECMWF runs many such models at their site, a run two or four times per day, and they have verification statistics too, no need to doubt the accuracy. The Google model is probably the best so far but ECMWF's own diffusion model was already on par with ENS and many point-forecast models (graph transformers, not diffusion) outperform state-of-the-art physical models. What is missing is initialization directly from observations. All the best-performing models initialize from ERA5 or other reconstruction.

akira2501 2y ago

> One caveat is that GenCast tested itself against an older version of ENS, which now operates at a higher resolution. The peer-reviewed research compares GenCast predictions to ENS forecasts for 2019, seeing how close each model got to real-world conditions that year. And GenCast was tested against and older model which performs worse. > The ENS system has improved significantly since 2019, according to ECMWF machine learning coordinator Matt Chantry. That makes it difficult to say how well GenCast might perform against ENS today. And the testing makes it "difficult to say." The obvious conclusion is "run a new set of tests" but they'd rather pay of the verge to publish half truths instead.

scellus 2y ago
But ECMWF itself runs a diffusion model that is practically on par with ENS in accuracy. They also seem to collaborate closely.