4 ms·

I thought I remembered this model being released many months ago? Or is this a new update to the model?

• xnx 2y ago
  New model. Much better than the last one.
  • dylan604 2y ago
    It's so new that it has no long term track record. I want to see how its record looks after say 1/3/5 years. Any 15 day forecast is just too far into the future for me to take seriously. Even the 3 day forecast is loose at best. By the time it reaches 7-10 days, some of the forecast is completely off when it reaches the 3 day window. "But DeepMind said GenCast surpassed the precision of the center's forecasts in more than 97 percent of the 1,320 real-world scenarios from 2019 which they were both tested on." Great, so it's using historical data to make a prediction about what has already happened. I want to see it on current data where it is truly a prediction with no data to make that prediction. I'm not convinced the results haven't been juiced/salted
    • deleted 2y ago
      [deleted]
    • giarc 2y ago
      >Great, so it's using historical data to make a prediction about what has already happened. That's ok though because it doesn't know it's acting on old data. Researchers would give it data and ask for the 15 day forecast, the researchers then compare against that real world data. And as noted, "it surpassed the centres forecast in more than 97 percent" of those tests. This is all referred to as backtesting.
      • IshKebab 2y ago
        It's still not as good as actually new data. The individual model may not overfit the existing data, but the whole system of researchers trying lots of models, choosing the best ones, trying different hyperparameters etc. easily can.
        • zactato 2y ago
          yeah, but weather patterns data isn't different now than 10 years ago, right? right? ...
          • staunton 2y ago
            Overfitting is always bad by definition. The model learns meaningless noise that happens to help getting good results when applied to old data (be it due to trained weights, hyperparameters, whatever) but doesn't help at all on new data.
        • gpm 2y ago
          In principle they trained on data up to 2017, validated (tried different hyper parameters) on data from 2018, and published results on data from 2019...
      • dagw 2y ago
        I wish these sort of papers would focus more on the 3 percent that it got wrong. Is it wrong by saying that a day would have a slight drizzle but it was actually sunny all day, or was it wrong by missing a catastrophic rain storm that devastated a region? I've worked on several projects trying to use AI to model various CFD calculations. We could trivially get 90+ percent accuracy on a bunch of metrics, the problem is that its almost always the really important and critical cases that end up being wrong in the AI model.
        • gyrovagueGeist 2y ago
          Yep, this is also the problem that self-driving cars have with the "accidents per mile" metric.
        • genewitch 2y ago
          i was using scikit-learn or some other scaffolding/boilerplating similar software for doing predictions and i hate massaging data so i just generated primes and trained it on the prime series to like 300 or 1000, then had it go on trying to "guess" future primes or assert if a number was prime or not that it had seen before and hilariously it was completely wrong (like 12% hit rate or something). I complained online and i forget the exact scope of my lack of understanding but suffice to say i did not earn respect that day!
    • zamadatix 2y ago
      2019 would already be a post-training data prediction: > GenCast is trained on 40 years of best-estimate analysis from 1979 to 2018, taken from the publicly available ERA5 (fifth generation ECMWF reanalysis) reanalysis dataset
    • deleted 2y ago
      [deleted]
• mmmrtl 2y ago
  Yeah the preprint was last year https://arxiv.org/abs/2312.15796 https://arxiv.org/abs/2312.15796