5 ms·

I'm friends with a meteorologist and the 15+ day forecast is the bane of their existence because you can't accurately forecast beyond a week so I would love to

I'm friends with a meteorologist and the 15+ day forecast is the bane of their existence because you can't accurately forecast beyond a week so I would love to know how they are measuring accuracy. The article doesn't say and I know the paper is going to go over my head.

n4r9 2y ago
I would guess that everyday they're comparing the current weather against the forecast from 15 days ago. Not a lot of data points to be sure, but perhaps enough to have confidence of very high accuracy.
- dgrin91 2y ago
  Alternatively the can do back testing - using historical data they feed a subset into the predictor, then compare it's predictions to actual history
- diego_sandoval 2y ago
  I don't think that's necessary. You can do a backtest for any point in the past, as long as you only use the data that was available until 15 days before the day being predicted.
  - mnau 2y ago
    So do they say. I am reminded of Google Flu Trends [0]. They likely also did similar "verification" and it didn't work. > The initial Google paper stated that the Google Flu Trends predictions were 97% accurate comparing with CDC data.[4] However subsequent reports asserted that Google Flu Trends' predictions have been very inaccurate, especially in two high-profile cases. Google Flu Trends failed to predict the 2009 spring pandemic[12] and over the interval 2011–2013 it consistently overestimated relative flu incidence, [0] https://en.wikipedia.org/wiki/Google_Flu_Trends https://en.wikipedia.org/wiki/Google_Flu_Trends
    - rcpt 2y ago
      Disclaimer I work at Google. One of the difficulties with using user data to understand society is that the company isn't a static entity. Engineers are always changing their algorithms for purposes that have nothing to do with the things you're trying to observe. For Google Flu Trends specifically here's a great paper https://gking.harvard.edu/files/gking/files/0314policyforumff.pdf https://gking.harvard.edu/files/gking/files/0314policyforumf...
    - genewitch 2y ago
      2013 was a bad year for h1n1 or whatever it was. It killed my sister while she was vacationing back east. I think it was aggressive but it was quite cold that winter and that may have completely screwed with their data/interpretation at that time. For instance, it snowed 5 times and stuck in Central Louisiana that winter (continuing into frebruary/march, whateer.. It's been colder since, but that year was a real outlier (in my experience on earth, this is my supposition, i have been trying to figure this out for 11 years)
  - margalabargala 2y ago
    This gets tricky. Once you look into your past, you're presumably looking at data that was used to generate your training corpus. So you would expect better accuracy on that than you would find on present/future predictions.
- YeGoblynQueenne 2y ago
  No need to guess. It's in the paper: https://www.nature.com/articles/s41586-024-08252-9 https://www.nature.com/articles/s41586-024-08252-9 See section "Baselines".
YeGoblynQueenne 2y ago
They're testing it on 2019 data. From the paper (https://www.nature.com/articles/s41586-024-08252-9 https://www.nature.com/articles/s41586-024-08252-9): >> We use 2019 as our test period, and, following the protocol in ref. 2, we initialize ML models using ERA5 at 06 UTC and 18 UTC, as these benefit from only 3 h of look-ahead (with the exception of sea surface temperature, which in ERA5 is updated once per 24 h). This ensures ML models are not afforded an unfair advantage by initializing from states with longer look-ahead windows. See Baselines section in the paper that explains the methodology in more depth. They basically feed the competing models with data from weather stations and predict the weather in a certain time period. Then they compare the prediction with the ground truth from that period. Plot twist: they measure accuracy in predicting the weather 5 years in the past.
beernet 2y ago
> you can't accurately forecast beyond a week Totally wrong. You cannot generalize such a statement because it depends on the micro and macro weather conditions. A very stable situation makes it very easy to forecast one week and beyond. On the other hand, there can be situations where you cannot accurately predict the next 12 hours (e.g. cold air pool).
- fransje26 2y ago
  > Totally wrong. You cannot generalize such a statement because it depends on the micro and macro weather conditions. A very stable situation makes it very easy to forecast one week and beyond. On the other hand, there can be situations where you cannot accurately predict the next 12 hours (e.g. cold air pool). And that is exactly where these AI models will break down. They will "shine" (or fool us) with how well they predict the stable situations, and will produce utter rubbish when the high, turbulent and dynamic weather fronts make prediction difficult. But of course, if your weather is "stable" 80% of the time, you can use those shiny examples to sell your tool, and count on user forgetfulness to get away with the 20% of nonsense predicted the rest of the time.
lukan 2y ago
The paper seems quite readable to me and I also lack the training. But this point is adressed in the beginning. "The highly non-linear physics of weather means that small initial uncertainties and errors can rapidly grow into large uncertainties about the future. Making important decisions often requires knowing not just a single probable scenario but the range of possible scenarios and how likely they are to occur."
DrBazza 2y ago
I would hope it's the 'persistence forecast' + x % better. https://en.wikipedia.org/wiki/Weather_forecasting#Persistence https://en.wikipedia.org/wiki/Weather_forecasting#Persistenc... IIRC the Metoffice in the UK does/did pay bonuses to staff based on modelling exceeding that criteria in a calendar year. Again, IIRC, in the UK the persistence forecast suggests something around 200-250 days of the year have the same weather as the previous day.

n4r9 2y ago

I would guess that everyday they're comparing the current weather against the forecast from 15 days ago. Not a lot of data points to be sure, but perhaps enough to have confidence of very high accuracy.

dgrin91 2y ago
Alternatively the can do back testing - using historical data they feed a subset into the predictor, then compare it's predictions to actual history
diego_sandoval 2y ago
I don't think that's necessary. You can do a backtest for any point in the past, as long as you only use the data that was available until 15 days before the day being predicted.
- mnau 2y ago
  So do they say. I am reminded of Google Flu Trends [0]. They likely also did similar "verification" and it didn't work. > The initial Google paper stated that the Google Flu Trends predictions were 97% accurate comparing with CDC data.[4] However subsequent reports asserted that Google Flu Trends' predictions have been very inaccurate, especially in two high-profile cases. Google Flu Trends failed to predict the 2009 spring pandemic[12] and over the interval 2011–2013 it consistently overestimated relative flu incidence, [0] https://en.wikipedia.org/wiki/Google_Flu_Trends https://en.wikipedia.org/wiki/Google_Flu_Trends
  - rcpt 2y ago
    Disclaimer I work at Google. One of the difficulties with using user data to understand society is that the company isn't a static entity. Engineers are always changing their algorithms for purposes that have nothing to do with the things you're trying to observe. For Google Flu Trends specifically here's a great paper https://gking.harvard.edu/files/gking/files/0314policyforumff.pdf https://gking.harvard.edu/files/gking/files/0314policyforumf...
  - genewitch 2y ago
    2013 was a bad year for h1n1 or whatever it was. It killed my sister while she was vacationing back east. I think it was aggressive but it was quite cold that winter and that may have completely screwed with their data/interpretation at that time. For instance, it snowed 5 times and stuck in Central Louisiana that winter (continuing into frebruary/march, whateer.. It's been colder since, but that year was a real outlier (in my experience on earth, this is my supposition, i have been trying to figure this out for 11 years)
- margalabargala 2y ago
  This gets tricky. Once you look into your past, you're presumably looking at data that was used to generate your training corpus. So you would expect better accuracy on that than you would find on present/future predictions.
YeGoblynQueenne 2y ago
No need to guess. It's in the paper: https://www.nature.com/articles/s41586-024-08252-9 https://www.nature.com/articles/s41586-024-08252-9 See section "Baselines".

YeGoblynQueenne 2y ago

They're testing it on 2019 data. From the paper (https://www.nature.com/articles/s41586-024-08252-9 https://www.nature.com/articles/s41586-024-08252-9): >> We use 2019 as our test period, and, following the protocol in ref. 2, we initialize ML models using ERA5 at 06 UTC and 18 UTC, as these benefit from only 3 h of look-ahead (with the exception of sea surface temperature, which in ERA5 is updated once per 24 h). This ensures ML models are not afforded an unfair advantage by initializing from states with longer look-ahead windows. See Baselines section in the paper that explains the methodology in more depth. They basically feed the competing models with data from weather stations and predict the weather in a certain time period. Then they compare the prediction with the ground truth from that period. Plot twist: they measure accuracy in predicting the weather 5 years in the past.

beernet 2y ago

> you can't accurately forecast beyond a week Totally wrong. You cannot generalize such a statement because it depends on the micro and macro weather conditions. A very stable situation makes it very easy to forecast one week and beyond. On the other hand, there can be situations where you cannot accurately predict the next 12 hours (e.g. cold air pool).

fransje26 2y ago
> Totally wrong. You cannot generalize such a statement because it depends on the micro and macro weather conditions. A very stable situation makes it very easy to forecast one week and beyond. On the other hand, there can be situations where you cannot accurately predict the next 12 hours (e.g. cold air pool). And that is exactly where these AI models will break down. They will "shine" (or fool us) with how well they predict the stable situations, and will produce utter rubbish when the high, turbulent and dynamic weather fronts make prediction difficult. But of course, if your weather is "stable" 80% of the time, you can use those shiny examples to sell your tool, and count on user forgetfulness to get away with the 20% of nonsense predicted the rest of the time.

lukan 2y ago

The paper seems quite readable to me and I also lack the training. But this point is adressed in the beginning. "The highly non-linear physics of weather means that small initial uncertainties and errors can rapidly grow into large uncertainties about the future. Making important decisions often requires knowing not just a single probable scenario but the range of possible scenarios and how likely they are to occur."

DrBazza 2y ago

I would hope it's the 'persistence forecast' + x % better. https://en.wikipedia.org/wiki/Weather_forecasting#Persistence https://en.wikipedia.org/wiki/Weather_forecasting#Persistenc... IIRC the Metoffice in the UK does/did pay bonuses to staff based on modelling exceeding that criteria in a calendar year. Again, IIRC, in the UK the persistence forecast suggests something around 200-250 days of the year have the same weather as the previous day.