5 ms·

Are all the inputs (from buoys, weather balloons, stations, etc) from decades of history stored, as well as past daily forecasts of existing weather models, so

Are all the inputs (from buoys, weather balloons, stations, etc) from decades of history stored, as well as past daily forecasts of existing weather models, so that this AI algorithm (and any future new ones) can be run across historic data and compared in terms of performance to existing methods? Is there a big Clearinghouse for this data? Kind of like how fintech algos can be run against historic stock market data to evaluate them.

lgeorget 2y ago
The World Meteorological Organization has data that national weather services exchange to make forecast. Apart from that, its on a country-by-country basis. In France for instance Météo-France has released all of its historical data in January 2024: https://www.data.gouv.fr/fr/organizations/meteo-france/#/datasets https://www.data.gouv.fr/fr/organizations/meteo-france/#/dat...
- graemep 2y ago
  But national models go beyond there borders and usually have some modelling outside so the exchanges will cover quite a lot of the world. For example, the UK's metoffice has a low resolution medium term global model: https://www.metoffice.gov.uk/research/approach/modelling-systems/unified-model/weather-forecasting https://www.metoffice.gov.uk/research/approach/modelling-sys...
- cship2 2y ago
  How does windy do it? Did they just scrap all the sites or api. Would be great to have global dump every 1 minute of all the countries. Or have it have in a radio broadcast.
  - jcd000 2y ago
    Windy uses openmeteo, which in turn aggregates many weather providers under one roof/api
mschuster91 2y ago
> Is there a big Clearinghouse for this data? Many commercial and even third party governments rely on the data from NOAA and its archives, on top of that the EU runs its own EUMETSAT fleet of data, plus a ton of national services - unfortunately, the result is there's a looooot of datasets. NOAA's dataset is public domain [1], EUMETSAT only requires attribution for most of its data [2]. On top of that you got the EU's Climate Data store [3], ECMWF [4], and ECA&D [5]. The service that many private weather services provide is to aggregate and weigh all of the publicly available datasets, and some also add in data from their own ground stations, commercially licensed "realtime" data from governmental services, and their own models as well. The interesting question is what DOGE will do regarding NOAA - it is increasingly possible that NOAA will shut down, either to be turned into a pay-for-play model, replaced by private services, or just carelessly dropped in its entirety. [1] https://www.ncei.noaa.gov/archive https://www.ncei.noaa.gov/archive [2] https://user.eumetsat.int/resources/user-guides/data-registration-and-licensing https://user.eumetsat.int/resources/user-guides/data-registr... [3] https://cds.climate.copernicus.eu/ https://cds.climate.copernicus.eu/ [4] https://www.ecmwf.int/en/forecasts/datasets https://www.ecmwf.int/en/forecasts/datasets [5] https://www.ecad.eu/dailydata/ https://www.ecad.eu/dailydata/
- Loughla 2y ago
  NOAA saves lives. Cutting that organization would be an absolute slap in the face to rural trump voters who rely on that system for weather alerts.
  - deleted 2y ago
    [deleted]
- sunshinesnacks 2y ago
  > The service that many private weather services provide is to aggregate and weigh all of the publicly available datasets Yes, and more people need to understand this. Too many people seem to think that commercial services won’t be impacted if NOAA stops doing what they do.
- K0balt 2y ago
_joel 2y ago
> Kind of like how fintech algos can be run against historic stock market data to evaluate them. Backtesting, that's called.
- axismundi 2y ago
  In weather it's called hindcast
  - trillic 2y ago
    afttest
NitpickLawyer 2y ago
> Are all the inputs (from buoys, weather balloons, stations, etc) from decades of history stored I was also thinking about smartphones. They have barometric data, and while it might vary from phone to phone, I'm sure something like a kalman filter + historic data could do something there. Think about gathering all the data from "stationary" phones, correlate that with weather sat data, and with real "ground truth" weather stations, and then go back 30 - 60 min / a day, and see what comes out.
- Havoc 2y ago
  >smartphones. They have barometric data For anyone else having a TIL moment: It's apparently for vertical position and supposedly sub meter accurate. o_O
  - p_l 2y ago
    Depends on smartphone (and smartwatches). Not all have it, at times it disappeared from brands that had it earlier. The smartwatch series I use explicitly include it because it's essentially the "all in one" version in a series that had smartwatches designed for aircraft use (including military versions where they serve as backup cabin pressure warning, apparently)
    - usrusr 2y ago
      Smartwatches include barometers because they enable elevation measurement far more accurate than than GNSS (GPS and GPS-alikes, they tend to be noticeably less accurate in the vertical than in the horizontal). In particular when both are combined: barometry for high frequency changes (going up a hill) continuously calibrated against a long term average of the GNSS elevation to filter out low frequency changes in barometry (weather influence). But back to device could weather observation, this would require continuous GNSS, devices don't do that (network location would not be good enough).
- orion138 2y ago
  Dark Sky used that data for hyper local forecasting… https://news.ycombinator.com/item?id=22740466 https://news.ycombinator.com/item?id=22740466 https://www.theverge.com/2015/6/22/8822767/dark-sky-weather-app-crowdsourced-data https://www.theverge.com/2015/6/22/8822767/dark-sky-weather-...
randoKalrisian 2y ago
[dead]
TheJoeMan 2y ago
It would also be nice to have a historical store of the weather predictions and not just instantaneous parameters. But I'm not aware of such record, perhaps because weathermen don't want a record of their (mis)predictions...
- ano-ther 2y ago
  Oh they absolutely do track their predictions and how the different models perform. That’s how they keep improving. This is a lay-person overview which cites some of the in-depth studies: https://ourworldindata.org/weather-forecasts https://ourworldindata.org/weather-forecasts
interludead 2y ago
Getting high-quality, harmonized data at global scale can still be a challenge, especially when it comes to observational coverage in developing regions
wewxjfq 2y ago
There's climate reanalysis, which combines historical observations with weather models to get clean data of past weather conditions, which is then used by researchers for various purposes. Most notably is ERA5 by ECMWF. [0] https://en.wikipedia.org/wiki/ECMWF_re-analysis https://en.wikipedia.org/wiki/ECMWF_re-analysis
serialdev 2y ago
Is there an equivalent modelling approach for earthquake prediction? A data repository for it widely shared on the same line as the top comment would work
roelschroeven 2y ago
What data is used to train this AI? The article doesn't say anything about that (tough I have to admit I didn't read it super carefully). My first thought would be exactly all this historical data, but then you can't use that same data to test the AI's performance. Are different subsets of the available historical data used for training vs testing?
- axismundi 2y ago
  It's most likely ERA5: https://cds.climate.copernicus.eu/datasets/reanalysis-era5-single-levels https://cds.climate.copernicus.eu/datasets/reanalysis-era5-s...
  - scellus 2y ago
    No, they say end-to-end, meaning they use raw obsevations. Most or all other medium-range models start with ERA5. There's a paper from Norway that tried end-to-end, but their results were not spectacular. That's the aim of many though, including ECMWF. Note that ECMWF already has their AIFS in production, so AI weather prediction is pretty mainstream nowadays. Google has a local nowcast model that uses raw observations, in production, but that's a different genre of forecasting than the medium-range models of Aardvark.
    - counters 2y ago
      > Google has a local nowcast model that uses raw observations, in production, but that's a different genre of forecasting than the medium-range models of Aardvark. It's very clear from the MetNet announcement blog[1] that they require HRRR or other NWP output at runtime. [1]: https://research.google/blog/metnet-3-a-state-of-the-art-neural-weather-model-available-in-google-products/ https://research.google/blog/metnet-3-a-state-of-the-art-neu...
    - wafngar 2y ago
      They train with ERA5 and observations.
guhidalg 2y ago
There are several places you can get this data, both historical and in near-realtime. I work on one of them: https://planetarycomputer.microsoft.com/catalog?filter=weather https://planetarycomputer.microsoft.com/catalog?filter=weath...
counters 2y ago
> Is there a big Clearinghouse for this data? The short answer is, "no." There are some projects like the "NNJA-AI" project at Brightband[1] which is attempting to create such a clearing house in order to focus research efforts across the community. [1]: https://www.brightband.com/data/nnja-ai https://www.brightband.com/data/nnja-ai

lgeorget 2y ago

The World Meteorological Organization has data that national weather services exchange to make forecast. Apart from that, its on a country-by-country basis. In France for instance Météo-France has released all of its historical data in January 2024: https://www.data.gouv.fr/fr/organizations/meteo-france/#/datasets https://www.data.gouv.fr/fr/organizations/meteo-france/#/dat...

graemep 2y ago
But national models go beyond there borders and usually have some modelling outside so the exchanges will cover quite a lot of the world. For example, the UK's metoffice has a low resolution medium term global model: https://www.metoffice.gov.uk/research/approach/modelling-systems/unified-model/weather-forecasting https://www.metoffice.gov.uk/research/approach/modelling-sys...
cship2 2y ago
How does windy do it? Did they just scrap all the sites or api. Would be great to have global dump every 1 minute of all the countries. Or have it have in a radio broadcast.
- jcd000 2y ago
  Windy uses openmeteo, which in turn aggregates many weather providers under one roof/api

mschuster91 2y ago

> Is there a big Clearinghouse for this data? Many commercial and even third party governments rely on the data from NOAA and its archives, on top of that the EU runs its own EUMETSAT fleet of data, plus a ton of national services - unfortunately, the result is there's a looooot of datasets. NOAA's dataset is public domain [1], EUMETSAT only requires attribution for most of its data [2]. On top of that you got the EU's Climate Data store [3], ECMWF [4], and ECA&D [5]. The service that many private weather services provide is to aggregate and weigh all of the publicly available datasets, and some also add in data from their own ground stations, commercially licensed "realtime" data from governmental services, and their own models as well. The interesting question is what DOGE will do regarding NOAA - it is increasingly possible that NOAA will shut down, either to be turned into a pay-for-play model, replaced by private services, or just carelessly dropped in its entirety. [1] https://www.ncei.noaa.gov/archive https://www.ncei.noaa.gov/archive [2] https://user.eumetsat.int/resources/user-guides/data-registration-and-licensing https://user.eumetsat.int/resources/user-guides/data-registr... [3] https://cds.climate.copernicus.eu/ https://cds.climate.copernicus.eu/ [4] https://www.ecmwf.int/en/forecasts/datasets https://www.ecmwf.int/en/forecasts/datasets [5] https://www.ecad.eu/dailydata/ https://www.ecad.eu/dailydata/

Loughla 2y ago
NOAA saves lives. Cutting that organization would be an absolute slap in the face to rural trump voters who rely on that system for weather alerts.
- deleted 2y ago
  [deleted]
sunshinesnacks 2y ago
> The service that many private weather services provide is to aggregate and weigh all of the publicly available datasets Yes, and more people need to understand this. Too many people seem to think that commercial services won’t be impacted if NOAA stops doing what they do.
K0balt 2y ago

_joel 2y ago

> Kind of like how fintech algos can be run against historic stock market data to evaluate them. Backtesting, that's called.

axismundi 2y ago
In weather it's called hindcast
- trillic 2y ago
  afttest

NitpickLawyer 2y ago

> Are all the inputs (from buoys, weather balloons, stations, etc) from decades of history stored I was also thinking about smartphones. They have barometric data, and while it might vary from phone to phone, I'm sure something like a kalman filter + historic data could do something there. Think about gathering all the data from "stationary" phones, correlate that with weather sat data, and with real "ground truth" weather stations, and then go back 30 - 60 min / a day, and see what comes out.

Havoc 2y ago
>smartphones. They have barometric data For anyone else having a TIL moment: It's apparently for vertical position and supposedly sub meter accurate. o_O
- p_l 2y ago
  Depends on smartphone (and smartwatches). Not all have it, at times it disappeared from brands that had it earlier. The smartwatch series I use explicitly include it because it's essentially the "all in one" version in a series that had smartwatches designed for aircraft use (including military versions where they serve as backup cabin pressure warning, apparently)
  - usrusr 2y ago
    Smartwatches include barometers because they enable elevation measurement far more accurate than than GNSS (GPS and GPS-alikes, they tend to be noticeably less accurate in the vertical than in the horizontal). In particular when both are combined: barometry for high frequency changes (going up a hill) continuously calibrated against a long term average of the GNSS elevation to filter out low frequency changes in barometry (weather influence). But back to device could weather observation, this would require continuous GNSS, devices don't do that (network location would not be good enough).
orion138 2y ago
Dark Sky used that data for hyper local forecasting… https://news.ycombinator.com/item?id=22740466 https://news.ycombinator.com/item?id=22740466 https://www.theverge.com/2015/6/22/8822767/dark-sky-weather-app-crowdsourced-data https://www.theverge.com/2015/6/22/8822767/dark-sky-weather-...

randoKalrisian 2y ago

[dead]

TheJoeMan 2y ago

It would also be nice to have a historical store of the weather predictions and not just instantaneous parameters. But I'm not aware of such record, perhaps because weathermen don't want a record of their (mis)predictions...

ano-ther 2y ago
Oh they absolutely do track their predictions and how the different models perform. That’s how they keep improving. This is a lay-person overview which cites some of the in-depth studies: https://ourworldindata.org/weather-forecasts https://ourworldindata.org/weather-forecasts

interludead 2y ago

Getting high-quality, harmonized data at global scale can still be a challenge, especially when it comes to observational coverage in developing regions

wewxjfq 2y ago

There's climate reanalysis, which combines historical observations with weather models to get clean data of past weather conditions, which is then used by researchers for various purposes. Most notably is ERA5 by ECMWF. [0] https://en.wikipedia.org/wiki/ECMWF_re-analysis https://en.wikipedia.org/wiki/ECMWF_re-analysis

serialdev 2y ago

Is there an equivalent modelling approach for earthquake prediction? A data repository for it widely shared on the same line as the top comment would work

roelschroeven 2y ago

What data is used to train this AI? The article doesn't say anything about that (tough I have to admit I didn't read it super carefully). My first thought would be exactly all this historical data, but then you can't use that same data to test the AI's performance. Are different subsets of the available historical data used for training vs testing?

axismundi 2y ago
It's most likely ERA5: https://cds.climate.copernicus.eu/datasets/reanalysis-era5-single-levels https://cds.climate.copernicus.eu/datasets/reanalysis-era5-s...
- scellus 2y ago
  No, they say end-to-end, meaning they use raw obsevations. Most or all other medium-range models start with ERA5. There's a paper from Norway that tried end-to-end, but their results were not spectacular. That's the aim of many though, including ECMWF. Note that ECMWF already has their AIFS in production, so AI weather prediction is pretty mainstream nowadays. Google has a local nowcast model that uses raw observations, in production, but that's a different genre of forecasting than the medium-range models of Aardvark.
  - counters 2y ago
    > Google has a local nowcast model that uses raw observations, in production, but that's a different genre of forecasting than the medium-range models of Aardvark. It's very clear from the MetNet announcement blog[1] that they require HRRR or other NWP output at runtime. [1]: https://research.google/blog/metnet-3-a-state-of-the-art-neural-weather-model-available-in-google-products/ https://research.google/blog/metnet-3-a-state-of-the-art-neu...
  - wafngar 2y ago
    They train with ERA5 and observations.

guhidalg 2y ago

There are several places you can get this data, both historical and in near-realtime. I work on one of them: https://planetarycomputer.microsoft.com/catalog?filter=weather https://planetarycomputer.microsoft.com/catalog?filter=weath...

counters 2y ago

> Is there a big Clearinghouse for this data? The short answer is, "no." There are some projects like the "NNJA-AI" project at Brightband[1] which is attempting to create such a clearing house in order to focus research efforts across the community. [1]: https://www.brightband.com/data/nnja-ai https://www.brightband.com/data/nnja-ai