4 ms·

Why? Most weather and climate datasets - including ERA5 - are highly structured on regular latitude-longitude grids. Even if you were solely doing timeseries a

Why? Most weather and climate datasets - including ERA5 - are highly structured on regular latitude-longitude grids. Even if you were solely doing timeseries analyses for specific locations plucked from this grid, the strength of this sort of dataset is its intrinsic spatiotemporal structure and context, and it makes very little sense to completely destroy the dataset's structure unless you were solely and exclusively to extract point timeseries. And even then, you'd probably want to decimate the data pretty dramatically, since there is very little use case for, say, a point timeseries of surface temperature in the middle of the ocean! The vast majority of research and operational applications of datasets like ERA5 are probably better suited by leveraging cloud-optimized replicas of the original dataset, such as ARCO-ERA5 published on the Google Public Datasets program [1]. These versions of the dataset preserve the original structure, and chunk it in ways that are amenable to massively parallel access via cloud storage. In almost any case I've encountered in my career, a generically chunked Zarr-based archive of a dataset like this will be more than performant enough for the majority of use cases that one might care about. [1]: https://cloud.google.com/storage/docs/public-datasets/era5 https://cloud.google.com/storage/docs/public-datasets/era5

rabernat 2y ago
True, but in fact, the Google ERA5 public data suffers from the exact chunking problem described in the post: it's optimized for spatial queries, not timeseries queries. I just ran a benchmark, and it took me 20 minutes to pull a timeseries of a single variable at a single point! This highlights the needs for timeseries-optimized chunking if that is your anticipated usage pattern.
- bitschubser_ 2y ago
  a good source for ERA5 historical data is https://open-meteo.com/en/docs/historical-weather-api https://open-meteo.com/en/docs/historical-weather-api (not affiliated, just a happy user) you can also run open-meteo locally, its quite fast for spatial and timeseries queries
  - open-meteo 2y ago
    Creator of Open-Meteo here. There is small tutorial to setup ERA5 locally: https://github.com/open-meteo/open-data/tree/main/tutorial_download_era5 https://github.com/open-meteo/open-data/tree/main/tutorial_d... Under the hood Open-Meteo is using a custom file format with time-series chunking and specialised compression for low-frequency weather data. General purpose time-series databases do not even get close to this setup.
orhmeh09 2y ago
That might be nice if someone would do it and teach others to use it. Some labs have an RDBMS-based pipeline with published algorithms and data that nobody wants to try to reimplement (and which nobody would be paid to do). About the best improvement we could get was moving from an ancient version of MySQL to Postgres + PostGIS. I think Timescale would have helped. There were other reasons also to run locally due to privacy, cluster access, funds etc.
PolarizedPoutin 2y ago
The main reason why was that it's a personal project and I wanted to do everything on my home server so that I wouldn't have to pay for cloud resources, and so that I could learn Postgres, TimescaleDB, and eventuallly PostGIS. But as rabernat pointed out in his comment, pulling out a long time series from the cloud replica is also slow. And I know I eventually want to perform complex spatio-temporal queries, e.g. computing the 99% percentile of summer temperatures in Chile from 1940-1980. I don't doubt that a cloud replica can be faster, but it's at odds with my budget of $0 haha.
- roter 2y ago
  I too need to do percentiles. One option is loop through the grids but bin/histogram it. You'll get a really good 99% from a 1 Kelvin bin width. Also, I've found the diurnal profile from ERA5 analysis can be abysmal in some locations. ERA5-Land is much better, high resolution, though only available over... er... land. To your point about not relying on cloud. Noted in the Google option [1] link above: > Update Frequency: The ERA5 dataset is currently not refreshed in the Google Cloud Public Dataset Program. The program provides ERA5 data spanning from 1940 to May 2023. Another alternative, Amazon [2], also deprecated: > The provider of this dataset will no longer maintain this dataset. We are open to talking with anyone else who might be willing to provide this dataset to the community. [2] https://registry.opendata.aws/ecmwf-era5/ https://registry.opendata.aws/ecmwf-era5/
  - PolarizedPoutin 2y ago
    Thanks for pointing out the status of ERA5 on GCP and AWS. I'll have to try the binning and compare it with listing all the values and finding the 99th percentile! But yeah the spatial part of that query might be the tougher part. And yeah I also considered whether to use ERA5 or ERA5-Land. I don't remember if ERA5-Land was missing a variable or I just wanted to look at some ocean grid cells, but I decided to start with vanilla ERA5.

rabernat 2y ago

True, but in fact, the Google ERA5 public data suffers from the exact chunking problem described in the post: it's optimized for spatial queries, not timeseries queries. I just ran a benchmark, and it took me 20 minutes to pull a timeseries of a single variable at a single point! This highlights the needs for timeseries-optimized chunking if that is your anticipated usage pattern.

bitschubser_ 2y ago
a good source for ERA5 historical data is https://open-meteo.com/en/docs/historical-weather-api https://open-meteo.com/en/docs/historical-weather-api (not affiliated, just a happy user) you can also run open-meteo locally, its quite fast for spatial and timeseries queries
- open-meteo 2y ago
  Creator of Open-Meteo here. There is small tutorial to setup ERA5 locally: https://github.com/open-meteo/open-data/tree/main/tutorial_download_era5 https://github.com/open-meteo/open-data/tree/main/tutorial_d... Under the hood Open-Meteo is using a custom file format with time-series chunking and specialised compression for low-frequency weather data. General purpose time-series databases do not even get close to this setup.

orhmeh09 2y ago

That might be nice if someone would do it and teach others to use it. Some labs have an RDBMS-based pipeline with published algorithms and data that nobody wants to try to reimplement (and which nobody would be paid to do). About the best improvement we could get was moving from an ancient version of MySQL to Postgres + PostGIS. I think Timescale would have helped. There were other reasons also to run locally due to privacy, cluster access, funds etc.

PolarizedPoutin 2y ago

The main reason why was that it's a personal project and I wanted to do everything on my home server so that I wouldn't have to pay for cloud resources, and so that I could learn Postgres, TimescaleDB, and eventuallly PostGIS. But as rabernat pointed out in his comment, pulling out a long time series from the cloud replica is also slow. And I know I eventually want to perform complex spatio-temporal queries, e.g. computing the 99% percentile of summer temperatures in Chile from 1940-1980. I don't doubt that a cloud replica can be faster, but it's at odds with my budget of $0 haha.

roter 2y ago
I too need to do percentiles. One option is loop through the grids but bin/histogram it. You'll get a really good 99% from a 1 Kelvin bin width. Also, I've found the diurnal profile from ERA5 analysis can be abysmal in some locations. ERA5-Land is much better, high resolution, though only available over... er... land. To your point about not relying on cloud. Noted in the Google option [1] link above: > Update Frequency: The ERA5 dataset is currently not refreshed in the Google Cloud Public Dataset Program. The program provides ERA5 data spanning from 1940 to May 2023. Another alternative, Amazon [2], also deprecated: > The provider of this dataset will no longer maintain this dataset. We are open to talking with anyone else who might be willing to provide this dataset to the community. [2] https://registry.opendata.aws/ecmwf-era5/ https://registry.opendata.aws/ecmwf-era5/
- PolarizedPoutin 2y ago
  Thanks for pointing out the status of ERA5 on GCP and AWS. I'll have to try the binning and compare it with listing all the values and finding the 99th percentile! But yeah the spatial part of that query might be the tougher part. And yeah I also considered whether to use ERA5 or ERA5-Land. I don't remember if ERA5-Land was missing a variable or I just wanted to look at some ocean grid cells, but I decided to start with vanilla ERA5.