3 ms·
This should be a benchmark. Could someone post me to where I can download the whole dataset?
by nojvek 3y ago
This should be a benchmark.
Could someone post me to where I can download the whole dataset?
- PolarizedPoutin 3y agoThe full dataset is quite huge (~9 petabytes and growing) out of which I'm using just ~8 terabytes. The data is freely available from the Climate Change Service [1] which has a nice API but download speeds can be a bit slow. NCAR's Research Data Archive [2] provides some of the data (as pre-generated NetCDF files) but at higher download speeds. It's not super well documented but I hosted the Python scripts I used to download the data on the accompanying GitHub repository [3]. [1]: https://cds.climate.copernicus.eu/cdsapp#!/dataset/reanalysis-era5-single-levels?tab=overview https://cds.climate.copernicus.eu/cdsapp#!/dataset/reanalysi... [2]: https://rda.ucar.edu/datasets/ds633-0/ https://rda.ucar.edu/datasets/ds633-0/ [3]: https://github.com/ali-ramadhan/timescaledb-insert-benchmarks/tree/main/download_data https://github.com/ali-ramadhan/timescaledb-insert-benchmark...