rabernat

3 ms·

1. 1.▲
   by rabernat 2y ago

   > It's possible but not very cost-effective to maintain separately-chunked versions of these large geospatial datasets. Like all things in tech, it's about tradeoffs. S3 storage costs about $275 TB a year. Typical weather datas
2. 2.▲
   by rabernat 2y ago

   True, but in fact, the Google ERA5 public data suffers from the exact chunking problem described in the post: it's optimized for spatial queries, not timeseries queries. I just ran a benchmark, and it took me 20 minutes to pull a times
3. 3.▲
   by rabernat 2y ago

   Great post! Hi Ali! I think what's missing here is an analysis of what is gained by moving the weather data into a RDBMS. The motivation is to speed up queries. But what's the baseline? As someone very familiar with this tech land
4. 4.▲
   by rabernat 4y ago

   The Zarr format is used in some genomics workflows (see https://github.com/zarr-developers/community/issues/19 ) and supports a wide range of modern compressors (e.g. Zstd, Zlib, BZ2, LZMA, ZFPY, Blosc, as wel
5. 5.▲
   by rabernat 4y ago

   PyTorch and JAX are used heavily in climate science on the ML side. For more general analytics, not so much. Many of our users like to use Xarray as a high-level API. There has been some work to integrate Xarray with PyTorch ( https:/&
6. 6.▲
   by rabernat 4y ago

   Agree 100%. This is big part of the motivation behind our new startup Earthmover: https://earthmover.io/ Our mission is to make it easier to work with scientific data at scale in the cloud, focusing mainly on the climate, w