4 ms·
curious to know how you guys store and process the data and make meaningful inferences out of the data collected. Redshift? BigQuery? Datazip?
by pkhodiyar 2y ago
curious to know how you guys store and process the data and make meaningful inferences out of the data collected. Redshift? BigQuery? Datazip?
- avecchi-sorc 2y agoInternally yes we use a data warehouse to store the raw observations. We go through a similar process to traditional NWP to produce forecasts except with a few AI steps in between. Unfortunately, the standard currently is to store these as large multi-dimensional data files like grib2/netCDF/zarr. In case you’re curious, here’s where NOAA stores all their GFS related forecasts: https://registry.opendata.aws/noaa-gfs-bdp-pds/ https://registry.opendata.aws/noaa-gfs-bdp-pds/