3 ms·
anyone have experience using one of these array data stores to handle large amounts of weather forecast data? At my job we've come up with our own clever postgr
by brootstrap 9y ago
anyone have experience using one of these array data stores to handle large amounts of weather forecast data? At my job we've come up with our own clever postgres solution to handle the large amounts of gridded binary data. Essentially we are inserting large 3D arrays (a series of images where each pixel represents a geographic location and a value, like windspeed or temperature or rainfall) into our postgres. Our solution is solid but am always on the lookup for new novel approaches.
- deleted 9y ago[deleted]
- deleted 9y ago[deleted]
- jakebol 9y agoJake from TileDB, Inc. I think this would be an ideal workload for array data stores (NetCDF a standard in this area uses HDF5 under the hood). You have <N> number of attributes that you want per grid-point over time (and you want to append to the time dimension). If you are ingesting Grib2 files then you can take advantage of compression as well. An array data store like TileDB should offer advantages for fast access, as you can get a pointer directly to the stored array and do not have to access the (serialized) data over a socket, especially if you are only interested in a subarray of the dataset.
- oliviervg1 9y agoHi both, this is exactly something that I’m looking at doing. We’ve got about 10TB of netcdf data coming in everyday and we’re looking for a cost efficient data store to provide fast access to individual grid points. S3 has proven to be too slow. Any chance I could pick your brains about using either Postgres or TileDB? Thanks!
- jakebol 9y agoAbsolutely! Drop us a line at hello@tiledb.io and tell us a little more about the problem you are trying to solve and we can go from there.
- StanSeltser 9y agoStanislav Seltser, Petacube Inc ingesting data into postgres makes sense for sparce data but not for dense data because it is waisting a lot of of space due to storage of coordinates with every data point and every weather variable. If you are using NOOA grib2 forecast files those are dense . Not to mention losing compression in postgres. TileDB will store data compressed, the dimension coordinates themselves will be compressed, plus column storage (each NetCDF variable) will make retrieval of dense weather data blazingly fast as oppose to postgres where you will have to scan the whole table