4 ms·
Very cool! Random weird suggestion: you could get very substantial further space savings (2x-16x) by storing the data in a compressed texture format such as AST
by modeless 5y ago
Very cool! Random weird suggestion: you could get very substantial further space savings (2x-16x) by storing the data in a compressed texture format such as ASTC HDR. What makes compressed textures different from other compressed image formats is that they are designed for random access of the compressed data, with fixed size blocks. Compression is lossy but you can choose the compression level to match your data. I expect that weather data is very suitable for compression so you should be able to get a large benefit with few artifacts.
- meteo-jeff 5y agoThanks for pointing me towards ASTC HDR. I did not know this one. Compression will be an important part to build a longer historic database. I tested zstd, but it only had a ratio of 30%. Currently I am using fixed size files, because I update all weather data "in-place". I can calculate which bytes to read for a given geographic location. If data is compressed, this would be very hard to do.
- modeless 5y agoYes, this is why I suggested compressed textures, they are fixed size too. You know exactly which bytes to read (or write) for any given pixel. But they can still achieve high compression ratios assuming the data is suitable, which I expect weather data is.
- meteo-jeff 5y agoNow I got it! Sorry, I did not get your point in the beginning. Compressed textures could be really interesting. So far I have no experience in this regard, but I will look into it