3 ms·

Looks really awesome! Very quick responses. I would be interested in seeing the implementation of the service, interesting choice going with Swift. I'm guessin

Looks really awesome! Very quick responses. I would be interested in seeing the implementation of the service, interesting choice going with Swift. I'm guessing your using something like Vapor for hosting the API? How are the files designed? I'm guessing you have some cheap way of mapping (lat, long) into a file location? Maybe just by truncating the coordinates to your precision and mapping them onto the file? Using some fancy-pants Uber Hexagons[0]. How is the forecast delivered? Hmm! Many questions :-). I've been thinking lately of similar (but not for weather) "geographical" API's, and how to store/query data cheaply, so this interested me :-) [0]: https://github.com/uber/h3 https://github.com/uber/h3

meteo-jeff 5y ago
Yes, the API part is using Vapor. But there is not much code that actually relies on Vapor. It is mostly doing request parameter parsing and routing. I use simple 2D lat lon grids for different regions and resolutions. E.g. the global grid uses 2879x1441 pixel grid. The third dimension is time. All data are stored directly as floating points on disk. Because I know the dimensions of the grid, I can read data exactly at the correct position on disk. I use Float16 as well, which saves me 50% disk space compared to Float32. Fancy hexagons like H3 are not necessary. They could reduce storage by ~30%, but require much more effort and I have to "modify" the forecast. I keep forecast data from national weather services as untouched as possible.
- modeless 5y ago
  Very cool! Random weird suggestion: you could get very substantial further space savings (2x-16x) by storing the data in a compressed texture format such as ASTC HDR. What makes compressed textures different from other compressed image formats is that they are designed for random access of the compressed data, with fixed size blocks. Compression is lossy but you can choose the compression level to match your data. I expect that weather data is very suitable for compression so you should be able to get a large benefit with few artifacts.
  - meteo-jeff 5y ago
    Thanks for pointing me towards ASTC HDR. I did not know this one. Compression will be an important part to build a longer historic database. I tested zstd, but it only had a ratio of 30%. Currently I am using fixed size files, because I update all weather data "in-place". I can calculate which bytes to read for a given geographic location. If data is compressed, this would be very hard to do.
    - modeless 5y ago
      Yes, this is why I suggested compressed textures, they are fixed size too. You know exactly which bytes to read (or write) for any given pixel. But they can still achieve high compression ratios assuming the data is suitable, which I expect weather data is.
      - meteo-jeff 5y ago
        Now I got it! Sorry, I did not get your point in the beginning. Compressed textures could be really interesting. So far I have no experience in this regard, but I will look into it

meteo-jeff 5y ago

Yes, the API part is using Vapor. But there is not much code that actually relies on Vapor. It is mostly doing request parameter parsing and routing. I use simple 2D lat lon grids for different regions and resolutions. E.g. the global grid uses 2879x1441 pixel grid. The third dimension is time. All data are stored directly as floating points on disk. Because I know the dimensions of the grid, I can read data exactly at the correct position on disk. I use Float16 as well, which saves me 50% disk space compared to Float32. Fancy hexagons like H3 are not necessary. They could reduce storage by ~30%, but require much more effort and I have to "modify" the forecast. I keep forecast data from national weather services as untouched as possible.

modeless 5y ago
Very cool! Random weird suggestion: you could get very substantial further space savings (2x-16x) by storing the data in a compressed texture format such as ASTC HDR. What makes compressed textures different from other compressed image formats is that they are designed for random access of the compressed data, with fixed size blocks. Compression is lossy but you can choose the compression level to match your data. I expect that weather data is very suitable for compression so you should be able to get a large benefit with few artifacts.
- meteo-jeff 5y ago
  Thanks for pointing me towards ASTC HDR. I did not know this one. Compression will be an important part to build a longer historic database. I tested zstd, but it only had a ratio of 30%. Currently I am using fixed size files, because I update all weather data "in-place". I can calculate which bytes to read for a given geographic location. If data is compressed, this would be very hard to do.
  - modeless 5y ago
    Yes, this is why I suggested compressed textures, they are fixed size too. You know exactly which bytes to read (or write) for any given pixel. But they can still achieve high compression ratios assuming the data is suitable, which I expect weather data is.
    - meteo-jeff 5y ago
      Now I got it! Sorry, I did not get your point in the beginning. Compressed textures could be really interesting. So far I have no experience in this regard, but I will look into it