5 ms·

Smaller Satellite Images

44 points by marklit 2y ago

kelseyfrog 2y ago
Better compression schemes for GeoTIFFs is great and all - I'm not going to complain about that; however, I'd like to add a little more context from a data engineering perspective especially for data in transit. As we all know, the best way to speed up some processes is by avoiding the step entirely, and as much as better compression schemes for GeoTIFFs shrink transport sizes, avoiding transfer of rasters reduces the cost to zero. Internally tiled GeoTIFFs (Cloud Optimized GeoTIFFs[1] in particular) allow portions of GeoTIFFs to be read using range reads instead. This can dramatically reduce transfer costs when only sections of the raster are required and pair well when used with vector data for look ups. In the specific case used by the author, omitting water-only tiles would potentially cut transfer costs by half. Additionally, if using multi-spectral imagery, band re-ordering can be an important subsequent optimization if certain indices are known in advance to be used dis-proportionally more. There's a lot of fun to be had in optimizing the heck out of imagery data and this barely scratches the surface. 1. https://cogeo.org/ https://cogeo.org/
mistrial9 2y ago
webp ?
unwind 2y ago
Ah, the "My Workstation" guy. I wonder if that text could be factored out and linked to in a sidebar or something (it's in every article I've seen from him, here).
- progbits 2y ago
  Presumably the workstation changes over time :) I like benchmarks that properly describe their environment for posterity.
  - fourteenfour 2y ago
    Sure, to me it seems overly verbose in ways that are not relevant to the article. 6 GHz i9-14900K CPU Nvidia GTX 1080 GPU 96 GB of DDR5 RAM ASRock Z790 Pro RS Motherboard Crucial T700 4 TB NVMe M.2 SSD 1,200-watt Corsair Power Supply Ubuntu 22 LTS via Microsoft's Ubuntu for Windows on Windows 11 Pro
jarl-ragnar 2y ago
I followed a similar approach recently to create map layers with Sentinel-2 imagery. Webp encoded tiles extracted from the CoGs available on AWS then embedded within a pmtiles file. I was seeing almost an order of magnitude reduction in file size compared to the CoGs with no visible image degradation.

44 points by marklit 2y ago

Better compression schemes for GeoTIFFs is great and all - I'm not going to complain about that; however, I'd like to add a little more context from a data engineering perspective especially for data in transit. As we all know, the best way to speed up some processes is by avoiding the step entirely, and as much as better compression schemes for GeoTIFFs shrink transport sizes, avoiding transfer of rasters reduces the cost to zero. Internally tiled GeoTIFFs (Cloud Optimized GeoTIFFs[1] in particular) allow portions of GeoTIFFs to be read using range reads instead. This can dramatically reduce transfer costs when only sections of the raster are required and pair well when used with vector data for look ups. In the specific case used by the author, omitting water-only tiles would potentially cut transfer costs by half. Additionally, if using multi-spectral imagery, band re-ordering can be an important subsequent optimization if certain indices are known in advance to be used dis-proportionally more. There's a lot of fun to be had in optimizing the heck out of imagery data and this barely scratches the surface. 1. https://cogeo.org/ https://cogeo.org/

mistrial9 2y ago

webp ?

unwind 2y ago

Ah, the "My Workstation" guy. I wonder if that text could be factored out and linked to in a sidebar or something (it's in every article I've seen from him, here).

progbits 2y ago
Presumably the workstation changes over time :) I like benchmarks that properly describe their environment for posterity.
- fourteenfour 2y ago
  Sure, to me it seems overly verbose in ways that are not relevant to the article. 6 GHz i9-14900K CPU Nvidia GTX 1080 GPU 96 GB of DDR5 RAM ASRock Z790 Pro RS Motherboard Crucial T700 4 TB NVMe M.2 SSD 1,200-watt Corsair Power Supply Ubuntu 22 LTS via Microsoft's Ubuntu for Windows on Windows 11 Pro

jarl-ragnar 2y ago

I followed a similar approach recently to create map layers with Sentinel-2 imagery. Webp encoded tiles extracted from the CoGs available on AWS then embedded within a pmtiles file. I was seeing almost an order of magnitude reduction in file size compared to the CoGs with no visible image degradation.