3 ms·

This is ridiculously cool, demo is slick and fast! I work pretty extensively with tiling pipelines and GeoTiffs right now as I'm building out a mass repository

by Soupy 3y ago

This is ridiculously cool, demo is slick and fast! I work pretty extensively with tiling pipelines and GeoTiffs right now as I'm building out a mass repository and platform for historical map and aerial analysis (https://pastmaps.com https://pastmaps.com - still very early so don't judge it too hard please) As part of this work, I similarly hit problems with using the raw GeoTiff files as my source and found that I was able to build some custom tiling hooks into MapLibre coupled with http range queries on the static files hosted in S3 to bypass the need for tiling. It does push the compute to the clients but I've found it's actually pretty fast even on older mobile devices. If this MapLibre GeoTiff source support is at all of interest, I'm happy to share some basic code or even open source some of that work. Here I was thinking I was the only weird dude on the internet messing with this stuff :D

tppiotrowski 3y ago
Ah. I hadn't even thought that I could do byte offsets on the fly. In general I would err on the side of over-sharing because until ChatGPT came along I struggled to understand what tools and options were to generate tiles as there wasn't much data I could find online. One thing specific in my case: the LiDAR GeoTiffs are in imperial feet with a 32 bit floating point precision. If you take the elevation range from sea level to Everest in meters (8848) and pack it into an int16, you can get .2 meter precision. That's plenty for ShadeMap so converting from float32 to int16 should save half the cloud storage space in theory and more when taking PNG compression into account.
- Soupy 3y ago
  This code is rough, and that's an understatement, but here's a basic gist showing how I'm doing what I'm doing in case it helps you in any way: https://gist.github.com/craigsc/fdb867f8971ff5b4ae42de4e0d7c229e https://gist.github.com/craigsc/fdb867f8971ff5b4ae42de4e0d7c... I'm similarly using R2 as my static hosting backend and it's been pretty fast and seamless note that i'm using the geotiff.js and fast-png libraries for the heavy-lifting behind the range-queries and the client-side png encoding. why reinvent the wheel, right?
  - blacha 3y ago
    Your browser has a very powerful image decoder built into it, offloading the PNG decoding into Javascript is very resource hungry. Using maplibre (or any map viewer) you can load blobs of image data out of a tiff and use `Image` or `Canvas` to render the data onto a map. Its even easier if the tiffs are already Cloud optimized as they perfectly align to a 1-to-1 map tile and they don't need to be rescaled, you can then just render the images onto the map. eg here is a viewer that loads webps out of a 15GB tiff and uses Canvas to render them onto a map [1] Unless you are trying to layer all your maps together, you also could stop reprojecting them into webmercator, or if your goal is to layer them, then storing them in webmercator would save a ton of user's compute time. There are a bunch of us that talk web maping and imagery in the #maplibre and #imagery slack channels in OSMUS's slack [2] [1] https://blayne.chard.com/cogeotiff-web/index.html?view=cog&i=https%3A%2F%2Fpublic.lo.chard.com%2F2019-new-zealand-sentinel.3857.lzw.cog.tiff&debug=true https://blayne.chard.com/cogeotiff-web/index.html?view=cog&i... [2] https://github.com/maplibre/maplibre-gl-js#getting-involved https://github.com/maplibre/maplibre-gl-js#getting-involved
    - tppiotrowski 3y ago
      Great points. Thank you for the links. The one trade off here is that uncompressed blobs will require longer downloads than PNG and I think usually the network transfer is slower than PNG decoding. But maybe the sample gist takes a Tiff blob and encodes it to a PNG on the client and then maplibre decodes the PNG to canvas. That would be quite inefficient if that's what it's doing.
      - blacha 3y ago
        Those comments were more at pastmaps. For elevation data, we store our DEM/DSM in S3 as LERC [1] COGS, LERC has a WASM bundle which I think can be used in the browser. We found LERC COGs to be one of the most space efficient ways of storing highresolution DEM/DSM data [2], If you wanted to you could fetch LERC tiles directly out of a remote COG and use that directly for the terrain heights. I am more focused on storage/archiving/publishing of our LiDAR capture program [3] than web based visualizations of it though, so I am unsure if a LERC COG would even be better for you than a PNG TerrainRGB. [1] https://www.npmjs.com/package/lerc https://www.npmjs.com/package/lerc [2] https://github.com/linz/elevation/tree/master/docs/tiff-compression https://github.com/linz/elevation/tree/master/docs/tiff-comp... [3] https://linz.maps.arcgis.com/apps/MapSeries/index.html?appid=2552c3a5cee24f7b87806b085c3fee8a https://linz.maps.arcgis.com/apps/MapSeries/index.html?appid...
jamessb 3y ago
> coupled with http range queries on the static files hosted in S3 to bypass the need for tiling Did you look into Cloud Optimized GeoTIFF format? https://www.cogeo.org/ https://www.cogeo.org/ It is supported by OpenLayers: https://openlayers.org/en/latest/examples/cog.html https://openlayers.org/en/latest/examples/cog.html (I don't think that Maplibre or Leaflet have built-in support for it)
- Soupy 3y ago
  I did actually! For my particular source dataset it was far easier just to lean on the plain old GeoTiff format since that's what my source data was already formatted in and my testing of the client-side on-the-fly tiling using range queries and fast-png for encoding resulted in close to par performance with 0 increase in hosting costs and 0 headache in building custom pipelines to encode the cloud-optimized versions. Basically, I'm lazy, I'm sure it's the "correct" answer for other use-cases I also have been digging pretty extensively into protomaps though for some newer non-GeoTiff datasets I'm in the process of pulling in, in my opinion it's the future for this space - https://protomaps.com/ https://protomaps.com/
avibryant 3y ago
See also https://geoblaze-gsoc.vercel.app https://geoblaze-gsoc.vercel.app and the underlying libraries it uses, which also do range queries on GeoTIFFs. I'm super interested in this space, including in helping financially support some projects. I already emailed Ted about this, but would be happy to chat to anyone doing this stuff. My email's in my profile. (Luckily this is niche enough I'm not worried about my inbox blowing up....)
Pabloski80 3y ago
You're not anymore "the weirdest", You two are unique still though. Share and exchange, and try to support each other. All gonna be good. I need to learn more where lies the issue with GeoTIFF format. Perhaps "pure conversion" pipeline infrastructure shared between your two projects could help. If there are two of You, perhaps there are more hitting the same wall.
fzeindl 3y ago
> This is ridiculously cool I think so too. And it reminds me that I refactored Leaflet's core event-handling and SVG rendering code 8 years ago. It's good to know your opensource contribution play part in such cool things.

by Soupy 3y ago

tppiotrowski 3y ago

Ah. I hadn't even thought that I could do byte offsets on the fly. In general I would err on the side of over-sharing because until ChatGPT came along I struggled to understand what tools and options were to generate tiles as there wasn't much data I could find online. One thing specific in my case: the LiDAR GeoTiffs are in imperial feet with a 32 bit floating point precision. If you take the elevation range from sea level to Everest in meters (8848) and pack it into an int16, you can get .2 meter precision. That's plenty for ShadeMap so converting from float32 to int16 should save half the cloud storage space in theory and more when taking PNG compression into account.

Soupy 3y ago
This code is rough, and that's an understatement, but here's a basic gist showing how I'm doing what I'm doing in case it helps you in any way: https://gist.github.com/craigsc/fdb867f8971ff5b4ae42de4e0d7c229e https://gist.github.com/craigsc/fdb867f8971ff5b4ae42de4e0d7c... I'm similarly using R2 as my static hosting backend and it's been pretty fast and seamless note that i'm using the geotiff.js and fast-png libraries for the heavy-lifting behind the range-queries and the client-side png encoding. why reinvent the wheel, right?
- blacha 3y ago
  Your browser has a very powerful image decoder built into it, offloading the PNG decoding into Javascript is very resource hungry. Using maplibre (or any map viewer) you can load blobs of image data out of a tiff and use `Image` or `Canvas` to render the data onto a map. Its even easier if the tiffs are already Cloud optimized as they perfectly align to a 1-to-1 map tile and they don't need to be rescaled, you can then just render the images onto the map. eg here is a viewer that loads webps out of a 15GB tiff and uses Canvas to render them onto a map [1] Unless you are trying to layer all your maps together, you also could stop reprojecting them into webmercator, or if your goal is to layer them, then storing them in webmercator would save a ton of user's compute time. There are a bunch of us that talk web maping and imagery in the #maplibre and #imagery slack channels in OSMUS's slack [2] [1] https://blayne.chard.com/cogeotiff-web/index.html?view=cog&i=https%3A%2F%2Fpublic.lo.chard.com%2F2019-new-zealand-sentinel.3857.lzw.cog.tiff&debug=true https://blayne.chard.com/cogeotiff-web/index.html?view=cog&i... [2] https://github.com/maplibre/maplibre-gl-js#getting-involved https://github.com/maplibre/maplibre-gl-js#getting-involved
  - tppiotrowski 3y ago
    Great points. Thank you for the links. The one trade off here is that uncompressed blobs will require longer downloads than PNG and I think usually the network transfer is slower than PNG decoding. But maybe the sample gist takes a Tiff blob and encodes it to a PNG on the client and then maplibre decodes the PNG to canvas. That would be quite inefficient if that's what it's doing.
    - blacha 3y ago
      Those comments were more at pastmaps. For elevation data, we store our DEM/DSM in S3 as LERC [1] COGS, LERC has a WASM bundle which I think can be used in the browser. We found LERC COGs to be one of the most space efficient ways of storing highresolution DEM/DSM data [2], If you wanted to you could fetch LERC tiles directly out of a remote COG and use that directly for the terrain heights. I am more focused on storage/archiving/publishing of our LiDAR capture program [3] than web based visualizations of it though, so I am unsure if a LERC COG would even be better for you than a PNG TerrainRGB. [1] https://www.npmjs.com/package/lerc https://www.npmjs.com/package/lerc [2] https://github.com/linz/elevation/tree/master/docs/tiff-compression https://github.com/linz/elevation/tree/master/docs/tiff-comp... [3] https://linz.maps.arcgis.com/apps/MapSeries/index.html?appid=2552c3a5cee24f7b87806b085c3fee8a https://linz.maps.arcgis.com/apps/MapSeries/index.html?appid...

jamessb 3y ago

> coupled with http range queries on the static files hosted in S3 to bypass the need for tiling Did you look into Cloud Optimized GeoTIFF format? https://www.cogeo.org/ https://www.cogeo.org/ It is supported by OpenLayers: https://openlayers.org/en/latest/examples/cog.html https://openlayers.org/en/latest/examples/cog.html (I don't think that Maplibre or Leaflet have built-in support for it)

Soupy 3y ago
I did actually! For my particular source dataset it was far easier just to lean on the plain old GeoTiff format since that's what my source data was already formatted in and my testing of the client-side on-the-fly tiling using range queries and fast-png for encoding resulted in close to par performance with 0 increase in hosting costs and 0 headache in building custom pipelines to encode the cloud-optimized versions. Basically, I'm lazy, I'm sure it's the "correct" answer for other use-cases I also have been digging pretty extensively into protomaps though for some newer non-GeoTiff datasets I'm in the process of pulling in, in my opinion it's the future for this space - https://protomaps.com/ https://protomaps.com/

avibryant 3y ago

See also https://geoblaze-gsoc.vercel.app https://geoblaze-gsoc.vercel.app and the underlying libraries it uses, which also do range queries on GeoTIFFs. I'm super interested in this space, including in helping financially support some projects. I already emailed Ted about this, but would be happy to chat to anyone doing this stuff. My email's in my profile. (Luckily this is niche enough I'm not worried about my inbox blowing up....)

Pabloski80 3y ago

You're not anymore "the weirdest", You two are unique still though. Share and exchange, and try to support each other. All gonna be good. I need to learn more where lies the issue with GeoTIFF format. Perhaps "pure conversion" pipeline infrastructure shared between your two projects could help. If there are two of You, perhaps there are more hitting the same wall.

fzeindl 3y ago

> This is ridiculously cool I think so too. And it reminds me that I refactored Leaflet's core event-handling and SVG rendering code 8 years ago. It's good to know your opensource contribution play part in such cool things.