5 ms·

Terracotta: Serverless GeoTiff Tile Server

72 points by xk3 5y ago

belter 5y ago
Interesting project but initially I though it was related to this: https://github.com/terracotta-oss https://github.com/terracotta-oss It's not :-)
houzi 5y ago
This is sorely missed in the GIS world: A well written and simple tool that isn't over engineered trying to cover all sorts of fringe use cases. Very clean code and good design at first glance. I get why rasterio is preferred, its API is less annoying, but I'm not sure putting a leaking abstraction over gdal helps much. It just creates an aluring veneer over gdal, without really solving the main problems of gdal, being its monolithic monstrosity of hardly portable yet highly necessary GIS tool functionality. Doesn't matter too much with gdal's warts, as this server should be deployed with ansible or in a container anyways, so portability is handled in a somewhat sane manner. Thanks for open sourcing this!
- SahAssar 5y ago
  Isn't this built on rasterio? I haven't used terracotta or rasterio, but your comment sounded like you compared them.
  - dionhaefner 5y ago
    I think the point was that Terracotta might be better off using GDAL directly instead of via rasterio. Maybe, but I think mapbox is doing a great job with rasterio, and so far we didn’t run into any unsolvable problems (although we got close sometimes). I think for the ease of installation alone rasterio is a blessing.
    - houzi 5y ago
      Yes. I'm having a hard time finding a good selling point for rasterio. > I think for the ease of installation alone rasterio is a blessing. How does rasterio simplify installation, when you still have to deal with gdal's C extension dependencies?
      - dionhaefner 5y ago
        Rasterio wheels ship with GDAL binaries. No C compiler needed.
  - houzi 5y ago
    Yes. Sorry for the ambiguous comment. Whenever I see a GIS application, I'm always curious to see how they solve the gdal problem. The best is always no GDAL, but that gets pretty hard to do quite quickly as GDAL contains some really great algorithms for dealing with non-trivial raster processing.
    - mistrial9 5y ago
      yes, GDAL :-)
anakaine 5y ago
Looks good! Any chance you might consider adding NetCDF support? Used in so many places, contains time, but currently requires a behemoth like geoserver to share as a service
- dionhaefner 5y ago
  Author here. Not really, since we rely on cloud-optimized GeoTiffs for performance, in particular their overviews. This makes it so we can serve tiles from huge datasets (10s of GB) in less than 100ms. I don’t think that would be possible with NetCDF.
Waterluvian 5y ago
For tile raster services, the story for caching already rendered tiles to a CDN needs to be first class. The Lambda portion is really just the minor detail: the thing that generates tiles if thy haven’t been generated yet. But depending on your zoom levels and map size, ideally you don’t even have a lambda service and just precache tiles. Then you’ve reduced your service to a dead simple file host problem.
- dionhaefner 5y ago
  That is true to some degree, but with Terracotta we allow the frontend to do manipulations on the fly. Like changing contrast, applying colormaps, or band math. So the number of possible tiles is infinite. If you don’t need that, you can just pre-generate all PNGs - then you don’t even need a server anymore, just a hash function to identify the right file and a big disk.
  - jhgb 5y ago
    > like changing contrast, applying colormaps, or band math Surely these are examples of things that a client could do with raster data independent of the capabilities of a server?
    - dionhaefner 5y ago
      Only if the client has access to the full bit depth. The rendered PNGs are uint8, but the underlying raw data is often stored in 16 or even 32 bit precision.
      - jhgb 5y ago
        I imagine that with HTTP range requests, it shouldn't be hard to access original TIFF data. Javascript or WebAssembly can then work with arbitrary bytevectors.
        dionhaefner 5y ago
        Yes, it’s possible. There are some proofs of concept for that IIRC. If you prefer to write everything in Javascript you can do that and do all heavy lifting in the client (plus some extra work to provide indexing and metadata like Terracotta).
MiscIdeaMaker99 5y ago
That's a pretty clever name.
- folli 5y ago
  cooked earth?
  - eCa 5y ago
    Terracotta is often used as tiles.
bottled_poe 5y ago
Does this project comply with any WebGIS standards for tile-based APIs? I couldn’t see this in the README or intro docs. Would be a shame for a project like this to miss out on uptake for only this reason.
- dionhaefner 5y ago
  I’m not sure which standards you are referring to (I guess that means no). If you have a link or open an issue we could scope this out.
deleted 5y ago
[deleted]

72 points by xk3 5y ago

Interesting project but initially I though it was related to this: https://github.com/terracotta-oss https://github.com/terracotta-oss It's not :-)

houzi 5y ago

This is sorely missed in the GIS world: A well written and simple tool that isn't over engineered trying to cover all sorts of fringe use cases. Very clean code and good design at first glance. I get why rasterio is preferred, its API is less annoying, but I'm not sure putting a leaking abstraction over gdal helps much. It just creates an aluring veneer over gdal, without really solving the main problems of gdal, being its monolithic monstrosity of hardly portable yet highly necessary GIS tool functionality. Doesn't matter too much with gdal's warts, as this server should be deployed with ansible or in a container anyways, so portability is handled in a somewhat sane manner. Thanks for open sourcing this!

SahAssar 5y ago
Isn't this built on rasterio? I haven't used terracotta or rasterio, but your comment sounded like you compared them.
- dionhaefner 5y ago
  I think the point was that Terracotta might be better off using GDAL directly instead of via rasterio. Maybe, but I think mapbox is doing a great job with rasterio, and so far we didn’t run into any unsolvable problems (although we got close sometimes). I think for the ease of installation alone rasterio is a blessing.
  - houzi 5y ago
    Yes. I'm having a hard time finding a good selling point for rasterio. > I think for the ease of installation alone rasterio is a blessing. How does rasterio simplify installation, when you still have to deal with gdal's C extension dependencies?
    - dionhaefner 5y ago
      Rasterio wheels ship with GDAL binaries. No C compiler needed.
- houzi 5y ago
  Yes. Sorry for the ambiguous comment. Whenever I see a GIS application, I'm always curious to see how they solve the gdal problem. The best is always no GDAL, but that gets pretty hard to do quite quickly as GDAL contains some really great algorithms for dealing with non-trivial raster processing.
  - mistrial9 5y ago
    yes, GDAL :-)

anakaine 5y ago

Looks good! Any chance you might consider adding NetCDF support? Used in so many places, contains time, but currently requires a behemoth like geoserver to share as a service

dionhaefner 5y ago
Author here. Not really, since we rely on cloud-optimized GeoTiffs for performance, in particular their overviews. This makes it so we can serve tiles from huge datasets (10s of GB) in less than 100ms. I don’t think that would be possible with NetCDF.

Waterluvian 5y ago

For tile raster services, the story for caching already rendered tiles to a CDN needs to be first class. The Lambda portion is really just the minor detail: the thing that generates tiles if thy haven’t been generated yet. But depending on your zoom levels and map size, ideally you don’t even have a lambda service and just precache tiles. Then you’ve reduced your service to a dead simple file host problem.

dionhaefner 5y ago
That is true to some degree, but with Terracotta we allow the frontend to do manipulations on the fly. Like changing contrast, applying colormaps, or band math. So the number of possible tiles is infinite. If you don’t need that, you can just pre-generate all PNGs - then you don’t even need a server anymore, just a hash function to identify the right file and a big disk.
- jhgb 5y ago
  > like changing contrast, applying colormaps, or band math Surely these are examples of things that a client could do with raster data independent of the capabilities of a server?
  - dionhaefner 5y ago
    Only if the client has access to the full bit depth. The rendered PNGs are uint8, but the underlying raw data is often stored in 16 or even 32 bit precision.
    - jhgb 5y ago
      I imagine that with HTTP range requests, it shouldn't be hard to access original TIFF data. Javascript or WebAssembly can then work with arbitrary bytevectors.
      - dionhaefner 5y ago
        Yes, it’s possible. There are some proofs of concept for that IIRC. If you prefer to write everything in Javascript you can do that and do all heavy lifting in the client (plus some extra work to provide indexing and metadata like Terracotta).

MiscIdeaMaker99 5y ago

That's a pretty clever name.

folli 5y ago
cooked earth?
- eCa 5y ago
  Terracotta is often used as tiles.

bottled_poe 5y ago

Does this project comply with any WebGIS standards for tile-based APIs? I couldn’t see this in the README or intro docs. Would be a shame for a project like this to miss out on uptake for only this reason.

dionhaefner 5y ago
I’m not sure which standards you are referring to (I guess that means no). If you have a link or open an issue we could scope this out.

deleted 5y ago

[deleted]