7 ms·

Serving Dynamic Vector Tiles from PostGIS

143 points by liotier 7y ago

crmrc114 7y ago
I have a pal who does GIS work in the oil and gas industry- I think its crazy how much influence ESRI has on that market. Would love to learn more about interaction with map data like this. For a non-gis person this was a fun read. So thanks for the post!
- bransonf 7y ago
  I have the same sentiment on ESRI. It’s basically all I got taught in my university courses, but it’s not what I ever want to use. It’s crazy that a privately held company holds like a third of the market share. And personally, I don’t think their software is that good. I find their documentation to be undesirable and their solutions to be strict. Case in point, the geodatabase (gdb) standard is purposefully meant to obfuscate the data within. No one has ever been able to explain to me why this is, and the standard has been open sourced by now. Not to mention, the number of times I’ve had ArcMap crash without any helpful information as to why it crashed... That said, ArcMap is the Excel of GIS. It captured market share (especially government contracts) two or three decades ago and no one has disrupted the desktop GIS platform. On the web front, however, I see companies like MapBox far outpacing anything ESRI is capable of yet. And to anyone looking to learn GIS: Post GIS, GDAL and any scripting language will make you more powerful than most of the people I know within the field.
  - trynewideas 7y ago
    oblig. mention for others outside of the ESRI sphere of influence that QGIS exists, is still FOSS, and still actively developed https://qgis.org/en/site/ https://qgis.org/en/site/
  - sleavey 7y ago
    > And to anyone looking to learn GIS: Post GIS, GDAL and any scripting language will make you more powerful than most of the people I know within the field. Funny this article was posted today, because yesterday I was looking into rendering a custom map for a ~100x100 km area from OpenStreetMap data for a particular application. I've got basically no experience making maps but I've dabbled with GDAL and Rasterio. I was thinking of using Mapnik with a dump of (part of) the OpenStreetMap database into a local PostGIS instance. Ideally the rendered tiles should be vector format. Do you think this approach seems reasonable or am I missing a potentially simpler way?
    - bransonf 7y ago
      Are you trying to render a static map or make something interactive? If this is static, Mapnik is a good call. It has some extra anti-aliasing under the hood and it’s exceptionally fast.
      - sleavey 7y ago
        Static; thanks for the info. I would ideally like to dump a bunch of SVG tiles for various zoom levels so I can store them in a static directory on my server rather than serve them dynamically. I take it that Mapnik is capable of dumps like this? And, I would like to use the Python bindings but they look relatively badly documented. Would you suggest a newbie like me uses the C or XML interfaces instead, if they are better documented?
        bransonf 7y ago
        I’ve only used the Python library, and I think I survived. I only used it for some visualization however. And I’m still unclear. Are you trying to serve these tiles to another application? Or are you trying to make a digital/print map? And SVG is probably excessive for static tiles. It won’t have the size reduction of a raster tile nor the benefit of a true vector solution.
        sleavey 7y ago
        > And I’m still unclear. Are you trying to serve these tiles to another application? Or are you trying to make a digital/print map? Serve them from local storage to a viewer application. > And SVG is probably excessive for static tiles. It won’t have the size reduction of a raster tile nor the benefit of a true vector solution. Ah, is there another vector format for tiles other than SVG? Or are you saying that I should just generate a bunch of compressed rasters?
        bransonf 7y ago
        Ah, I haven’t really got into that territory with mapnik. But to the second point, yes you should just generate a bunch of Raster tiles. And before doing this, ask yourself if you really need to. If this isn’t a huge project, Mapbox is an easy packed solution. Otherwise, there are dozens of really good tile providers already.
  - aldoushuxley001 7y ago
    QGIS does an admirable job competing with ArcGIS.
anonymousCmntr 7y ago
I use this project for serving vector tiles with PostGIS and Django REST. https://github.com/corteva/djangorestframework-mvt https://github.com/corteva/djangorestframework-mvt
jokoon 7y ago
I wish I could learn to build my own tiles, vector or PNG. I don't really understand where the data comes from, how is data gathered and assembled. I'm also really curious about the choices involving the zoom level, how do you decide to render things depending on the zoom level, when is data discarded, to have good detail or better performance and lighter tiles. I would really be willing to try build lighter maps so I can have my own mapping software on a desktop machine. The data sizes and hardware requirements involved are generally pretty big. It could be interesting to see how much details one could achieve to make a "portable" map browser when limiting the data size to 2GB, 5GB or 10GB. I would really like to ask why, on some mapping software, you can't see names of large enough cities/places/regions that are obviously there. It often makes it difficult to browse maps properly.
- sleavey 7y ago
  I think many systems are backed by a PostGIS database (an extension for Postgres) with features and their coordinates. Map zoom levels define which features should be visible as layers on the map. A rendering frontend then grabs relevant data for the layer being viewed and builds the tiles.
- snodnipper 7y ago
  huge topic. > I wish I could learn to build my own tiles, vector or PNG. I don't really understand where the data comes from, how is data gathered and assembled. There are many data providers out there. You might be interested in OpenMapTiles, which is a pipeline from OpenStreetMap (OSM) data. https://github.com/openmaptiles/openmaptiles https://github.com/openmaptiles/openmaptiles Also check out Maputnik http://maputnik.github.io/editor/ http://maputnik.github.io/editor/ If you want to learn about "tiling schemes" then head over to https://www.maptiler.com/google-maps-coordinates-tile-bounds-projection/ https://www.maptiler.com/google-maps-coordinates-tile-bounds... > I'm also really curious about the choices involving the zoom level, how do you decide to render things depending on the zoom level, when is data discarded, to have good detail or better performance and lighter tiles. I would really be willing to try build lighter maps so I can have my own mapping software on a desktop machine. Lots of different considerations - is a human going to look at the map? If so then a cartographer will determine what is going to be shown at a given scale. There are other constraints too, such as limited space to show data and also hidden constraints, such as the maximum amount of data for a region (e.g. ~500kb per tile in the case of mapbox vector tiles) > The data sizes and hardware requirements involved are generally pretty big. It could be interesting to see how much details one could achieve to make a "portable" map browser when limiting the data size to 2GB, 5GB or 10GB. Lots of projects out there doing impressive things there. Quadtree tiles get you so far...k-d trees might yield other useful properties. Skobbler have some pretty impressive data compression technology (~12GiB for global coverage, routable and searchable...with some limitations - skobbler.com/apps). Of course the trick is to discard all that you don't need. > I would really like to ask why, on some mapping software, you can't see names of large enough cities/places/regions that are obviously there. It often makes it difficult to browse maps properly. If there is limited budget then the effort to create appropriate labels is limited. Data sources can be limited / incomplete...there can be nuances between jurisdictions etc. and of course label prioritisation has been a longstanding problem. What happens when you rotate a map and the text labels collide with one another...which ones do you keep...which do you discard etc. These things are also context dependent...why not include continent names? Or region names? Or province names? What about the difference between physical and political geography? A cartographer can help ensure that the right information is available at the right time...whilst acknowledging that they have to tell little white lies in every map they make.
flippmoke 7y ago
Author of Mapbox's Vector Tile specification here and also contributor to some of the code that is used by PostGIS and I wanted to add some additional clarity on some topics associated with Vector Tiles and dynamic serving of them that seems to be a new trend. The Vector Tiles specification was designed for map visualization but has expanded into other uses as well, but in general the purpose is to be able to quickly provide a complete subset of data for a specific area that is highly cacheable. Most of this provided speed and cache-ability is specifically gained by preprocessing all the data you will use in your map into tiles. The general steps for turning raw data into Vector Tiles are: 1. Determine a hierarchy of your data. For example if you are talking about roads at some zoom levels you will want to see only highways or major roads while at other zoom levels you will want all your data. 2. For each tile at each zoom level; Select your data following your hierarchy rules, simplify your data based on your zoom level (for example you might need less points to display your road) and then clip your data to your tile and encode it to your Vector Tile. The problem is that doing these steps is often very complex and requires thought about the cartography of your final resulting map, but it can also drastically effect performance. If you are dynamically serving tiles from PostGIS it is very hard to reduce large quantities of data quickly in some cases. For example take a very detailed coastline of a large lake that is very precise and you are wanting to serve this dynamically. If you are attempting to serve this data on demand each time you need a tile you have to simplify and clip a potentially massive polygon. While this might work for single requests, if you increase in scale this quickly adds lots of load to a PostGIS server. The only solution is to cache the resulting tiles for a longer period to limit load on your database or to preprocess all your data before serving. Preprocessing of all the tiles is already something other tiling tools such as tippecanoe are really good at doing and comes with the benefit of helping you determine a hierarchy for your data. Preprocessing might seem excessive when it comes to making potentially millions of tiles, but in general it makes your application faster because it is simply serving an already created tile. Therefore, if your data does not very change quickly I would almost always suggest using preprocessing over dynamic rendering of tiles. You might spend more effort maintaining something than you expect if you start using PostGIS to create tiles on demand over existing tiling tools.
- durkie 7y ago

143 points by liotier 7y ago

I have a pal who does GIS work in the oil and gas industry- I think its crazy how much influence ESRI has on that market. Would love to learn more about interaction with map data like this. For a non-gis person this was a fun read. So thanks for the post!

bransonf 7y ago
I have the same sentiment on ESRI. It’s basically all I got taught in my university courses, but it’s not what I ever want to use. It’s crazy that a privately held company holds like a third of the market share. And personally, I don’t think their software is that good. I find their documentation to be undesirable and their solutions to be strict. Case in point, the geodatabase (gdb) standard is purposefully meant to obfuscate the data within. No one has ever been able to explain to me why this is, and the standard has been open sourced by now. Not to mention, the number of times I’ve had ArcMap crash without any helpful information as to why it crashed... That said, ArcMap is the Excel of GIS. It captured market share (especially government contracts) two or three decades ago and no one has disrupted the desktop GIS platform. On the web front, however, I see companies like MapBox far outpacing anything ESRI is capable of yet. And to anyone looking to learn GIS: Post GIS, GDAL and any scripting language will make you more powerful than most of the people I know within the field.
- trynewideas 7y ago
  oblig. mention for others outside of the ESRI sphere of influence that QGIS exists, is still FOSS, and still actively developed https://qgis.org/en/site/ https://qgis.org/en/site/
- sleavey 7y ago
  > And to anyone looking to learn GIS: Post GIS, GDAL and any scripting language will make you more powerful than most of the people I know within the field. Funny this article was posted today, because yesterday I was looking into rendering a custom map for a ~100x100 km area from OpenStreetMap data for a particular application. I've got basically no experience making maps but I've dabbled with GDAL and Rasterio. I was thinking of using Mapnik with a dump of (part of) the OpenStreetMap database into a local PostGIS instance. Ideally the rendered tiles should be vector format. Do you think this approach seems reasonable or am I missing a potentially simpler way?
  - bransonf 7y ago
    Are you trying to render a static map or make something interactive? If this is static, Mapnik is a good call. It has some extra anti-aliasing under the hood and it’s exceptionally fast.
    - sleavey 7y ago
      Static; thanks for the info. I would ideally like to dump a bunch of SVG tiles for various zoom levels so I can store them in a static directory on my server rather than serve them dynamically. I take it that Mapnik is capable of dumps like this? And, I would like to use the Python bindings but they look relatively badly documented. Would you suggest a newbie like me uses the C or XML interfaces instead, if they are better documented?
      - bransonf 7y ago
        I’ve only used the Python library, and I think I survived. I only used it for some visualization however. And I’m still unclear. Are you trying to serve these tiles to another application? Or are you trying to make a digital/print map? And SVG is probably excessive for static tiles. It won’t have the size reduction of a raster tile nor the benefit of a true vector solution.
        sleavey 7y ago
        > And I’m still unclear. Are you trying to serve these tiles to another application? Or are you trying to make a digital/print map? Serve them from local storage to a viewer application. > And SVG is probably excessive for static tiles. It won’t have the size reduction of a raster tile nor the benefit of a true vector solution. Ah, is there another vector format for tiles other than SVG? Or are you saying that I should just generate a bunch of compressed rasters?
        bransonf 7y ago
        Ah, I haven’t really got into that territory with mapnik. But to the second point, yes you should just generate a bunch of Raster tiles. And before doing this, ask yourself if you really need to. If this isn’t a huge project, Mapbox is an easy packed solution. Otherwise, there are dozens of really good tile providers already.
- aldoushuxley001 7y ago
  QGIS does an admirable job competing with ArcGIS.

anonymousCmntr 7y ago

I use this project for serving vector tiles with PostGIS and Django REST. https://github.com/corteva/djangorestframework-mvt https://github.com/corteva/djangorestframework-mvt

jokoon 7y ago

I wish I could learn to build my own tiles, vector or PNG. I don't really understand where the data comes from, how is data gathered and assembled. I'm also really curious about the choices involving the zoom level, how do you decide to render things depending on the zoom level, when is data discarded, to have good detail or better performance and lighter tiles. I would really be willing to try build lighter maps so I can have my own mapping software on a desktop machine. The data sizes and hardware requirements involved are generally pretty big. It could be interesting to see how much details one could achieve to make a "portable" map browser when limiting the data size to 2GB, 5GB or 10GB. I would really like to ask why, on some mapping software, you can't see names of large enough cities/places/regions that are obviously there. It often makes it difficult to browse maps properly.

sleavey 7y ago
I think many systems are backed by a PostGIS database (an extension for Postgres) with features and their coordinates. Map zoom levels define which features should be visible as layers on the map. A rendering frontend then grabs relevant data for the layer being viewed and builds the tiles.
snodnipper 7y ago
huge topic. > I wish I could learn to build my own tiles, vector or PNG. I don't really understand where the data comes from, how is data gathered and assembled. There are many data providers out there. You might be interested in OpenMapTiles, which is a pipeline from OpenStreetMap (OSM) data. https://github.com/openmaptiles/openmaptiles https://github.com/openmaptiles/openmaptiles Also check out Maputnik http://maputnik.github.io/editor/ http://maputnik.github.io/editor/ If you want to learn about "tiling schemes" then head over to https://www.maptiler.com/google-maps-coordinates-tile-bounds-projection/ https://www.maptiler.com/google-maps-coordinates-tile-bounds... > I'm also really curious about the choices involving the zoom level, how do you decide to render things depending on the zoom level, when is data discarded, to have good detail or better performance and lighter tiles. I would really be willing to try build lighter maps so I can have my own mapping software on a desktop machine. Lots of different considerations - is a human going to look at the map? If so then a cartographer will determine what is going to be shown at a given scale. There are other constraints too, such as limited space to show data and also hidden constraints, such as the maximum amount of data for a region (e.g. ~500kb per tile in the case of mapbox vector tiles) > The data sizes and hardware requirements involved are generally pretty big. It could be interesting to see how much details one could achieve to make a "portable" map browser when limiting the data size to 2GB, 5GB or 10GB. Lots of projects out there doing impressive things there. Quadtree tiles get you so far...k-d trees might yield other useful properties. Skobbler have some pretty impressive data compression technology (~12GiB for global coverage, routable and searchable...with some limitations - skobbler.com/apps). Of course the trick is to discard all that you don't need. > I would really like to ask why, on some mapping software, you can't see names of large enough cities/places/regions that are obviously there. It often makes it difficult to browse maps properly. If there is limited budget then the effort to create appropriate labels is limited. Data sources can be limited / incomplete...there can be nuances between jurisdictions etc. and of course label prioritisation has been a longstanding problem. What happens when you rotate a map and the text labels collide with one another...which ones do you keep...which do you discard etc. These things are also context dependent...why not include continent names? Or region names? Or province names? What about the difference between physical and political geography? A cartographer can help ensure that the right information is available at the right time...whilst acknowledging that they have to tell little white lies in every map they make.

flippmoke 7y ago

Author of Mapbox's Vector Tile specification here and also contributor to some of the code that is used by PostGIS and I wanted to add some additional clarity on some topics associated with Vector Tiles and dynamic serving of them that seems to be a new trend. The Vector Tiles specification was designed for map visualization but has expanded into other uses as well, but in general the purpose is to be able to quickly provide a complete subset of data for a specific area that is highly cacheable. Most of this provided speed and cache-ability is specifically gained by preprocessing all the data you will use in your map into tiles. The general steps for turning raw data into Vector Tiles are: 1. Determine a hierarchy of your data. For example if you are talking about roads at some zoom levels you will want to see only highways or major roads while at other zoom levels you will want all your data. 2. For each tile at each zoom level; Select your data following your hierarchy rules, simplify your data based on your zoom level (for example you might need less points to display your road) and then clip your data to your tile and encode it to your Vector Tile. The problem is that doing these steps is often very complex and requires thought about the cartography of your final resulting map, but it can also drastically effect performance. If you are dynamically serving tiles from PostGIS it is very hard to reduce large quantities of data quickly in some cases. For example take a very detailed coastline of a large lake that is very precise and you are wanting to serve this dynamically. If you are attempting to serve this data on demand each time you need a tile you have to simplify and clip a potentially massive polygon. While this might work for single requests, if you increase in scale this quickly adds lots of load to a PostGIS server. The only solution is to cache the resulting tiles for a longer period to limit load on your database or to preprocess all your data before serving. Preprocessing of all the tiles is already something other tiling tools such as tippecanoe are really good at doing and comes with the benefit of helping you determine a hierarchy for your data. Preprocessing might seem excessive when it comes to making potentially millions of tiles, but in general it makes your application faster because it is simply serving an already created tile. Therefore, if your data does not very change quickly I would almost always suggest using preprocessing over dynamic rendering of tiles. You might spend more effort maintaining something than you expect if you start using PostGIS to create tiles on demand over existing tiling tools.

durkie 7y ago