5 ms·

I wish I could learn to build my own tiles, vector or PNG. I don't really understand where the data comes from, how is data gathered and assembled. I'm also re

by jokoon 7y ago

I wish I could learn to build my own tiles, vector or PNG. I don't really understand where the data comes from, how is data gathered and assembled. I'm also really curious about the choices involving the zoom level, how do you decide to render things depending on the zoom level, when is data discarded, to have good detail or better performance and lighter tiles. I would really be willing to try build lighter maps so I can have my own mapping software on a desktop machine. The data sizes and hardware requirements involved are generally pretty big. It could be interesting to see how much details one could achieve to make a "portable" map browser when limiting the data size to 2GB, 5GB or 10GB. I would really like to ask why, on some mapping software, you can't see names of large enough cities/places/regions that are obviously there. It often makes it difficult to browse maps properly.

sleavey 7y ago
I think many systems are backed by a PostGIS database (an extension for Postgres) with features and their coordinates. Map zoom levels define which features should be visible as layers on the map. A rendering frontend then grabs relevant data for the layer being viewed and builds the tiles.
snodnipper 7y ago
huge topic. > I wish I could learn to build my own tiles, vector or PNG. I don't really understand where the data comes from, how is data gathered and assembled. There are many data providers out there. You might be interested in OpenMapTiles, which is a pipeline from OpenStreetMap (OSM) data. https://github.com/openmaptiles/openmaptiles https://github.com/openmaptiles/openmaptiles Also check out Maputnik http://maputnik.github.io/editor/ http://maputnik.github.io/editor/ If you want to learn about "tiling schemes" then head over to https://www.maptiler.com/google-maps-coordinates-tile-bounds-projection/ https://www.maptiler.com/google-maps-coordinates-tile-bounds... > I'm also really curious about the choices involving the zoom level, how do you decide to render things depending on the zoom level, when is data discarded, to have good detail or better performance and lighter tiles. I would really be willing to try build lighter maps so I can have my own mapping software on a desktop machine. Lots of different considerations - is a human going to look at the map? If so then a cartographer will determine what is going to be shown at a given scale. There are other constraints too, such as limited space to show data and also hidden constraints, such as the maximum amount of data for a region (e.g. ~500kb per tile in the case of mapbox vector tiles) > The data sizes and hardware requirements involved are generally pretty big. It could be interesting to see how much details one could achieve to make a "portable" map browser when limiting the data size to 2GB, 5GB or 10GB. Lots of projects out there doing impressive things there. Quadtree tiles get you so far...k-d trees might yield other useful properties. Skobbler have some pretty impressive data compression technology (~12GiB for global coverage, routable and searchable...with some limitations - skobbler.com/apps). Of course the trick is to discard all that you don't need. > I would really like to ask why, on some mapping software, you can't see names of large enough cities/places/regions that are obviously there. It often makes it difficult to browse maps properly. If there is limited budget then the effort to create appropriate labels is limited. Data sources can be limited / incomplete...there can be nuances between jurisdictions etc. and of course label prioritisation has been a longstanding problem. What happens when you rotate a map and the text labels collide with one another...which ones do you keep...which do you discard etc. These things are also context dependent...why not include continent names? Or region names? Or province names? What about the difference between physical and political geography? A cartographer can help ensure that the right information is available at the right time...whilst acknowledging that they have to tell little white lies in every map they make.
bransonf 7y ago
I’ll try my best to explain the process. The data comes from places like the Census Bureau (roads, place names) and then a lot of it has to be collected by the like of OpenStreetMap/Google/Other Providers. (GIS Data is big business) For Vector based approaches (See mapbox) these data are stored in special built databases and usually simplified geometries are served to the browser. The benefit is continuous zoom, but the pitfall is more server side computation and hence cost. Because of the cost/compute, raster tiles (PNG, jpg, any pixel format) have been much more popular. These start the same, you collect all these data and put them in a database. The difference is the added step of rendering tiles. This one-off computation saves you work from then on. See maps.stamen.com for an example of tiles made from OSM data. And you’re right about place names sometimes not being apparent. This is a trade off when using open data and auto generated tiles. With something like MapBox’s vector tiles, you have individual decimal level control of things like labels. And zoom level is another computational trade off. You start at 0 and define an arbitrary end. The higher the number, the computation/data increases four fold each n. O(4^n) And as far as why the size requirements are so big, geospatial data is big. You have to record information on every point for vectors which depending on quality can be a ton. And for rasters, we’re talking trillions of pixels really. That’s why all of this is server side. And lastly to your point about lightweight desktop software, tiles don’t really have a place in the data process. They’re only really useful for the visualization aspect. And frankly, I think we’re reaching the capacity of the technique, we just might have some headroom in server efficiency.
- snodnipper 7y ago
  +1 > And lastly to your point about lightweight desktop software, tiles don’t really have a place in the data process. They’re only really useful for the visualization aspect. And frankly, I think we’re reaching the capacity of the technique, we just might have some headroom in server efficiency. Not totally sure what you mean on your last point...data can be feature centric (e.g. stored by feature id) or area centric (stored by area location) etc. Storing data by location is important far beyond visualisation and is abstracted in databases such as PostGIS/Postgres (a branded data structure). That said, I acknowledge that ArcGIS Pro, QGIS etc. have limited support for tiled data but of course that is changing. Safe funded much (all?) of the OGR MVT development afaik.
sp332 7y ago
This article from 2010 goes into a lot of detail about which labels to show and how to place them. https://news.ycombinator.com/item?id=1963612 https://news.ycombinator.com/item?id=1963612
trynewideas 7y ago
> I wish I could learn to build my own tiles, vector or PNG. I don't really understand where the data comes from, how is data gathered and assembled. The tools are rapidly evolving. There's no great single entry point and the best advice I can give is pretty generic: find a small-scale thing you want to do and do research toward accomplishing it. The post you're commenting on is about how PostGIS databases mostly do this work for vector tiles on its own now, so "to build your own tiles", you'd set up a PostGIS database and re-read this post. A year or two ago the advice would be pretty different. A year or two from now and the advice will be _completely_ different. That said, from zero http://geojson.io http://geojson.io is a dead straightforward way to do basic operations with GeoJSON data. You can paste in JSON and it renders on the map; you can draw on the map and it generates GeoJSON. (https://tilejson.io https://tilejson.io does the same for raster tile sets.) Real-world data is massive and overwhelming to work with — just drawing your own fake maps in geojson.io and working with that might make some of its concepts easier to digest. Maperitve[1] is a free and relatively straightforward app focused on taking geo data as input and outputting maps. Work with its rendering rules and you'll understand some of the challenges with rendering at different zoom levels or in different contexts. Then this post from 2018[2] on Tippecanoe (tile and data converter), TileServer GL (tile server), and Leaflet (Javascript front end to view served tiles) covers how to round-trip a package of vector tiles to GeoJSON data and back. It's straightforward, works with a relatively small area of data, doesn't require GIS experience, and though outdated it's still relevant for understanding by practice how a data-to-tiles pipeline can work. Raster tiles are a little difficult to recommend learning as tooling has mostly moved on from it in favor of vector tiles, which pack more information and flexibility into less data, and I honestly don't know what tools still reliably do that work — once upon a time I used TileMill but it was already abandoned by then and has been very lightly maintained since. Re: optimization, here's another more advanced post[3] using real-world data that illustrates some of the challenges. The end-game is to get to a point where you can open something like QGIS[4], a heavyweight tool that can do all of the above and way too much more, or Maputnik[5], a vector tile styling tool using a CSS-ish language, and not get immediately lost. > I would really like to ask why, on some mapping software, you can't see names of large enough cities/places/regions that are obviously there. You won't get a great answer why to that question, I'm afraid. It's dependent on and configured in whatever the front end is, generally done algorithmically, and in some cases manually edited. An art as much as a science, and as fallible as both combined. (See Justin O'Beirne's incredible reviews of Apple Map updates[6] for an example.) No single labeling strategy will make anyone (much less everyone) happy and most end-user tools don't expose customizability. 1: http://maperitive.net/docs/TwoMinutesIntro.html http://maperitive.net/docs/TwoMinutesIntro.html 2: https://medium.com/@kennethchambers/using-tippecanoe-tileserver-gl-and-leaflet-to-serve-scale-independent-and-really-cool-looking-751368d821c7 https://medium.com/@kennethchambers/using-tippecanoe-tileser... 3: https://medium.com/@ibesora/a-data-driven-journey-through-vector-tile-optimization-4a1dbd4f3a27 https://medium.com/@ibesora/a-data-driven-journey-through-ve... 4: https://qgis.org/en/site/ https://qgis.org/en/site/ 5: https://maputnik.github.io https://maputnik.github.io 6: https://www.justinobeirne.com/new-apple-maps https://www.justinobeirne.com/new-apple-maps
deleted 7y ago
[deleted]

by jokoon 7y ago

sleavey 7y ago

I think many systems are backed by a PostGIS database (an extension for Postgres) with features and their coordinates. Map zoom levels define which features should be visible as layers on the map. A rendering frontend then grabs relevant data for the layer being viewed and builds the tiles.

snodnipper 7y ago

huge topic. > I wish I could learn to build my own tiles, vector or PNG. I don't really understand where the data comes from, how is data gathered and assembled. There are many data providers out there. You might be interested in OpenMapTiles, which is a pipeline from OpenStreetMap (OSM) data. https://github.com/openmaptiles/openmaptiles https://github.com/openmaptiles/openmaptiles Also check out Maputnik http://maputnik.github.io/editor/ http://maputnik.github.io/editor/ If you want to learn about "tiling schemes" then head over to https://www.maptiler.com/google-maps-coordinates-tile-bounds-projection/ https://www.maptiler.com/google-maps-coordinates-tile-bounds... > I'm also really curious about the choices involving the zoom level, how do you decide to render things depending on the zoom level, when is data discarded, to have good detail or better performance and lighter tiles. I would really be willing to try build lighter maps so I can have my own mapping software on a desktop machine. Lots of different considerations - is a human going to look at the map? If so then a cartographer will determine what is going to be shown at a given scale. There are other constraints too, such as limited space to show data and also hidden constraints, such as the maximum amount of data for a region (e.g. ~500kb per tile in the case of mapbox vector tiles) > The data sizes and hardware requirements involved are generally pretty big. It could be interesting to see how much details one could achieve to make a "portable" map browser when limiting the data size to 2GB, 5GB or 10GB. Lots of projects out there doing impressive things there. Quadtree tiles get you so far...k-d trees might yield other useful properties. Skobbler have some pretty impressive data compression technology (~12GiB for global coverage, routable and searchable...with some limitations - skobbler.com/apps). Of course the trick is to discard all that you don't need. > I would really like to ask why, on some mapping software, you can't see names of large enough cities/places/regions that are obviously there. It often makes it difficult to browse maps properly. If there is limited budget then the effort to create appropriate labels is limited. Data sources can be limited / incomplete...there can be nuances between jurisdictions etc. and of course label prioritisation has been a longstanding problem. What happens when you rotate a map and the text labels collide with one another...which ones do you keep...which do you discard etc. These things are also context dependent...why not include continent names? Or region names? Or province names? What about the difference between physical and political geography? A cartographer can help ensure that the right information is available at the right time...whilst acknowledging that they have to tell little white lies in every map they make.

bransonf 7y ago

I’ll try my best to explain the process. The data comes from places like the Census Bureau (roads, place names) and then a lot of it has to be collected by the like of OpenStreetMap/Google/Other Providers. (GIS Data is big business) For Vector based approaches (See mapbox) these data are stored in special built databases and usually simplified geometries are served to the browser. The benefit is continuous zoom, but the pitfall is more server side computation and hence cost. Because of the cost/compute, raster tiles (PNG, jpg, any pixel format) have been much more popular. These start the same, you collect all these data and put them in a database. The difference is the added step of rendering tiles. This one-off computation saves you work from then on. See maps.stamen.com for an example of tiles made from OSM data. And you’re right about place names sometimes not being apparent. This is a trade off when using open data and auto generated tiles. With something like MapBox’s vector tiles, you have individual decimal level control of things like labels. And zoom level is another computational trade off. You start at 0 and define an arbitrary end. The higher the number, the computation/data increases four fold each n. O(4^n) And as far as why the size requirements are so big, geospatial data is big. You have to record information on every point for vectors which depending on quality can be a ton. And for rasters, we’re talking trillions of pixels really. That’s why all of this is server side. And lastly to your point about lightweight desktop software, tiles don’t really have a place in the data process. They’re only really useful for the visualization aspect. And frankly, I think we’re reaching the capacity of the technique, we just might have some headroom in server efficiency.

snodnipper 7y ago
+1 > And lastly to your point about lightweight desktop software, tiles don’t really have a place in the data process. They’re only really useful for the visualization aspect. And frankly, I think we’re reaching the capacity of the technique, we just might have some headroom in server efficiency. Not totally sure what you mean on your last point...data can be feature centric (e.g. stored by feature id) or area centric (stored by area location) etc. Storing data by location is important far beyond visualisation and is abstracted in databases such as PostGIS/Postgres (a branded data structure). That said, I acknowledge that ArcGIS Pro, QGIS etc. have limited support for tiled data but of course that is changing. Safe funded much (all?) of the OGR MVT development afaik.

sp332 7y ago

This article from 2010 goes into a lot of detail about which labels to show and how to place them. https://news.ycombinator.com/item?id=1963612 https://news.ycombinator.com/item?id=1963612

trynewideas 7y ago

> I wish I could learn to build my own tiles, vector or PNG. I don't really understand where the data comes from, how is data gathered and assembled. The tools are rapidly evolving. There's no great single entry point and the best advice I can give is pretty generic: find a small-scale thing you want to do and do research toward accomplishing it. The post you're commenting on is about how PostGIS databases mostly do this work for vector tiles on its own now, so "to build your own tiles", you'd set up a PostGIS database and re-read this post. A year or two ago the advice would be pretty different. A year or two from now and the advice will be _completely_ different. That said, from zero http://geojson.io http://geojson.io is a dead straightforward way to do basic operations with GeoJSON data. You can paste in JSON and it renders on the map; you can draw on the map and it generates GeoJSON. (https://tilejson.io https://tilejson.io does the same for raster tile sets.) Real-world data is massive and overwhelming to work with — just drawing your own fake maps in geojson.io and working with that might make some of its concepts easier to digest. Maperitve[1] is a free and relatively straightforward app focused on taking geo data as input and outputting maps. Work with its rendering rules and you'll understand some of the challenges with rendering at different zoom levels or in different contexts. Then this post from 2018[2] on Tippecanoe (tile and data converter), TileServer GL (tile server), and Leaflet (Javascript front end to view served tiles) covers how to round-trip a package of vector tiles to GeoJSON data and back. It's straightforward, works with a relatively small area of data, doesn't require GIS experience, and though outdated it's still relevant for understanding by practice how a data-to-tiles pipeline can work. Raster tiles are a little difficult to recommend learning as tooling has mostly moved on from it in favor of vector tiles, which pack more information and flexibility into less data, and I honestly don't know what tools still reliably do that work — once upon a time I used TileMill but it was already abandoned by then and has been very lightly maintained since. Re: optimization, here's another more advanced post[3] using real-world data that illustrates some of the challenges. The end-game is to get to a point where you can open something like QGIS[4], a heavyweight tool that can do all of the above and way too much more, or Maputnik[5], a vector tile styling tool using a CSS-ish language, and not get immediately lost. > I would really like to ask why, on some mapping software, you can't see names of large enough cities/places/regions that are obviously there. You won't get a great answer why to that question, I'm afraid. It's dependent on and configured in whatever the front end is, generally done algorithmically, and in some cases manually edited. An art as much as a science, and as fallible as both combined. (See Justin O'Beirne's incredible reviews of Apple Map updates[6] for an example.) No single labeling strategy will make anyone (much less everyone) happy and most end-user tools don't expose customizability. 1: http://maperitive.net/docs/TwoMinutesIntro.html http://maperitive.net/docs/TwoMinutesIntro.html 2: https://medium.com/@kennethchambers/using-tippecanoe-tileserver-gl-and-leaflet-to-serve-scale-independent-and-really-cool-looking-751368d821c7 https://medium.com/@kennethchambers/using-tippecanoe-tileser... 3: https://medium.com/@ibesora/a-data-driven-journey-through-vector-tile-optimization-4a1dbd4f3a27 https://medium.com/@ibesora/a-data-driven-journey-through-ve... 4: https://qgis.org/en/site/ https://qgis.org/en/site/ 5: https://maputnik.github.io https://maputnik.github.io 6: https://www.justinobeirne.com/new-apple-maps https://www.justinobeirne.com/new-apple-maps

deleted 7y ago

[deleted]