4 ms·

Hi HN! This is my first blog, inspired by many posts read here. I attempt to explain how Vector Tiles in GIS tools are constructed from scratch, and how easy it

by ajd555 1y ago

Hi HN! This is my first blog, inspired by many posts read here. I attempt to explain how Vector Tiles in GIS tools are constructed from scratch, and how easy it is. This also sets up future blog posts about MapLibre tiles as well as some tricks in the backend to pre-generate the MVT files to serve them up quickly, and update tiles for real time updates on a map. I'll be happy to answer any questions or feedback!

jjgreen 1y ago
Interesting read, thanks
stevage 1y ago
I'm not entirely clear on why you had to go all the way to "from scratch". Are there really no libraries in Go that can generate tiles more conveniently? (I've been basically a full time freelance Mapbox consultant for the last 7 years. When I'm generating tiles, it's either statically (tippecanoe), through PostGIS, or occasionally using a JavaScript library. Never had to get my hands quite as dirty as you here...) It is a real shame that Mapbox GL JS doesn't support GeoJSON vector tiles, which makes a lot of this stuff easier. Yes, they're not as compact as PBF, but for ease of development, debuggability etc, they're a great step along the way and flatten the learning curve. (From memory, OpenLayers does support them...) Also curious whether you looked at https://docs.protomaps.com/pmtiles/ https://docs.protomaps.com/pmtiles/
- ajd555 1y ago
  Honestly, simple curiosity + wanted to keep it in Go
- ajd555 1y ago
  I have not looked at PMTiles, so far everything holds nicely generated in memory, so I haven't had the need to store this information. I'll keep it in mind. Any idea why Mapbox GL JS doesn't support GeoJSON vector tiles?
  - stevage 1y ago
    As far as I know, the design of Mapbox GL JS was very heavily geared towards their own needs of producing high performance maps that would be loaded on millions of devices. Obviously they'd never use MVT in that use case, so they didn't bother supporting it. There are lots of areas where they could have made the lives of developers a lot easier. Another that comes to mind is just how hard it is to look inside vector tiles - they don't really provide much in the way of tools for inspection, etc. I had to build https://stevage.github.io/vector-inspector/ https://stevage.github.io/vector-inspector/ and https://www.npmjs.com/package/tileinfo https://www.npmjs.com/package/tileinfo for instance.
    - ajd555 1y ago
      This is amazing, I hadn't come across these tools, these are extremely useful!
      - stevage 1y ago
        Heh, I feel like you're me 8 or 9 years ago, trying to get my head around vector tiles for the first time.
    - sbma44 1y ago
      hey, longtime Mapbox employee here. I appreciate all the work you're doing here to help people wrap their heads around vector tiles! This is an old technology at this point, and as you've explained, there are robust tools for moving from GeoJSON to tilesets. It's cool to pull apart the nuts and bolts of a thing (and the Mapbox Vector Tile Spec is open) but there are easier ways to accomplish this objective. A question for you: > Obviously they'd never use MVT in that use case, so they didn't bother supporting it. What does this mean? Mapbox GL (JS and native) both support MVT, of course--that's why we created it! Perhaps you were referring to something else? Higher in this post I see a reference to "GeoJSON vector tiles" and I'm curious what that means. GeoJSON is very convenient (Mapbox helped support its enshrinement as IETF RFC 7496), but one of the hard parts of tiling is slicing features and knowing when to simplify them. Once you've done that, you might as well cram it into a protobuf or other highly efficient container. When you pass Mapbox GL a GeoJSON, it actually cuts the geometry into tiles in memory and uses those for rendering. Some other general notes: - the process of tiling is lossy (or should be). if you zoom out to see all of north america, your GeoJSON of neighborhood address points is going to overlap. you should drop most of them from the low-zoomlevel tiles. Tippecanoe does this in brilliant and highly tuneable ways. This applies to geometry simplification, too. Folks should keep this in mind when doing size comparisons. - map tiling is fundamentally about moving compute into preprocessing and assembling geometry from highly parallelized fetches. MVT is a technology built on and for S3-like services. it's been exciting to see new approaches to this problem that offer lovely ergonomics around deployment etc, but if you have cloud infra, a hot cache, and are optimizing for performance, MVT remains hard to beat - we continue to research additional optimizations for VT, but the technology has stood the test of time, and proven useful in many different contexts beyond map rendering, including routing and geocoding
nilsingwersen 1y ago
I understand that you probably wanted to implement this from scratch but did you take a look at tippecanoe? https://github.com/felt/tippecanoe https://github.com/felt/tippecanoe originally by mapbox but got forked by felt.

by ajd555 1y ago

jjgreen 1y ago

Interesting read, thanks

stevage 1y ago

I'm not entirely clear on why you had to go all the way to "from scratch". Are there really no libraries in Go that can generate tiles more conveniently? (I've been basically a full time freelance Mapbox consultant for the last 7 years. When I'm generating tiles, it's either statically (tippecanoe), through PostGIS, or occasionally using a JavaScript library. Never had to get my hands quite as dirty as you here...) It is a real shame that Mapbox GL JS doesn't support GeoJSON vector tiles, which makes a lot of this stuff easier. Yes, they're not as compact as PBF, but for ease of development, debuggability etc, they're a great step along the way and flatten the learning curve. (From memory, OpenLayers does support them...) Also curious whether you looked at https://docs.protomaps.com/pmtiles/ https://docs.protomaps.com/pmtiles/

ajd555 1y ago
Honestly, simple curiosity + wanted to keep it in Go
ajd555 1y ago
I have not looked at PMTiles, so far everything holds nicely generated in memory, so I haven't had the need to store this information. I'll keep it in mind. Any idea why Mapbox GL JS doesn't support GeoJSON vector tiles?
- stevage 1y ago
  As far as I know, the design of Mapbox GL JS was very heavily geared towards their own needs of producing high performance maps that would be loaded on millions of devices. Obviously they'd never use MVT in that use case, so they didn't bother supporting it. There are lots of areas where they could have made the lives of developers a lot easier. Another that comes to mind is just how hard it is to look inside vector tiles - they don't really provide much in the way of tools for inspection, etc. I had to build https://stevage.github.io/vector-inspector/ https://stevage.github.io/vector-inspector/ and https://www.npmjs.com/package/tileinfo https://www.npmjs.com/package/tileinfo for instance.
  - ajd555 1y ago
    This is amazing, I hadn't come across these tools, these are extremely useful!
    - stevage 1y ago
      Heh, I feel like you're me 8 or 9 years ago, trying to get my head around vector tiles for the first time.
  - sbma44 1y ago
    hey, longtime Mapbox employee here. I appreciate all the work you're doing here to help people wrap their heads around vector tiles! This is an old technology at this point, and as you've explained, there are robust tools for moving from GeoJSON to tilesets. It's cool to pull apart the nuts and bolts of a thing (and the Mapbox Vector Tile Spec is open) but there are easier ways to accomplish this objective. A question for you: > Obviously they'd never use MVT in that use case, so they didn't bother supporting it. What does this mean? Mapbox GL (JS and native) both support MVT, of course--that's why we created it! Perhaps you were referring to something else? Higher in this post I see a reference to "GeoJSON vector tiles" and I'm curious what that means. GeoJSON is very convenient (Mapbox helped support its enshrinement as IETF RFC 7496), but one of the hard parts of tiling is slicing features and knowing when to simplify them. Once you've done that, you might as well cram it into a protobuf or other highly efficient container. When you pass Mapbox GL a GeoJSON, it actually cuts the geometry into tiles in memory and uses those for rendering. Some other general notes: - the process of tiling is lossy (or should be). if you zoom out to see all of north america, your GeoJSON of neighborhood address points is going to overlap. you should drop most of them from the low-zoomlevel tiles. Tippecanoe does this in brilliant and highly tuneable ways. This applies to geometry simplification, too. Folks should keep this in mind when doing size comparisons. - map tiling is fundamentally about moving compute into preprocessing and assembling geometry from highly parallelized fetches. MVT is a technology built on and for S3-like services. it's been exciting to see new approaches to this problem that offer lovely ergonomics around deployment etc, but if you have cloud infra, a hot cache, and are optimizing for performance, MVT remains hard to beat - we continue to research additional optimizations for VT, but the technology has stood the test of time, and proven useful in many different contexts beyond map rendering, including routing and geocoding

nilsingwersen 1y ago

I understand that you probably wanted to implement this from scratch but did you take a look at tippecanoe? https://github.com/felt/tippecanoe https://github.com/felt/tippecanoe originally by mapbox but got forked by felt.