6 ms·

I built a vector map from scratch

228 points by kocheez75 4y ago

Hi HN I've used a lot of vector maps in the past, and was always fascinated by the technology, so I decided to try and build one from scratch as a way to learn more about how it works, and also as a reason to (finally) learn WebGL. I've uploaded the source to GitHub https://github.com/kochis/webgl-map https://github.com/kochis/webgl-map Hope someone finds it useful / informative, and open to any feedback or tips as well. Cheers!

just_mc 4y ago
This is great. I was thinking about doing something like this recently; however, I was considering WebGPU. Did you give any consideration to WebGL vs WebGPU for this purpose?
- kocheez75 4y ago
  Mostly just because I was using https://webglfundamentals.org/ https://webglfundamentals.org/ as my main resource. Will probably take a stab at implementing with WebGPU at some point.
coryetzkorn 4y ago
Wow I didn’t know maps are actually just a bunch of triangles. Really nice + comprehensive post.
- m2fkxy 4y ago
  if only you knew how right you are! https://en.wikipedia.org/wiki/Triangulation_(surveying) https://en.wikipedia.org/wiki/Triangulation_(surveying)
gregsadetsky 4y ago
Fantastic post, such a wonderful step by step guide! Although everything loaded and “worked” on mobile (I’m using iOS), two small interaction hiccups would be great to solve: - when panning (on mobile), the panned distance seemed to track the finger’s movement with some factor i.e. when moving “1 cm” of “finger space” (i.e. real world movement), the map moved by more than “1 cm”. (Sorry for the weird units, ha) - the common pinch zoom gesture is either being detected as something else or just not working? I was able to zoom, somehow, but not in any controlled way These don’t take anything away from the tutorial! I think that fixing these would make the learning/demo experience even smoother, that’s all. I may give it a stab since you shared the code :) Cheers --- Opened an issue to track this: https://github.com/kochis/webgl-map/issues/1 https://github.com/kochis/webgl-map/issues/1
- kocheez75 4y ago
  Yeah, I think this is a known issue with hammer.js[1]. I noticed it too, but didn't spend too much time digging in. [1]: https://stackoverflow.com/questions/42872774/hammer-js-exhibits-pinch-lag-when-changing-pinch-direction-tested-on-android https://stackoverflow.com/questions/42872774/hammer-js-exhib...
  - app4soft 4y ago
    As I see demo map[0] uses OpenStreetMap Data, but in such case on demo map page[0] there should be copyright[1] info. [0] https://ckochis.com/webgl-map-demo https://ckochis.com/webgl-map-demo [1] https://www.openstreetmap.org/copyright https://www.openstreetmap.org/copyright
    - kocheez75 4y ago
      Good callout, I’ll add that in. Most map libs add that automatically, so an oversight on my part.
rckclmbr 4y ago
I've been working with maps/mapbox for like 10 years, and knew of webgl and used libraries that used it but never myself. I've never dug into this level of detail for rendering. It's been a while since I've seen such a high-effort post on HN, thanks for this, it's great.
jeffbee 4y ago
The tile server appears to be toast.
- kocheez75 4y ago
  Ah, was afraid that might happen. I’m doing some pretty aggressive CF caching, but it’s running on a very small (single instance) server :( —- update Looks like I hit the CF worker limit. Should be back up and running. Thanks for the heads up!
ardit33 4y ago
This is pretty cool! Initially I thought it was a data structure :D
simonw 4y ago
This tutorial is absolutely fantastic.
- kocheez75 4y ago
  Thanks!
trynewideas 4y ago
Having done this, with the added difficulty layer of completely geo-referencing a non-Earth world map from scratch in QGIS instead of using existing data, this is comprehensive, easy to read, with great examples. Fantastic work. It makes me want to finally write up the geo-referencing work in a similar format.
- laurenkay 4y ago
  All while having a newborn and being a fantastic dad :)
fgeiger 4y ago
Whoa, zooming and panning too fast crashed the entirety of Firefox. All windows just gone in an instant. How is that even possible nowadays?
- tejtm 4y ago
  noting that the blinking off of FF is reproducible (not even doing anything fast)
- jstimpfle 4y ago
  The memory consumption on FF is out of control. Just loading the app and not interacting, it's going up at 1GB per few seconds for me with no perceivable upper bound. I killed the tab before risking system instability. With Chrome I'm not seeing this behaviour - the memory is stable at around 500-600M (probably around 400M for the app).
  - kocheez75 4y ago
    Interesting, my hunch would be something about the webgl buffers not getting freed, since they’re running in a tight loop. I’ll do some digging.
    - somishere 4y ago
      Came here to say the same thing. Great article btw. Finished it with examples/js disabled :)
    - jstimpfle 4y ago
      From a quick look it seems like you're doing gl.createBuffer() in a loop each time you're drawing, and never call gl.deleteBuffer(). It seems like WebGL will call deleteBuffer() automatically when the object is garbage collected by the JS runtime, but calling it explicitly when you're done with it will probably fix this particular issue. Also a suggestion, try to allocate your buffers only once on setup, and reuse them each time you're drawing. Just call gl.bufferData() in case you need to push new data. Don't use gl.bufferSubData() unless you come up with a clever way to avoid pipeline stalls such as maybe a rotating queue of buffers. This approach might or might not improve performance, but I see little reason not to do it. To make this work you prbably have to create a vertex arrays object explicitly using gl.createVertexArray(). This object is used to capture the vertex attrib pointers. Call gl.bindVertexArray() before gl.enableVertexArray() and gl.vertexAttribPointer(). Later, in a simple draw you need then only call gl.useProgram() and gl.bindVertexArray(), but there is no need to bind buffers or set vertex attrib pointers.
mourner 4y ago
Nice tutorial! Brace yourself for the next step (text rendering and label placement) — this is way more difficult than it may seem at first, and contributes to a lot of complexity in Mapbox GL JS.
- kocheez75 4y ago
  Yeah, that’s precisely why I didn’t tackle that. I was wondering though, does it render text in GL on the web, or leverage the DOM? (I’m just using absolute positioning for the tile labels)
  - mourner 4y ago
    It renders in WebGL with SDFs, but rendering is a small part of the problem — placing labels along paths including curved ones, avoiding collisions etc. gets very complex very fast.
  - jahewson 4y ago
    They’re drawn in WebGL using signed distance fields. https://blog.mapbox.com/drawing-text-with-signed-distance-fields-in-mapbox-gl-b0933af6f817 https://blog.mapbox.com/drawing-text-with-signed-distance-fi...
butz 4y ago
Is there any simple way to generate some tiles for a small region (e.g. park or tiny village) to serve pbf files from static server (or CDN)? Map data won't change too often.
- kocheez75 4y ago
  I believe Protomaps[1] might be what you’re looking for. [1]: https://protomaps.com/ https://protomaps.com/
mcovalt 4y ago
Thanks for sharing this! I recently went through the process of generating vector tiles for the world at all zoom levels using OpenMapTiles. Seemed like a fun way to make my computer sweat for a while. The process was pretty straightforward, but thought I might as well share the resulting mbtiles file here if anyone wants it. Link to torrent: https://kiwiziti.com/vector.mbtiles.torrent https://kiwiziti.com/vector.mbtiles.torrent
omniglottal 4y ago
This reliably and repeatedly locks up and crashes my mobile embedded browser and its parent app, my HN reader (Materialistic). I've never seen a more aggressively crashing site in all my years... took me 6 tries to post this comment, only once I bypassed the linked article/site.
- kocheez75 4y ago
  Ah, sorry to hear about that! I actually just pushed a fix for a similar crashing issue in Firefox, would be curious to hear if that resolved your issues as well.
lxe 4y ago
I miss posts like these. Great work! Easy to understand and the result is just incredible.
ricardobeat 4y ago
The demo map crashed my iPhone in the weirdest way. After zooming in and out a bit, the screen went dark, then fully pink, then it restarted.
- davesque 4y ago
  Yep, my phone too. I'm on a pixel 6.
  - slissenburg 4y ago
    My Motorola g9 with Chrome too. When I restarted Chrome it crashed again and again untill I finally managed to close the tab containing the blog post. edit: But I like your blog post, now that I opened it on my Mac.
qwertox 4y ago
I've been digging deep into the MapLibre GL Native source code this week to understand how it is built, and it turned out to be a huge and very complex project. Yesterday I was wondering how hard it would be to implement a custom map which makes use of vector and raster tiles, and here is an article which is basically a step-by-step tutorial on how to build one. Wow! It is amazing, very succinct and well-written, congratulations and thanks for sharing!
vnm 4y ago
Looks great on desktop, but crashes Chrome on Android (beefy specs). Great work and awesome topic anyway!

228 points by kocheez75 4y ago

just_mc 4y ago

This is great. I was thinking about doing something like this recently; however, I was considering WebGPU. Did you give any consideration to WebGL vs WebGPU for this purpose?

kocheez75 4y ago
Mostly just because I was using https://webglfundamentals.org/ https://webglfundamentals.org/ as my main resource. Will probably take a stab at implementing with WebGPU at some point.

coryetzkorn 4y ago

Wow I didn’t know maps are actually just a bunch of triangles. Really nice + comprehensive post.

m2fkxy 4y ago
if only you knew how right you are! https://en.wikipedia.org/wiki/Triangulation_(surveying) https://en.wikipedia.org/wiki/Triangulation_(surveying)

gregsadetsky 4y ago

Fantastic post, such a wonderful step by step guide! Although everything loaded and “worked” on mobile (I’m using iOS), two small interaction hiccups would be great to solve: - when panning (on mobile), the panned distance seemed to track the finger’s movement with some factor i.e. when moving “1 cm” of “finger space” (i.e. real world movement), the map moved by more than “1 cm”. (Sorry for the weird units, ha) - the common pinch zoom gesture is either being detected as something else or just not working? I was able to zoom, somehow, but not in any controlled way These don’t take anything away from the tutorial! I think that fixing these would make the learning/demo experience even smoother, that’s all. I may give it a stab since you shared the code :) Cheers --- Opened an issue to track this: https://github.com/kochis/webgl-map/issues/1 https://github.com/kochis/webgl-map/issues/1

kocheez75 4y ago
Yeah, I think this is a known issue with hammer.js[1]. I noticed it too, but didn't spend too much time digging in. [1]: https://stackoverflow.com/questions/42872774/hammer-js-exhibits-pinch-lag-when-changing-pinch-direction-tested-on-android https://stackoverflow.com/questions/42872774/hammer-js-exhib...
- app4soft 4y ago
  As I see demo map[0] uses OpenStreetMap Data, but in such case on demo map page[0] there should be copyright[1] info. [0] https://ckochis.com/webgl-map-demo https://ckochis.com/webgl-map-demo [1] https://www.openstreetmap.org/copyright https://www.openstreetmap.org/copyright
  - kocheez75 4y ago
    Good callout, I’ll add that in. Most map libs add that automatically, so an oversight on my part.

rckclmbr 4y ago

I've been working with maps/mapbox for like 10 years, and knew of webgl and used libraries that used it but never myself. I've never dug into this level of detail for rendering. It's been a while since I've seen such a high-effort post on HN, thanks for this, it's great.

jeffbee 4y ago

The tile server appears to be toast.

kocheez75 4y ago
Ah, was afraid that might happen. I’m doing some pretty aggressive CF caching, but it’s running on a very small (single instance) server :( —- update Looks like I hit the CF worker limit. Should be back up and running. Thanks for the heads up!

ardit33 4y ago

This is pretty cool! Initially I thought it was a data structure :D

simonw 4y ago

This tutorial is absolutely fantastic.

kocheez75 4y ago
Thanks!

trynewideas 4y ago

Having done this, with the added difficulty layer of completely geo-referencing a non-Earth world map from scratch in QGIS instead of using existing data, this is comprehensive, easy to read, with great examples. Fantastic work. It makes me want to finally write up the geo-referencing work in a similar format.

laurenkay 4y ago
All while having a newborn and being a fantastic dad :)

fgeiger 4y ago

Whoa, zooming and panning too fast crashed the entirety of Firefox. All windows just gone in an instant. How is that even possible nowadays?

tejtm 4y ago
noting that the blinking off of FF is reproducible (not even doing anything fast)
jstimpfle 4y ago
The memory consumption on FF is out of control. Just loading the app and not interacting, it's going up at 1GB per few seconds for me with no perceivable upper bound. I killed the tab before risking system instability. With Chrome I'm not seeing this behaviour - the memory is stable at around 500-600M (probably around 400M for the app).
- kocheez75 4y ago
  Interesting, my hunch would be something about the webgl buffers not getting freed, since they’re running in a tight loop. I’ll do some digging.
  - somishere 4y ago
    Came here to say the same thing. Great article btw. Finished it with examples/js disabled :)
  - jstimpfle 4y ago
    From a quick look it seems like you're doing gl.createBuffer() in a loop each time you're drawing, and never call gl.deleteBuffer(). It seems like WebGL will call deleteBuffer() automatically when the object is garbage collected by the JS runtime, but calling it explicitly when you're done with it will probably fix this particular issue. Also a suggestion, try to allocate your buffers only once on setup, and reuse them each time you're drawing. Just call gl.bufferData() in case you need to push new data. Don't use gl.bufferSubData() unless you come up with a clever way to avoid pipeline stalls such as maybe a rotating queue of buffers. This approach might or might not improve performance, but I see little reason not to do it. To make this work you prbably have to create a vertex arrays object explicitly using gl.createVertexArray(). This object is used to capture the vertex attrib pointers. Call gl.bindVertexArray() before gl.enableVertexArray() and gl.vertexAttribPointer(). Later, in a simple draw you need then only call gl.useProgram() and gl.bindVertexArray(), but there is no need to bind buffers or set vertex attrib pointers.

mourner 4y ago

Nice tutorial! Brace yourself for the next step (text rendering and label placement) — this is way more difficult than it may seem at first, and contributes to a lot of complexity in Mapbox GL JS.

kocheez75 4y ago
Yeah, that’s precisely why I didn’t tackle that. I was wondering though, does it render text in GL on the web, or leverage the DOM? (I’m just using absolute positioning for the tile labels)
- mourner 4y ago
  It renders in WebGL with SDFs, but rendering is a small part of the problem — placing labels along paths including curved ones, avoiding collisions etc. gets very complex very fast.
- jahewson 4y ago
  They’re drawn in WebGL using signed distance fields. https://blog.mapbox.com/drawing-text-with-signed-distance-fields-in-mapbox-gl-b0933af6f817 https://blog.mapbox.com/drawing-text-with-signed-distance-fi...

butz 4y ago

Is there any simple way to generate some tiles for a small region (e.g. park or tiny village) to serve pbf files from static server (or CDN)? Map data won't change too often.

kocheez75 4y ago
I believe Protomaps[1] might be what you’re looking for. [1]: https://protomaps.com/ https://protomaps.com/

mcovalt 4y ago

Thanks for sharing this! I recently went through the process of generating vector tiles for the world at all zoom levels using OpenMapTiles. Seemed like a fun way to make my computer sweat for a while. The process was pretty straightforward, but thought I might as well share the resulting mbtiles file here if anyone wants it. Link to torrent: https://kiwiziti.com/vector.mbtiles.torrent https://kiwiziti.com/vector.mbtiles.torrent

omniglottal 4y ago

This reliably and repeatedly locks up and crashes my mobile embedded browser and its parent app, my HN reader (Materialistic). I've never seen a more aggressively crashing site in all my years... took me 6 tries to post this comment, only once I bypassed the linked article/site.

kocheez75 4y ago
Ah, sorry to hear about that! I actually just pushed a fix for a similar crashing issue in Firefox, would be curious to hear if that resolved your issues as well.

lxe 4y ago

I miss posts like these. Great work! Easy to understand and the result is just incredible.

ricardobeat 4y ago

The demo map crashed my iPhone in the weirdest way. After zooming in and out a bit, the screen went dark, then fully pink, then it restarted.

davesque 4y ago
Yep, my phone too. I'm on a pixel 6.
- slissenburg 4y ago
  My Motorola g9 with Chrome too. When I restarted Chrome it crashed again and again untill I finally managed to close the tab containing the blog post. edit: But I like your blog post, now that I opened it on my Mac.

qwertox 4y ago

I've been digging deep into the MapLibre GL Native source code this week to understand how it is built, and it turned out to be a huge and very complex project. Yesterday I was wondering how hard it would be to implement a custom map which makes use of vector and raster tiles, and here is an article which is basically a step-by-step tutorial on how to build one. Wow! It is amazing, very succinct and well-written, congratulations and thanks for sharing!

vnm 4y ago

Looks great on desktop, but crashes Chrome on Android (beefy specs). Great work and awesome topic anyway!