4 ms·

If you are looking for maps themselves (a la Google Maps) I highly recommend checking out Protomaps [0]. It provides a single file (PMTile format) that contains

If you are looking for maps themselves (a la Google Maps) I highly recommend checking out Protomaps [0]. It provides a single file (PMTile format) that contains all the map data and you can request ranges from the single file to get the required vector data you need for a given area/zoom. So you can setup something like the following: S3 (host the myfile.pmtiles) <- Lambda (takes the x/y/z from the path and requests the correct range) <- Cloudfront cache tile response Then you can setup tiles.mydomain.com (or use the cloudfront domain directly) and then use Leaflet or similar on the frontend to fetch/render the tiles. For Leaflet you use the protomaps plugin/lib and give it a url like "https://tiles.yourdomain.com/20230408/{z}/{x}/{y}.mvt https://tiles.yourdomain.com/20230408/{z}/{x}/{y}.mvt" where "20230408" maps to "20230408.pmtiles" in your S3 bucket. Now I can drop new pmtiles files into that bucket and update my clients to use the new source. And since the tiles are in vector format you can theme them however you want in the client which is neat. Lastly you don't have to use the 100+GB whole-earth tileset. You can use a tool [1] (provided by the same guy) to download a dataset for just a given geographical region. The .pmtiles file is a little over 100GB but the whole setup took me only an hour or two max to get running and will cost way less than Google Maps to run. [0] https://protomaps.com/ https://protomaps.com/ [1] https://app.protomaps.com/downloads/small_map https://app.protomaps.com/downloads/small_map

mdaniel 3y ago
their demo isn't doing them any favors since it seems to stop at some random zoomlevel, eliding the kind of detail that I'd care about if I were evaluating that solution digging into their GH repos surfaces https://protomaps.github.io/PMTiles/?url=https%3A%2F%2Fr2-public.protomaps.com%2Fprotomaps-sample-datasets%2Fprotomaps-basemap-opensource-20230408.pmtiles https://protomaps.github.io/PMTiles/?url=https%3A%2F%2Fr2-pu... which is at least more detailed if not quite a GMaps killer
- qwertox 3y ago
  I agree, if it weren't for your comment, I was about to ditch it as a candidate because it stopped zooming at a bad level. But your link shows that it contains a tremendous amount of data. Edit: And I missed parents link to https://app.protomaps.com/downloads/small_map https://app.protomaps.com/downloads/small_map which is actually better.
- bdon 3y ago
  I just updated https://protomaps.com https://protomaps.com it to use the more detailed file you found, thanks. So instead of a zoom level of 12 it should work down to 20+ - the actual dataset contains up to zoom 15, which is ~1 billion tiles in a single 110GB file. It's important to me that I showcase the system working as advertised - you are free to download the 110GB file from that static storage with no restrictions. Unfortunately, the demo is slow because I'm hosting it on Cloudflare R2 which has no outgoing bandwidth fees. Before, it was on DigitalOcean Spaces, which is low latency but would cost me $1-2 out of my own pocket each time someone clicked the link. The Lambda/Workers integration as described in the other comment is the preferred solution to this for developers operating this at planet scale. Of course, the static storage solution is perfect for small to medium-sized areas, even on GitHub Pages.
  - joshstrange 3y ago
    I just wanted to say thank you for everything you've done for Protomaps. It was very easy to get started with and other than a small Vue/ES6-related issue I was able to get it integrated into my codebase very quickly.
  - yencabulator 3y ago
    > Unfortunately, the demo is slow because I'm hosting it on Cloudflare R2 which has no outgoing bandwidth fees. Can you expand on this a little? Cloudflare is supposed to be pretty competent as CDN and with caching.
    - bdon 3y ago
      This is Cloudflare's new blob storage product R2. Putting the free Cloudflare CDN and cache infront of R2 doesn't work with HTTP Range headers, likely for product differentiation from their Video CDN. The latency of R2 is also much higher than S3 or GCP, but I know the R2 team is working hard on improving this.
      - yencabulator 3y ago
        FWIW you can add your own caching logic with Workers, if the CDN for some reason doesn't do Range. The Cache API exposed to Workers is quite useful.
simmschi 3y ago
So one Lambda execution per tile?
- wiredfool 3y ago
  It shouldn’t be required if the range requests are right and you have a compatible renderer in the client. To expand a bit -- There are adapters for mapbox(gl/libre) and leaflet, and so on that will allow you to use a .pmtiles directly from the client with a range request supporting server, like s3 or nginx. I've used protomaps basemaps to generate an Ireland and EU basemaps for a project from OSM data. I still haven't quite figured out the tuning for which features should be shown, combined, or excluded yet, and the generated layer names are sort of, but not quite compatible with similar base layers + styles from mapbox (which uses the same base data, but a different, proprietary conversion). This part takes a reasonable skill as a GIS person and some taste in the design of the feature symbology. The downside of doing this on S3 directly is that you might wind up publishing a link to a 100 gig file that would cost $10 if someone just downloaded it. Lambda invocations seem cheap compared to that risk. OTOH, throwing it on a hetzner box is quite reasonable.
  - bdon 3y ago
    The transformation of OpenStreetMap features into the basemap is open source and available under a permissive license now at http://github.com/protomaps/basemaps http://github.com/protomaps/basemaps This is the focus of most development for the next few months - like you said, it's also a matter of aesthetics and context, like if the map is underlying another PMTiles data layer it should be lower contrast then if it's overlaid with pin markers. The end goal is to have a flexible basemap solution adaptable to most apps but you are free to roll your own proprietary style on top of the open source tiles if you'd like (there are already companies doing this)
    - wiredfool 3y ago
      Yes, that's what I was using for my basemaps, it's quite fast for Ireland on a relatively fast machine (minutes), but Europe as a whole was hours. (hetzner quad core 8 thread ~3yr old intel machine with 64G ram) I've got a couple of goals, or at least evaluations for goals -- 1) Have a stripped down basemap like Carto Light, but with different emphasis for the "bicycling friendly" roads, and selectively higher contrast/emphasis for those minor through roads that discourage car traffic but are perfect for a bike. (e.g. Irish boreens, those 1 lane or double track "paved" L roads that are everywhere outside of the major cities.) I find that I simply can't see the minor roads on apple/google maps when I'm out unless I'm zoomed in to the point that I can barely see a network. (fading eyesight and super low contrast). 2) Another lighter base map, but with transit focused features to compliment some of the open data in Ireland around transit -- the routes and timetables and so on. I'm aiming for a site that can be 100% statically hosted but will show the routes in a more friendly manner than Bus Eireann's site. 3) If it goes well, I might be doing this in a more commercial context for some clients who are currently using mapbox for their basemaps but due to political concerns need to insert in different names/boundaries for disputed areas/features. We'd love to be able to self host, but quality is a major concern there. The bugs that I'm seeing in the 1 and 2 cases are things like handling the look of freeway interchanges/flyovers at close zooms, consistently getting rivers to show at appropriate zooms because they're into two separate feature types depending on the width, with the narrow bits getting dropped. And the general mismatch between the styling I'm used to from the mapbox converted layer names/feature types and what's coming out of pmtiles/basemaps. The feature bits look like tippecanoe coalesce/drop densest features tuning, but I'm also looking at just dropping out entire feature sets to drop the size of the tiles. That should help the coalesce/dropping behavior as well.
Borg3 3y ago
Hmm, thats cool, but I need to have raster maps pushed to my endpoints. Do you know any easy way to use mvt maps and generate tiles from them on the fly (CGI backend + caching) and then use Leaflet to render those? I have my own tiny personal project for maps renderer using Leaflet. For now I use 3 sources for tiles (Google, OSM and Esri) but having more sources would be very cool.
- joshstrange 3y ago
  I don't personally know how to do that. It appears the PMTiles format does support raster maps but the dataset I use is only vector as far as I know.

mdaniel 3y ago

their demo isn't doing them any favors since it seems to stop at some random zoomlevel, eliding the kind of detail that I'd care about if I were evaluating that solution digging into their GH repos surfaces https://protomaps.github.io/PMTiles/?url=https%3A%2F%2Fr2-public.protomaps.com%2Fprotomaps-sample-datasets%2Fprotomaps-basemap-opensource-20230408.pmtiles https://protomaps.github.io/PMTiles/?url=https%3A%2F%2Fr2-pu... which is at least more detailed if not quite a GMaps killer

qwertox 3y ago
I agree, if it weren't for your comment, I was about to ditch it as a candidate because it stopped zooming at a bad level. But your link shows that it contains a tremendous amount of data. Edit: And I missed parents link to https://app.protomaps.com/downloads/small_map https://app.protomaps.com/downloads/small_map which is actually better.
bdon 3y ago
I just updated https://protomaps.com https://protomaps.com it to use the more detailed file you found, thanks. So instead of a zoom level of 12 it should work down to 20+ - the actual dataset contains up to zoom 15, which is ~1 billion tiles in a single 110GB file. It's important to me that I showcase the system working as advertised - you are free to download the 110GB file from that static storage with no restrictions. Unfortunately, the demo is slow because I'm hosting it on Cloudflare R2 which has no outgoing bandwidth fees. Before, it was on DigitalOcean Spaces, which is low latency but would cost me $1-2 out of my own pocket each time someone clicked the link. The Lambda/Workers integration as described in the other comment is the preferred solution to this for developers operating this at planet scale. Of course, the static storage solution is perfect for small to medium-sized areas, even on GitHub Pages.
- joshstrange 3y ago
  I just wanted to say thank you for everything you've done for Protomaps. It was very easy to get started with and other than a small Vue/ES6-related issue I was able to get it integrated into my codebase very quickly.
- yencabulator 3y ago
  > Unfortunately, the demo is slow because I'm hosting it on Cloudflare R2 which has no outgoing bandwidth fees. Can you expand on this a little? Cloudflare is supposed to be pretty competent as CDN and with caching.
  - bdon 3y ago
    This is Cloudflare's new blob storage product R2. Putting the free Cloudflare CDN and cache infront of R2 doesn't work with HTTP Range headers, likely for product differentiation from their Video CDN. The latency of R2 is also much higher than S3 or GCP, but I know the R2 team is working hard on improving this.
    - yencabulator 3y ago
      FWIW you can add your own caching logic with Workers, if the CDN for some reason doesn't do Range. The Cache API exposed to Workers is quite useful.

simmschi 3y ago

So one Lambda execution per tile?

wiredfool 3y ago
It shouldn’t be required if the range requests are right and you have a compatible renderer in the client. To expand a bit -- There are adapters for mapbox(gl/libre) and leaflet, and so on that will allow you to use a .pmtiles directly from the client with a range request supporting server, like s3 or nginx. I've used protomaps basemaps to generate an Ireland and EU basemaps for a project from OSM data. I still haven't quite figured out the tuning for which features should be shown, combined, or excluded yet, and the generated layer names are sort of, but not quite compatible with similar base layers + styles from mapbox (which uses the same base data, but a different, proprietary conversion). This part takes a reasonable skill as a GIS person and some taste in the design of the feature symbology. The downside of doing this on S3 directly is that you might wind up publishing a link to a 100 gig file that would cost $10 if someone just downloaded it. Lambda invocations seem cheap compared to that risk. OTOH, throwing it on a hetzner box is quite reasonable.
- bdon 3y ago
  The transformation of OpenStreetMap features into the basemap is open source and available under a permissive license now at http://github.com/protomaps/basemaps http://github.com/protomaps/basemaps This is the focus of most development for the next few months - like you said, it's also a matter of aesthetics and context, like if the map is underlying another PMTiles data layer it should be lower contrast then if it's overlaid with pin markers. The end goal is to have a flexible basemap solution adaptable to most apps but you are free to roll your own proprietary style on top of the open source tiles if you'd like (there are already companies doing this)
  - wiredfool 3y ago
    Yes, that's what I was using for my basemaps, it's quite fast for Ireland on a relatively fast machine (minutes), but Europe as a whole was hours. (hetzner quad core 8 thread ~3yr old intel machine with 64G ram) I've got a couple of goals, or at least evaluations for goals -- 1) Have a stripped down basemap like Carto Light, but with different emphasis for the "bicycling friendly" roads, and selectively higher contrast/emphasis for those minor through roads that discourage car traffic but are perfect for a bike. (e.g. Irish boreens, those 1 lane or double track "paved" L roads that are everywhere outside of the major cities.) I find that I simply can't see the minor roads on apple/google maps when I'm out unless I'm zoomed in to the point that I can barely see a network. (fading eyesight and super low contrast). 2) Another lighter base map, but with transit focused features to compliment some of the open data in Ireland around transit -- the routes and timetables and so on. I'm aiming for a site that can be 100% statically hosted but will show the routes in a more friendly manner than Bus Eireann's site. 3) If it goes well, I might be doing this in a more commercial context for some clients who are currently using mapbox for their basemaps but due to political concerns need to insert in different names/boundaries for disputed areas/features. We'd love to be able to self host, but quality is a major concern there. The bugs that I'm seeing in the 1 and 2 cases are things like handling the look of freeway interchanges/flyovers at close zooms, consistently getting rivers to show at appropriate zooms because they're into two separate feature types depending on the width, with the narrow bits getting dropped. And the general mismatch between the styling I'm used to from the mapbox converted layer names/feature types and what's coming out of pmtiles/basemaps. The feature bits look like tippecanoe coalesce/drop densest features tuning, but I'm also looking at just dropping out entire feature sets to drop the size of the tiles. That should help the coalesce/dropping behavior as well.

Borg3 3y ago

Hmm, thats cool, but I need to have raster maps pushed to my endpoints. Do you know any easy way to use mvt maps and generate tiles from them on the fly (CGI backend + caching) and then use Leaflet to render those? I have my own tiny personal project for maps renderer using Leaflet. For now I use 3 sources for tiles (Google, OSM and Esri) but having more sources would be very cool.

joshstrange 3y ago
I don't personally know how to do that. It appears the PMTiles format does support raster maps but the dataset I use is only vector as far as I know.