12 ms·

A new way to make maps with OpenStreetMap

508 points by hampelm 5y ago

Tomte 5y ago
> Existing formats like MBTiles are based on SQLite. This limits the ability to self-host maps to those confident running a server in production. Is that comical, or is there a deeper meaning I don't understand?
- leokennis 5y ago
  I read it as: existing solutions are too hard for simple people with less technical skill.
  - deleted 5y ago
    [deleted]
  - bdon 5y ago
    Author here; a narrower way to express what I meant: A frontend developer who wants to put a map in their application and wants control over the hosting and data faces some obstacles when using MBtiles: * Need a virtual machine and disk space * Need to choose a particular program to read the MBTiles and serve it via HTTP * Might need reverse proxy and service monitor (systemd or sysv init) to keep it running 24/7 * Need to deal with SSL termination The popularity of serverless platforms = evidence there's a strong market for delivering software without all of this.
- codetrotter 5y ago
  Probably what they mean is that they want to be able to host the maps as static files that are served to the web. So that you can FTP some html, js, css, images and map data files to a web host and call it a day.
- rakoo 5y ago
  AFAIU MBTiles still need some application server to transform x/y/z into proper tiles to be served, which means provisioning a server, scaling it, maintaning it, and all the fun that comes with it; PMTiles is a simple blob that the frontend will download (only desired parts) and render directly in the browser.
  - alohec 5y ago
    You can use tippecanoe (https://github.com/mapbox/tippecanoe https://github.com/mapbox/tippecanoe) with the --output-to-directory flag to output the individual .pbf tiles instead of a single MBTiles file. These tiles can then be hosted on something like s3 and your map-rendering client can query the necessary tiles, no server required.
- incanus77 5y ago
  Originally, MBTiles was designed for mobile offline. Then, Mapbox started serving directly from the SQLite for web & other clients, creating the Node.js SQLite bindings in the process. Eventually, they had an unpacker for MBTiles uploads that would stick the tiles in CDN. So, really no more complex than old school static content hosting. Source: I created MBTiles ;-)
  - bdon 5y ago
    Author of OP here, I use the tile unpacker as well! I didn't point this out in the OP, but I found the ergonomics of uploading tiles to S3 etc get bad once you're over a couple tens of thousands of tiles, as each upload has some overhead. I've tested PMTiles on S3 successfully with hundreds of millions of tiles. Deduplication is also important to save space on redundant ocean/earth tiles; MBTiles can deduplicate via views, PMTiles deduplicates by reference, but unpacked Z/X/Y tiles on S3 can't.
    - incanus77 5y ago
      Neat! I'm hoping to dig into your design more... looks great.
    - TheTaytay 5y ago
      When I was playing around with hosting rendered Minecraft slippy maps, I found the upload overhead to S3 to be pretty poor due to the “lots of tiny files” problem. (It renders as hundreds of thousands of rasterized image files for the various zoom levels). You could parallelize it, but it was still poor, and made me want to come up with a way to pack them into fewer files and fetch portions of them at render time so that the upload could be faster. It sounds like that’s what you did for the the PMTiles format?
      - bdon 5y ago
        That's exactly what PMTiles is for, and there's nothing in it specific to maps or geographic coordinates, just a Z/X/Y pyramid structure. I chose to support Leaflet for now because it is very popular; there's other viewers like OpenSeadragon for which a decoder plugin would be possible.
- deleted 5y ago
  [deleted]
- jleedev 5y ago
  For comparison, Cloud Optimized GeoTIFF (COG) is specifically designed to be readable with HTTP Range requests, so the single file can be placed on S3 and be immediately usable. As others pointed out, the MVTs inside can be unpacked for static serving. But a SQLite database is not itself usable in this way.
- MrMartian 5y ago
  So I fee like it actually missed the mark as to why it's so impactful. When you get to global scale tiling with building accuracy, you literally end up with hundreds of millions of tiles. The upload time / cost per upload STACKS. However, the tilesets combined usually sits between 100GB and 200GB. ONE upload of ~150GB seems is faster and more cost efficient via S3.
c0nsumer 5y ago
This is really interesting to me. I do a lot of mapping of natural surface single track trails (for hiking and mountain biking) along the way be sure to properly name things, add intersection markers, and create relations (routes and superroutes) to document the systems. The biggest downside thus far is a good way of displaying the data. mtb.waymarkedtrails.org gets close, but I'd really like something that displays relations in the specified color, has sane intersection markers, etc. I feel like there should be something out there which effectively lets me design a style sheet and then apply it to a map which I embed somewhere (maybe even in an app). This... feels like it? Or if not this, can any of you recommend another system? Ideally I'd love something I can self-host. I tried getting a copy of waymarkedtrails going and then modifying that, but it's a bit too specialized and I kept running into quirks getting the toolchain set up. EDIT: Looks like this might not include the highway=path data that I need, much relations. Alas.
- ourguile 5y ago
  Have you used GaiaGPS at all? I use it along with ridewithgps for looking at trail conditions and mapping points of interest on my rides and hikes. You can import your own custom map layers as well. The developers are fairly consistently adding new content as well, might not hurt to reach out.
  - c0nsumer 5y ago
    I have! It's great, but it doesn't show route relations for paths.
- bdon 5y ago
  Author of OP here. Is this information that you're interested in contributing, and is appropriate for, OpenStreetMap? The end goal is to build a system that lets you customize the tileset to include any node, way, relation, or tag in OSM. This gets quite tricky when considering automated cartographic generalization (how such features will appear when zoomed out) but it's one of my next steps.
  - 5y ago
brunoqc 5y ago
> Pricing > coming soon...
- kennydude 5y ago
  At least it's open source and self hostable so if it's too expensive for you, you can host it yourself
  - brunoqc 5y ago
    Ah fair enough.
iandanforth 5y ago
Very interesting, I'm currently using Google My Maps as a dead-simple custom mapping tool. I'm using it to collect layers of information as I conduct a nationwide housing search. It would be great if there were a solution that was as simple to get started with as My Maps but also had the flexibility to easily add custom data layers programmatically.
- jMyles 5y ago
  Curious about your tool chain for real estate metadata. Is there a way to get a more retail-like experience and not have to deal so much with agents?
  - iandanforth 5y ago
    Paying a guy in Kenya to scrape data for me. Swear I tried to pay every national real estate API out there and no one is interested in talking to you at less than 7k a month and a full on lead gen business.
- loa_in_ 5y ago
  > It would be great if there were a solution that was as simple to get started with as My Maps but also had the flexibility to easily add custom data layers programmatically. I believe _with some trickery_ you can use Google Earth for that
  - wizzwizz4 5y ago
    The computer version, not the web version, right?
    - loa_in_ 5y ago
      Yes
- DoreenMichele 5y ago
  I'm curious what sorts of data you are collecting and how it helps to have it in map form rather than some other file format.
mourner 5y ago
This is very nice! Happy to see a few of my libraries in the dependencies :) Also check out https://github.com/kothic/kothic-js https://github.com/kothic/kothic-js, an older but similar attempt at making a full-fledged map renderer using 2D Canvas. It has a pretty good label rendering in particular which you might find useful.
- xrd 5y ago
  This looks great. I am super excited about the parent tool, but unsure whether/when I can generate these bundles programmatically. I tried the web based tool and it works great, but what if I need to do it many, many times? Can your tool do this easily? Is there a way to generate tiles (tiles.pmtiles) that are imported in the same way? If I had stumbled on your project, I would not have been enthusiastic about it with just the writeup as-is. Seeing the parent project makes me see the possibilities: a completely independent mapping solution, free of any server/service tie in and additional costs. That is super exciting, and I would love to know if and how your project compares.
  - bdon 5y ago
    I'd like to understand what you mean by doing it many times - do you mean for different areas of the world? I haven't thought about API access to the bundles in detail yet, but you can contact me via email at brandon@protomaps.com if you have some idea of what you need.
- bdon 5y ago
  Thank you! Really appreciate the commitment to simplicity among the libraries I'm using like potpack, rbush, and polylabel. I did take a look at Kothic and agree the labeling is great; I did feel like I needed to write yet another Canvas renderer to take advantage of async/await and also eventually expose runtime-extendable custom Symbolizers.
alwayshumans 5y ago
You can already host vector tiles within S3 that work with Mapbox and other libraries. Not really sure what problem this is trying to solve.
- xrd 5y ago
  Is this a simple process? Can you share a link to how this is done, I'm interested. I always assumed it was fairly complicated.
  - alohec 5y ago
    At least for simple GeoJSON you can use tippecanoe (https://github.com/mapbox/tippecanoe https://github.com/mapbox/tippecanoe) with --output-to-directory to get individual .pbf tiles. Most rendering clients will have some scheme where you can provide a root url and then the tiles need to be stored in some defined structure beneath that (e.g. root/z/x/y.pbf) (posted something similar to another response)
  - gorbypark 5y ago
    I’ve used the utility tippecanoe with success to generate vector tiles from geojson sources and am hosting them statically. It’s been overall pretty easy. The only gotcha I can remember is I needed to pass the no compression flag since mapbox gl can’t read compressed tiles (and the file host will gzip everything anyways).
    - d-yakovlev 5y ago
      What issues have you run into with gzipped tiles? I store a lot of gzipped individual tiles for use in Mapbox GL maps on S3 with no problem - the only necessary change was making sure that S3 was emitting `content-encoding` headers.
      - gorbypark 5y ago
        I couldn’t get the tiles to show up at all when they were compressed by tippecanoe. I’m not sure what the problems is, to be honest.
tppiotrowski 5y ago
Super excited for your project. I have been using map tiles from Mapbox, MapTiler, OSM Buildings for my project and am coming to the point that I need to merge elevation data with building height data in order to cast realistic shadows across the earth. Unfortunately elevation data and building data come from two different tile sets from two different tile providers and it's a lot of download/processing overhead to merge the two data sources. I need to make my own tiles! I spent part of today on the OSM wiki trying to wrap my head around Mapnik, how tiles are generated, how I am going to host this cheaply because my project is non-commercial, and then the HN gods smiled on me. Thank you for taking this on and I'm coming along for this journey!
- gorbypark 5y ago
  I’ve had success using tippecanoe to generate mvt tiles from geojson sources, which in turn came from various sources (osm and otherwise) from QGIS. I didn’t look into this project too much but it appears to be bundling mvt tiles into a single file? Either way, straight up mvt tiles are easy to generate and can be easily hosted statically on any web server/s3 bucket/cdn.
  - tppiotrowski 5y ago
    Thanks for sharing your experience. Much of this vocabulary is still foreign to me but you've given me some the relevant keywords to Google. One of the main challenges when entering a highly specialised field is finding the vocabulary to express yourself.
    - schoenobates 5y ago
      I'd also recommend having a look at https://openmaptiles.org/ https://openmaptiles.org/. They have workflows and editors for working with MVT and Maplibre. You can combine a base map from maplibre with other sources using Openlayers [0] [0]: https://openlayers.org/en/latest/examples/mapbox-layer.html https://openlayers.org/en/latest/examples/mapbox-layer.html
- TurkishPoptart 5y ago
ed25519FUUU 5y ago
What’s the easiest way to self-host OSM data to use like a offline google maps?
hardwaresofton 5y ago
Does anyone have a really good (maybe illustrated?) guide/short reference to the world of GIS? Every time I see GIS stuff mentioned the problem space itself, solution jargon, and various movements in the ecosystem (projects being unsupported, stewards/companies changing etc) always throws me for a loop. It's certainly the case that I just haven't built enough mapping-related software to be comfortable but if there's a resource out there that's like "baby's first GIS application" I'd love to read through it and try to ram the concepts into my gray matter again for the day when I do work on one.
- SteveCoast 5y ago
  I didn't know any GIS when I started OpenStreetMap and I think it was for the better. A fresh approach often yields a lot of progress.
  - hardwaresofton 5y ago
    > when I started OpenStreetMap > Created by Steve Coast in the UK in 2004[0] Thanks for starting this way back then, the amount of value created directly and indirectly is immeasurable. That said, I'm very doubtful of the likelihood of me contributing something novel and useful to the GIS world -- I'm keep to learn from and stand on the shoulders of giants that got us here for at least the first bit of my foray into GIS before trying to contribute something fresh, RTFM and all that. To be fair it has taken 15 years to whittle down this particular wheel, starting from scratch feels almost disrespectful -- I'll leave that for younger explorers. [0]: https://en.wikipedia.org/wiki/OpenStreetMap https://en.wikipedia.org/wiki/OpenStreetMap
- bdon 5y ago
  Author of the OP here. I don't have a GIS background in particular; I do have a decade of experience building web mapping applications, and here's some of the foundational software I use: http://leafletjs.com http://leafletjs.com https://shapely.readthedocs.io/ https://shapely.readthedocs.io/ https://github.com/libgeos/geos https://github.com/libgeos/geos https://rasterio.readthedocs.io/ https://rasterio.readthedocs.io/ https://gdal.org https://gdal.org http://www.angusj.com/delphi/clipper.php http://www.angusj.com/delphi/clipper.php
Lammy 5y ago
This looks like exactly what I was looking for as the base for a geolocation visualization of photo galleries on my Jekyll site. Thanks!
canada_dry 5y ago
Got the demo up and running. I've been using leaflet for my map tiling and it just works. The advantage of promomaps being offline is great. So, now... how does one add their own POI's, routes, polygons??
- bdon 5y ago
  Author here, so there are a few options: 1) If your information belongs in OpenStreetMap, you can add it via an editor like the web editor at https://openstreetmap.org https://openstreetmap.org - this will also benefit all other OSM users. You can then "refresh" your Protomaps download to get a new map. 2) If there isn't many point and polygons, it may sense to add them as Leaflet layers, especially if you want them to be interactive 3) Other options are creating vector tiles of your own data and merging or displaying them in the renderer (https://github.com/mapbox/tippecanoe https://github.com/mapbox/tippecanoe is a great tool to do this from GeoJSON) but I don't have much to support this yet.
  - canada_dry 5y ago
    Thanks for the prompt reply (and for the awesome tool). I'm getting old LOL... my question re adding POIs, etc. was a brain-fart. As you note, I'll just use leaflet layers!!
  - canada_dry 5y ago
    Hi again Brandon! Been playing around a bit with the protomap's pmtiles. I notice that POIs have bbox, geom, and just a couple addt'l properties (e.g. name, amenity). Would it be possible to have the option of exporting the street address into the properties of each poi? I'd like to be able to (fully offline) generate a delivery route given a list of addresses (that fall within the specific pmtiles region). Any suggestions or possibility of including this data?
    - bdon 5y ago
      I am working on making the set of information included in the tiles configurable via a JSON definition, kind of like Overpass Turbo if you've used that. The reason I'm doing it this way is because data completeness and performance - the total size of the tiles - are direct tradeoffs. If I keep augmenting the dataset with tags people request, they'll become too big for everyone! So I'm exploring this idea of "a la carte" tags in tile sets, but am still gathering requirements. Can you email me at brandon@protomaps.com with more details of the OSM tags you want, as well as the area you're interested in so we can work together on checking out the data availability there?
airswimmer 5y ago
As it was said > "In about five minutes, you can select any area in the world and get a self-contained map that runs locally, offline or serverlessly on S3 - check out the Getting Started guide." 5 mins? It took me a night to build the map system using openstreetmap in the original way
hijklmno 5y ago
Rooting for this!
ambientlight 5y ago
Congrats for getting to the front page, would be cool to read the story how you are building protomaps as an indie dev on indie hackers or somewhere.
- bdon 5y ago
  Thank you!
pharmakom 5y ago
Question that maybe HN can answer: If I want to build a free open-source web application with a map view that can be annotated, works on mobile etc and uses Open Street Map data, what are the best options right now?
- bdon 5y ago
  For annotation UIs on the web I would check out how http://geojson.io http://geojson.io works, the implementation is on GitHub: http://github.com/mapbox/geojson.io http://github.com/mapbox/geojson.io
- 1337shadow 5y ago
  Great question, GeoDjango with leaflet seems popular.
motiejus 5y ago
Is the author around? Curiously, the editor does not render (well, it renders a black screen) in Firefox 78.9.0. I see this error in devtools: Uncaught SyntaxError: expected expression, got '=' protomaps.min.js:1:52669 On props update af913d4d.0fa8ea22.js:1:7975 ReferenceError: protomaps is not defined (Chromium is fine)
- bdon 5y ago
  Yes, I'll look into this. Can you email me at brandon@protomaps.com or make an issue at http://github.com/protomaps/protomaps.js http://github.com/protomaps/protomaps.js ? I may need to make a get a sourcemap-enabled build online as well.
  - bdon 5y ago
    I think I've fixed this now, and it exposed some other problems in my code and build process... can you verify everything works? Thanks for reporting!
    - motiejus 5y ago
      It worked! Thanks for a swift turnaround.
distalx 5y ago
Also, there is this comprehensive resource available on OSM. https://switch2osm.org/ https://switch2osm.org/
dvdkon 5y ago
In the past I made a map of train lines in and around Prague, which required quite a complex stack, since the information I wanted was in relations. How does Protomaps deal with relations?
- bdon 5y ago
  It can deal with relations more flexibly than PostGIS-based workflows, because the tiles are generated from a near lossless representation of OSM instead of having to convert to OGC Simple Features first. Can you email me at brandon@protomaps.com with what specific tags and relations you are interested in? I want to support transit layers as part of basemaps and ideally this is an option when generating a PMTiles archive.
jamesrr39 5y ago
Really interesting project/product! It's really nice to see new ways of rendering OSM data, and alternative map services. I am developing another way to run your own map server (without mapnik!) from a low-end computer (https://github.com/jamesrr39/ownmap-app https://github.com/jamesrr39/ownmap-app in case you're interested), but a little bit different to you in that I'm currently focusing on rendering raster tiles. I'm really happy I saw this thread, however, I hadn't heard of the OSM Express, MBTiles or PMTiles formats before and was/am rolling my own format, so it's really interesting looking at some ideas from these. Thanks again for sharing your project!
orra 5y ago
On the off chance the author is still hanging around here: Do you mean it when you say on the blog that OSMX is faster than a relational database? e.g. faster than PostGIS. And I presume you mean because of the up front cost of conversion?
- bdon 5y ago
  I gave a talk on this subject early on in this project, you can watch it here: https://2019.stateofthemap.us/program/sun/osm-express-a-spatial-file-format-for-the-planet.html https://2019.stateofthemap.us/program/sun/osm-express-a-spat... I personally avoid blanket statements about a certain design being faster than relational databases; for this project as a whole I try to take a "systems programming" approach; I make things faster by doing less work, in this case because I am designing for one specific use case and no more. For OSMX vs. PostGIS, this entails: 1. PostGIS is a client-server database, OSMX is embedded (in-process). PostGIS must be written defensively for cases like multiple writers; this isn't important for a system that occasionally has a single writer. 2. PostGIS indexes must be built for arbitrary query plans, OSMX only has two kinds of access patterns: by primary key and by S2 cell. S2 cell indexing is an approximation, but will always return more data than you asked for and never less. This is an acceptable downside. 3. PostGIS must treat geodata as OGC Simple Features like points, lines and polygons, this involves lossy conversion from the OSM data model, which is topological. 4. OSMX is built on LMDB, which is an outstanding and underrated piece of database tech (I think the author posts on HN). It's paired with a message format that does not require deserialization (CapnProto) so certain operations common in working with OSM data, such as accessing the coordinates for an OSM node, do not need to malloc for each node. OSM data can be many times larger than RAM; memory-mapping the entire database means you don't need to write your own caching layer; spatial access patterns exhibit high locality since cells are arranged on space-filling curves.
  - orra 5y ago
    Thank you for the detailed answer. I have tinkered with PostGIS before, and it seemed great. Nonetheless, you make several good points about how a special purpose tool/format can be faster by avoiding certain work. As for LMDB, I haven't used it, but I have heard about it previously. A full-text search server called MeiliSearch uses it internally. --- Separately, I think your broader project is really cool. Simplifying the whole end to end process is incredibly useful. I have previously thought "I would love to host a regional map, but I have no idea how".
hutch120 5y ago
@bdon Awesome! The geospatial industry is in need of simpler tooling for specific jobs such as generating tilesets. Currently complex tools such as QGIS and GDAL do everything, but the learning curve is very steep. An addon you might consider a feature to drop an image (and adjust it to fit) over the OSM data to generate a XYZ tilesets directly on the customer AWS S3 (or equivalent storage). This might be a good way to monetise your offering... I know my customers would buy it.

508 points by hampelm 5y ago

> Existing formats like MBTiles are based on SQLite. This limits the ability to self-host maps to those confident running a server in production. Is that comical, or is there a deeper meaning I don't understand?

leokennis 5y ago
I read it as: existing solutions are too hard for simple people with less technical skill.
- deleted 5y ago
  [deleted]
- bdon 5y ago
  Author here; a narrower way to express what I meant: A frontend developer who wants to put a map in their application and wants control over the hosting and data faces some obstacles when using MBtiles: * Need a virtual machine and disk space * Need to choose a particular program to read the MBTiles and serve it via HTTP * Might need reverse proxy and service monitor (systemd or sysv init) to keep it running 24/7 * Need to deal with SSL termination The popularity of serverless platforms = evidence there's a strong market for delivering software without all of this.
codetrotter 5y ago
Probably what they mean is that they want to be able to host the maps as static files that are served to the web. So that you can FTP some html, js, css, images and map data files to a web host and call it a day.
rakoo 5y ago
AFAIU MBTiles still need some application server to transform x/y/z into proper tiles to be served, which means provisioning a server, scaling it, maintaning it, and all the fun that comes with it; PMTiles is a simple blob that the frontend will download (only desired parts) and render directly in the browser.
- alohec 5y ago
  You can use tippecanoe (https://github.com/mapbox/tippecanoe https://github.com/mapbox/tippecanoe) with the --output-to-directory flag to output the individual .pbf tiles instead of a single MBTiles file. These tiles can then be hosted on something like s3 and your map-rendering client can query the necessary tiles, no server required.
incanus77 5y ago
Originally, MBTiles was designed for mobile offline. Then, Mapbox started serving directly from the SQLite for web & other clients, creating the Node.js SQLite bindings in the process. Eventually, they had an unpacker for MBTiles uploads that would stick the tiles in CDN. So, really no more complex than old school static content hosting. Source: I created MBTiles ;-)
- bdon 5y ago
  Author of OP here, I use the tile unpacker as well! I didn't point this out in the OP, but I found the ergonomics of uploading tiles to S3 etc get bad once you're over a couple tens of thousands of tiles, as each upload has some overhead. I've tested PMTiles on S3 successfully with hundreds of millions of tiles. Deduplication is also important to save space on redundant ocean/earth tiles; MBTiles can deduplicate via views, PMTiles deduplicates by reference, but unpacked Z/X/Y tiles on S3 can't.
  - incanus77 5y ago
    Neat! I'm hoping to dig into your design more... looks great.
  - TheTaytay 5y ago
    When I was playing around with hosting rendered Minecraft slippy maps, I found the upload overhead to S3 to be pretty poor due to the “lots of tiny files” problem. (It renders as hundreds of thousands of rasterized image files for the various zoom levels). You could parallelize it, but it was still poor, and made me want to come up with a way to pack them into fewer files and fetch portions of them at render time so that the upload could be faster. It sounds like that’s what you did for the the PMTiles format?
    - bdon 5y ago
      That's exactly what PMTiles is for, and there's nothing in it specific to maps or geographic coordinates, just a Z/X/Y pyramid structure. I chose to support Leaflet for now because it is very popular; there's other viewers like OpenSeadragon for which a decoder plugin would be possible.
deleted 5y ago
[deleted]
jleedev 5y ago
For comparison, Cloud Optimized GeoTIFF (COG) is specifically designed to be readable with HTTP Range requests, so the single file can be placed on S3 and be immediately usable. As others pointed out, the MVTs inside can be unpacked for static serving. But a SQLite database is not itself usable in this way.
MrMartian 5y ago
So I fee like it actually missed the mark as to why it's so impactful. When you get to global scale tiling with building accuracy, you literally end up with hundreds of millions of tiles. The upload time / cost per upload STACKS. However, the tilesets combined usually sits between 100GB and 200GB. ONE upload of ~150GB seems is faster and more cost efficient via S3.

c0nsumer 5y ago

This is really interesting to me. I do a lot of mapping of natural surface single track trails (for hiking and mountain biking) along the way be sure to properly name things, add intersection markers, and create relations (routes and superroutes) to document the systems. The biggest downside thus far is a good way of displaying the data. mtb.waymarkedtrails.org gets close, but I'd really like something that displays relations in the specified color, has sane intersection markers, etc. I feel like there should be something out there which effectively lets me design a style sheet and then apply it to a map which I embed somewhere (maybe even in an app). This... feels like it? Or if not this, can any of you recommend another system? Ideally I'd love something I can self-host. I tried getting a copy of waymarkedtrails going and then modifying that, but it's a bit too specialized and I kept running into quirks getting the toolchain set up. EDIT: Looks like this might not include the highway=path data that I need, much relations. Alas.

ourguile 5y ago
Have you used GaiaGPS at all? I use it along with ridewithgps for looking at trail conditions and mapping points of interest on my rides and hikes. You can import your own custom map layers as well. The developers are fairly consistently adding new content as well, might not hurt to reach out.
- c0nsumer 5y ago
  I have! It's great, but it doesn't show route relations for paths.
bdon 5y ago
Author of OP here. Is this information that you're interested in contributing, and is appropriate for, OpenStreetMap? The end goal is to build a system that lets you customize the tileset to include any node, way, relation, or tag in OSM. This gets quite tricky when considering automated cartographic generalization (how such features will appear when zoomed out) but it's one of my next steps.
- 5y ago

brunoqc 5y ago

> Pricing > coming soon...

kennydude 5y ago
At least it's open source and self hostable so if it's too expensive for you, you can host it yourself
- brunoqc 5y ago
  Ah fair enough.

iandanforth 5y ago

Very interesting, I'm currently using Google My Maps as a dead-simple custom mapping tool. I'm using it to collect layers of information as I conduct a nationwide housing search. It would be great if there were a solution that was as simple to get started with as My Maps but also had the flexibility to easily add custom data layers programmatically.

jMyles 5y ago
Curious about your tool chain for real estate metadata. Is there a way to get a more retail-like experience and not have to deal so much with agents?
- iandanforth 5y ago
  Paying a guy in Kenya to scrape data for me. Swear I tried to pay every national real estate API out there and no one is interested in talking to you at less than 7k a month and a full on lead gen business.
loa_in_ 5y ago
> It would be great if there were a solution that was as simple to get started with as My Maps but also had the flexibility to easily add custom data layers programmatically. I believe _with some trickery_ you can use Google Earth for that
- wizzwizz4 5y ago
  The computer version, not the web version, right?
  - loa_in_ 5y ago
    Yes
DoreenMichele 5y ago
I'm curious what sorts of data you are collecting and how it helps to have it in map form rather than some other file format.

mourner 5y ago

This is very nice! Happy to see a few of my libraries in the dependencies :) Also check out https://github.com/kothic/kothic-js https://github.com/kothic/kothic-js, an older but similar attempt at making a full-fledged map renderer using 2D Canvas. It has a pretty good label rendering in particular which you might find useful.

xrd 5y ago
This looks great. I am super excited about the parent tool, but unsure whether/when I can generate these bundles programmatically. I tried the web based tool and it works great, but what if I need to do it many, many times? Can your tool do this easily? Is there a way to generate tiles (tiles.pmtiles) that are imported in the same way? If I had stumbled on your project, I would not have been enthusiastic about it with just the writeup as-is. Seeing the parent project makes me see the possibilities: a completely independent mapping solution, free of any server/service tie in and additional costs. That is super exciting, and I would love to know if and how your project compares.
- bdon 5y ago
  I'd like to understand what you mean by doing it many times - do you mean for different areas of the world? I haven't thought about API access to the bundles in detail yet, but you can contact me via email at brandon@protomaps.com if you have some idea of what you need.
bdon 5y ago
Thank you! Really appreciate the commitment to simplicity among the libraries I'm using like potpack, rbush, and polylabel. I did take a look at Kothic and agree the labeling is great; I did feel like I needed to write yet another Canvas renderer to take advantage of async/await and also eventually expose runtime-extendable custom Symbolizers.

alwayshumans 5y ago

You can already host vector tiles within S3 that work with Mapbox and other libraries. Not really sure what problem this is trying to solve.

xrd 5y ago
Is this a simple process? Can you share a link to how this is done, I'm interested. I always assumed it was fairly complicated.
- alohec 5y ago
  At least for simple GeoJSON you can use tippecanoe (https://github.com/mapbox/tippecanoe https://github.com/mapbox/tippecanoe) with --output-to-directory to get individual .pbf tiles. Most rendering clients will have some scheme where you can provide a root url and then the tiles need to be stored in some defined structure beneath that (e.g. root/z/x/y.pbf) (posted something similar to another response)
- gorbypark 5y ago
  I’ve used the utility tippecanoe with success to generate vector tiles from geojson sources and am hosting them statically. It’s been overall pretty easy. The only gotcha I can remember is I needed to pass the no compression flag since mapbox gl can’t read compressed tiles (and the file host will gzip everything anyways).
  - d-yakovlev 5y ago
    What issues have you run into with gzipped tiles? I store a lot of gzipped individual tiles for use in Mapbox GL maps on S3 with no problem - the only necessary change was making sure that S3 was emitting `content-encoding` headers.
    - gorbypark 5y ago
      I couldn’t get the tiles to show up at all when they were compressed by tippecanoe. I’m not sure what the problems is, to be honest.

tppiotrowski 5y ago

Super excited for your project. I have been using map tiles from Mapbox, MapTiler, OSM Buildings for my project and am coming to the point that I need to merge elevation data with building height data in order to cast realistic shadows across the earth. Unfortunately elevation data and building data come from two different tile sets from two different tile providers and it's a lot of download/processing overhead to merge the two data sources. I need to make my own tiles! I spent part of today on the OSM wiki trying to wrap my head around Mapnik, how tiles are generated, how I am going to host this cheaply because my project is non-commercial, and then the HN gods smiled on me. Thank you for taking this on and I'm coming along for this journey!

gorbypark 5y ago
I’ve had success using tippecanoe to generate mvt tiles from geojson sources, which in turn came from various sources (osm and otherwise) from QGIS. I didn’t look into this project too much but it appears to be bundling mvt tiles into a single file? Either way, straight up mvt tiles are easy to generate and can be easily hosted statically on any web server/s3 bucket/cdn.
- tppiotrowski 5y ago
  Thanks for sharing your experience. Much of this vocabulary is still foreign to me but you've given me some the relevant keywords to Google. One of the main challenges when entering a highly specialised field is finding the vocabulary to express yourself.
  - schoenobates 5y ago
    I'd also recommend having a look at https://openmaptiles.org/ https://openmaptiles.org/. They have workflows and editors for working with MVT and Maplibre. You can combine a base map from maplibre with other sources using Openlayers [0] [0]: https://openlayers.org/en/latest/examples/mapbox-layer.html https://openlayers.org/en/latest/examples/mapbox-layer.html
TurkishPoptart 5y ago

ed25519FUUU 5y ago

What’s the easiest way to self-host OSM data to use like a offline google maps?

hardwaresofton 5y ago

Does anyone have a really good (maybe illustrated?) guide/short reference to the world of GIS? Every time I see GIS stuff mentioned the problem space itself, solution jargon, and various movements in the ecosystem (projects being unsupported, stewards/companies changing etc) always throws me for a loop. It's certainly the case that I just haven't built enough mapping-related software to be comfortable but if there's a resource out there that's like "baby's first GIS application" I'd love to read through it and try to ram the concepts into my gray matter again for the day when I do work on one.

SteveCoast 5y ago
I didn't know any GIS when I started OpenStreetMap and I think it was for the better. A fresh approach often yields a lot of progress.
- hardwaresofton 5y ago
  > when I started OpenStreetMap > Created by Steve Coast in the UK in 2004[0] Thanks for starting this way back then, the amount of value created directly and indirectly is immeasurable. That said, I'm very doubtful of the likelihood of me contributing something novel and useful to the GIS world -- I'm keep to learn from and stand on the shoulders of giants that got us here for at least the first bit of my foray into GIS before trying to contribute something fresh, RTFM and all that. To be fair it has taken 15 years to whittle down this particular wheel, starting from scratch feels almost disrespectful -- I'll leave that for younger explorers. [0]: https://en.wikipedia.org/wiki/OpenStreetMap https://en.wikipedia.org/wiki/OpenStreetMap
bdon 5y ago
Author of the OP here. I don't have a GIS background in particular; I do have a decade of experience building web mapping applications, and here's some of the foundational software I use: http://leafletjs.com http://leafletjs.com https://shapely.readthedocs.io/ https://shapely.readthedocs.io/ https://github.com/libgeos/geos https://github.com/libgeos/geos https://rasterio.readthedocs.io/ https://rasterio.readthedocs.io/ https://gdal.org https://gdal.org http://www.angusj.com/delphi/clipper.php http://www.angusj.com/delphi/clipper.php

Lammy 5y ago

This looks like exactly what I was looking for as the base for a geolocation visualization of photo galleries on my Jekyll site. Thanks!

canada_dry 5y ago

Got the demo up and running. I've been using leaflet for my map tiling and it just works. The advantage of promomaps being offline is great. So, now... how does one add their own POI's, routes, polygons??

bdon 5y ago
Author here, so there are a few options: 1) If your information belongs in OpenStreetMap, you can add it via an editor like the web editor at https://openstreetmap.org https://openstreetmap.org - this will also benefit all other OSM users. You can then "refresh" your Protomaps download to get a new map. 2) If there isn't many point and polygons, it may sense to add them as Leaflet layers, especially if you want them to be interactive 3) Other options are creating vector tiles of your own data and merging or displaying them in the renderer (https://github.com/mapbox/tippecanoe https://github.com/mapbox/tippecanoe is a great tool to do this from GeoJSON) but I don't have much to support this yet.
- canada_dry 5y ago
  Thanks for the prompt reply (and for the awesome tool). I'm getting old LOL... my question re adding POIs, etc. was a brain-fart. As you note, I'll just use leaflet layers!!
- canada_dry 5y ago
  Hi again Brandon! Been playing around a bit with the protomap's pmtiles. I notice that POIs have bbox, geom, and just a couple addt'l properties (e.g. name, amenity). Would it be possible to have the option of exporting the street address into the properties of each poi? I'd like to be able to (fully offline) generate a delivery route given a list of addresses (that fall within the specific pmtiles region). Any suggestions or possibility of including this data?
  - bdon 5y ago
    I am working on making the set of information included in the tiles configurable via a JSON definition, kind of like Overpass Turbo if you've used that. The reason I'm doing it this way is because data completeness and performance - the total size of the tiles - are direct tradeoffs. If I keep augmenting the dataset with tags people request, they'll become too big for everyone! So I'm exploring this idea of "a la carte" tags in tile sets, but am still gathering requirements. Can you email me at brandon@protomaps.com with more details of the OSM tags you want, as well as the area you're interested in so we can work together on checking out the data availability there?

airswimmer 5y ago

As it was said > "In about five minutes, you can select any area in the world and get a self-contained map that runs locally, offline or serverlessly on S3 - check out the Getting Started guide." 5 mins? It took me a night to build the map system using openstreetmap in the original way

hijklmno 5y ago

Rooting for this!

ambientlight 5y ago

Congrats for getting to the front page, would be cool to read the story how you are building protomaps as an indie dev on indie hackers or somewhere.

bdon 5y ago
Thank you!

pharmakom 5y ago

Question that maybe HN can answer: If I want to build a free open-source web application with a map view that can be annotated, works on mobile etc and uses Open Street Map data, what are the best options right now?

bdon 5y ago
For annotation UIs on the web I would check out how http://geojson.io http://geojson.io works, the implementation is on GitHub: http://github.com/mapbox/geojson.io http://github.com/mapbox/geojson.io
1337shadow 5y ago
Great question, GeoDjango with leaflet seems popular.

motiejus 5y ago

Is the author around? Curiously, the editor does not render (well, it renders a black screen) in Firefox 78.9.0. I see this error in devtools: Uncaught SyntaxError: expected expression, got '=' protomaps.min.js:1:52669 On props update af913d4d.0fa8ea22.js:1:7975 ReferenceError: protomaps is not defined (Chromium is fine)

bdon 5y ago
Yes, I'll look into this. Can you email me at brandon@protomaps.com or make an issue at http://github.com/protomaps/protomaps.js http://github.com/protomaps/protomaps.js ? I may need to make a get a sourcemap-enabled build online as well.
- bdon 5y ago
  I think I've fixed this now, and it exposed some other problems in my code and build process... can you verify everything works? Thanks for reporting!
  - motiejus 5y ago
    It worked! Thanks for a swift turnaround.

distalx 5y ago

Also, there is this comprehensive resource available on OSM. https://switch2osm.org/ https://switch2osm.org/

dvdkon 5y ago

In the past I made a map of train lines in and around Prague, which required quite a complex stack, since the information I wanted was in relations. How does Protomaps deal with relations?

bdon 5y ago
It can deal with relations more flexibly than PostGIS-based workflows, because the tiles are generated from a near lossless representation of OSM instead of having to convert to OGC Simple Features first. Can you email me at brandon@protomaps.com with what specific tags and relations you are interested in? I want to support transit layers as part of basemaps and ideally this is an option when generating a PMTiles archive.

jamesrr39 5y ago

Really interesting project/product! It's really nice to see new ways of rendering OSM data, and alternative map services. I am developing another way to run your own map server (without mapnik!) from a low-end computer (https://github.com/jamesrr39/ownmap-app https://github.com/jamesrr39/ownmap-app in case you're interested), but a little bit different to you in that I'm currently focusing on rendering raster tiles. I'm really happy I saw this thread, however, I hadn't heard of the OSM Express, MBTiles or PMTiles formats before and was/am rolling my own format, so it's really interesting looking at some ideas from these. Thanks again for sharing your project!

orra 5y ago

On the off chance the author is still hanging around here: Do you mean it when you say on the blog that OSMX is faster than a relational database? e.g. faster than PostGIS. And I presume you mean because of the up front cost of conversion?

bdon 5y ago
I gave a talk on this subject early on in this project, you can watch it here: https://2019.stateofthemap.us/program/sun/osm-express-a-spatial-file-format-for-the-planet.html https://2019.stateofthemap.us/program/sun/osm-express-a-spat... I personally avoid blanket statements about a certain design being faster than relational databases; for this project as a whole I try to take a "systems programming" approach; I make things faster by doing less work, in this case because I am designing for one specific use case and no more. For OSMX vs. PostGIS, this entails: 1. PostGIS is a client-server database, OSMX is embedded (in-process). PostGIS must be written defensively for cases like multiple writers; this isn't important for a system that occasionally has a single writer. 2. PostGIS indexes must be built for arbitrary query plans, OSMX only has two kinds of access patterns: by primary key and by S2 cell. S2 cell indexing is an approximation, but will always return more data than you asked for and never less. This is an acceptable downside. 3. PostGIS must treat geodata as OGC Simple Features like points, lines and polygons, this involves lossy conversion from the OSM data model, which is topological. 4. OSMX is built on LMDB, which is an outstanding and underrated piece of database tech (I think the author posts on HN). It's paired with a message format that does not require deserialization (CapnProto) so certain operations common in working with OSM data, such as accessing the coordinates for an OSM node, do not need to malloc for each node. OSM data can be many times larger than RAM; memory-mapping the entire database means you don't need to write your own caching layer; spatial access patterns exhibit high locality since cells are arranged on space-filling curves.
- orra 5y ago
  Thank you for the detailed answer. I have tinkered with PostGIS before, and it seemed great. Nonetheless, you make several good points about how a special purpose tool/format can be faster by avoiding certain work. As for LMDB, I haven't used it, but I have heard about it previously. A full-text search server called MeiliSearch uses it internally. --- Separately, I think your broader project is really cool. Simplifying the whole end to end process is incredibly useful. I have previously thought "I would love to host a regional map, but I have no idea how".

hutch120 5y ago

@bdon Awesome! The geospatial industry is in need of simpler tooling for specific jobs such as generating tilesets. Currently complex tools such as QGIS and GDAL do everything, but the learning curve is very steep. An addon you might consider a feature to drop an image (and adjust it to fit) over the OSM data to generate a XYZ tilesets directly on the customer AWS S3 (or equivalent storage). This might be a good way to monetise your offering... I know my customers would buy it.