4 ms·

For anyone looking to migrate to an OpenStreetMap-backed map and location data provider, https://switch2osm.org https://switch2osm.org has a good overview of ho

For anyone looking to migrate to an OpenStreetMap-backed map and location data provider, https://switch2osm.org https://switch2osm.org has a good overview of how, why, and who is offering services. I’m partial to my own (in bio), of course, but I would be happy to answer questions for anyone looking to switch. Respond or email me (also in bio).

fouc 6y ago
Why is it hard to setup our own tile "provider" ?
- rburhum 6y ago
  Because it is not just installing various server side software components and setting up client software to consume it - you also have to deal with processing data to do it. Once you have done it a few times it is quite easy, but the amount of details for the beginner can be overwhelming.
- ohgodplsno 6y ago
  Rendering tiles for a single country at most levels is a massive database and a few days of processing. Then, network costs are insane.
- lukeqsee 6y ago
  The difficulty depends on what size of market you're trying to serve at what reliability target with geographic constraints, how often you want to update your data, and how many map tiles / second you want to serve. Consider a few different scenarios: 1. If you're trying to serve a single website and map style, with limited traffic in one region, this fairly easily accomplished using a single map tile server using one of the established methods (something like 8 cores, 32GB of RAM, and a few hundred GBs of disk will get you most of the way there). The difficulty is mostly determine how you want to set it up and then provisioning the server to do it. This process will probably take you a few days. Updates will be infrequent, map data will be stale, and speed will be tolerable in your targeted geographic region. 2. Let's say you take a step up. You need to serve global traffic with one style, but only a few requests / second. You can still do this via a single tile server, but now you need a CDN to make it fast. Cache misses will be slow since it has to render and then send the tile to the CDN (up to 500ms of latency due to geographic constraints and cache fill times). 3. Let's say you now need multiple styles, global traffic, 10s of tile requests / second, and good reliabilty. Now you need to distribute your servers, so you'll probably need three or four servers of at least the above mentioned specs, you need to keep them in sync. 4. Now let's say you need multiple styles, global traffic, and up-to-date data. You'll need to keep the OSM data stored locally so you can update it (minimum 500GB in Postgres + PostGIS). You need to distribute this all around the world. You need to handle updates, but do them in sync. All of these are short of actually offering a commercial service for multiple companies, since in that case you need a global service structure, a CDN that's pretty efficient, enough tile servers to handle spikes in demand (pretty common in map tile services), and probably a couple layers of cache to keep everything moving smoothly. There are a few projects that make this easier. For instance, openmaptiles.org is what our solution was modeled after (and we still use their schema, mostly unchanged), but in the end it's hard because it's a lot of data, that requires a lot of compute (either up-front or at a moments notice) to be responsive, and requires a lot of orchestration to keep it all well-oiled.
- bdon 6y ago
  Because user expectations are high for interactive web maps - zooming and panning should retrieve new data near-instantly. OpenStreetMap is an unwieldy but not "big" dataset - it fits easily on a consumer grade SSD. You need to index the dataset so that retrieving a specific slice is extremely fast - doing point in polygon tests, clipping operations on source geometries that have tens of thousands of vertices, etc should not happen at query time. This inevitably means pre-rendering as much as possible. In addition, this needs to work for every intermediate tile level when zooming out - each parent tile covers 4 child tiles, so you need some strategy for decimating the amount of data so that the tile sizes don't increase exponentially as you zoom out. This is beyond a pure computer programming problem and becomes a visual design problem as well - features such as roads or transit layers should form a sensible hierarchy with less important features removed. I've been working on this class of problems for a couple years now and have a related presentation day 2 of the conference mentioned in TFA - details in bio if you'd like to talk more
  - berkes 6y ago
    > OpenStreetMap is an unwieldy but not "big" dataset Do note that while not BigData big, it _is_ large. Your PostgreSQL database (postgis) on your macbook won't be able to import `planet.pbf` in any reasonable time (think: weeks). Your digital-ocean VPS won't have enough diskspace to process a weekly planet.pbf and the free or cheap tier of your RDS won't be able to handle planet.pbf either. http://download.geofabrik.de/ http://download.geofabrik.de/ (also one of the StateOfTheMap conference sponsors) is a good place to find "chopped up" downloads to avoid that: just get only your country, a province or even just one city. On my developer env I always run everything through with `luxembourg.pbf`. or `iceland.pbf`.Luxembourg-latest does still contain nearly 2.5 million datapoints: `osm2geojson luxembourg-latest.osm | wc -l #=> 2417784`
    - bdon 6y ago
      You are describing a problem inherent to using PostgreSQL, not OpenStreetMap itself. My presentation is specifically about my solution to this, which can easily import planet.pbf in 7-8 hours on a laptop with SSD, and can cut extracts for cities and countries like you describe based on minutely data: https://2020.stateofthemap.org/sessions/JDNTHK/ https://2020.stateofthemap.org/sessions/JDNTHK/ https://github.com/protomaps/OSMExpress https://github.com/protomaps/OSMExpress
- usrusr 6y ago
  In addition to everything others have already said, I think one of the hardest parts is continually dealing with occasional upstream outages and API changes. Initial automation setup is exciting, repeatedly revisiting it after it worked just long enough to forget almost everything is a chore. It's a good fit for "as a service" outsourcing to full-time experts.
mariushn 6y ago
In Providers, would you please list other API services, besides tiles? 1. Autocomplete places Mapbox has one, but I found many examples of villages present in OSM, but not in Mapbox. I wrote to support with specific examples and links to OSM, eg https://www.openstreetmap.org/search?query=dalghiu#map=13/45.5713/25.9307 https://www.openstreetmap.org/search?query=dalghiu#map=13/45... but they didn't even reply in 10 days (no support contract). 2. Directions (route, distance & time) Thanks
- carstenhag 6y ago
  Hm that's pretty weird. I have had both a technical and a billing support request, both have been answered within a day. Including a technical analysis on what likely caused a bug. No support contract or payment either.

fouc 6y ago

Why is it hard to setup our own tile "provider" ?

rburhum 6y ago
Because it is not just installing various server side software components and setting up client software to consume it - you also have to deal with processing data to do it. Once you have done it a few times it is quite easy, but the amount of details for the beginner can be overwhelming.
ohgodplsno 6y ago
Rendering tiles for a single country at most levels is a massive database and a few days of processing. Then, network costs are insane.
lukeqsee 6y ago
The difficulty depends on what size of market you're trying to serve at what reliability target with geographic constraints, how often you want to update your data, and how many map tiles / second you want to serve. Consider a few different scenarios: 1. If you're trying to serve a single website and map style, with limited traffic in one region, this fairly easily accomplished using a single map tile server using one of the established methods (something like 8 cores, 32GB of RAM, and a few hundred GBs of disk will get you most of the way there). The difficulty is mostly determine how you want to set it up and then provisioning the server to do it. This process will probably take you a few days. Updates will be infrequent, map data will be stale, and speed will be tolerable in your targeted geographic region. 2. Let's say you take a step up. You need to serve global traffic with one style, but only a few requests / second. You can still do this via a single tile server, but now you need a CDN to make it fast. Cache misses will be slow since it has to render and then send the tile to the CDN (up to 500ms of latency due to geographic constraints and cache fill times). 3. Let's say you now need multiple styles, global traffic, 10s of tile requests / second, and good reliabilty. Now you need to distribute your servers, so you'll probably need three or four servers of at least the above mentioned specs, you need to keep them in sync. 4. Now let's say you need multiple styles, global traffic, and up-to-date data. You'll need to keep the OSM data stored locally so you can update it (minimum 500GB in Postgres + PostGIS). You need to distribute this all around the world. You need to handle updates, but do them in sync. All of these are short of actually offering a commercial service for multiple companies, since in that case you need a global service structure, a CDN that's pretty efficient, enough tile servers to handle spikes in demand (pretty common in map tile services), and probably a couple layers of cache to keep everything moving smoothly. There are a few projects that make this easier. For instance, openmaptiles.org is what our solution was modeled after (and we still use their schema, mostly unchanged), but in the end it's hard because it's a lot of data, that requires a lot of compute (either up-front or at a moments notice) to be responsive, and requires a lot of orchestration to keep it all well-oiled.
bdon 6y ago
Because user expectations are high for interactive web maps - zooming and panning should retrieve new data near-instantly. OpenStreetMap is an unwieldy but not "big" dataset - it fits easily on a consumer grade SSD. You need to index the dataset so that retrieving a specific slice is extremely fast - doing point in polygon tests, clipping operations on source geometries that have tens of thousands of vertices, etc should not happen at query time. This inevitably means pre-rendering as much as possible. In addition, this needs to work for every intermediate tile level when zooming out - each parent tile covers 4 child tiles, so you need some strategy for decimating the amount of data so that the tile sizes don't increase exponentially as you zoom out. This is beyond a pure computer programming problem and becomes a visual design problem as well - features such as roads or transit layers should form a sensible hierarchy with less important features removed. I've been working on this class of problems for a couple years now and have a related presentation day 2 of the conference mentioned in TFA - details in bio if you'd like to talk more
- berkes 6y ago
  > OpenStreetMap is an unwieldy but not "big" dataset Do note that while not BigData big, it _is_ large. Your PostgreSQL database (postgis) on your macbook won't be able to import `planet.pbf` in any reasonable time (think: weeks). Your digital-ocean VPS won't have enough diskspace to process a weekly planet.pbf and the free or cheap tier of your RDS won't be able to handle planet.pbf either. http://download.geofabrik.de/ http://download.geofabrik.de/ (also one of the StateOfTheMap conference sponsors) is a good place to find "chopped up" downloads to avoid that: just get only your country, a province or even just one city. On my developer env I always run everything through with `luxembourg.pbf`. or `iceland.pbf`.Luxembourg-latest does still contain nearly 2.5 million datapoints: `osm2geojson luxembourg-latest.osm | wc -l #=> 2417784`
  - bdon 6y ago
    You are describing a problem inherent to using PostgreSQL, not OpenStreetMap itself. My presentation is specifically about my solution to this, which can easily import planet.pbf in 7-8 hours on a laptop with SSD, and can cut extracts for cities and countries like you describe based on minutely data: https://2020.stateofthemap.org/sessions/JDNTHK/ https://2020.stateofthemap.org/sessions/JDNTHK/ https://github.com/protomaps/OSMExpress https://github.com/protomaps/OSMExpress
usrusr 6y ago
In addition to everything others have already said, I think one of the hardest parts is continually dealing with occasional upstream outages and API changes. Initial automation setup is exciting, repeatedly revisiting it after it worked just long enough to forget almost everything is a chore. It's a good fit for "as a service" outsourcing to full-time experts.

mariushn 6y ago

In Providers, would you please list other API services, besides tiles? 1. Autocomplete places Mapbox has one, but I found many examples of villages present in OSM, but not in Mapbox. I wrote to support with specific examples and links to OSM, eg https://www.openstreetmap.org/search?query=dalghiu#map=13/45.5713/25.9307 https://www.openstreetmap.org/search?query=dalghiu#map=13/45... but they didn't even reply in 10 days (no support contract). 2. Directions (route, distance & time) Thanks

carstenhag 6y ago
Hm that's pretty weird. I have had both a technical and a billing support request, both have been answered within a day. Including a technical analysis on what likely caused a bug. No support contract or payment either.