3 ms·

Because user expectations are high for interactive web maps - zooming and panning should retrieve new data near-instantly. OpenStreetMap is an unwieldy but not

by bdon 6y ago

Because user expectations are high for interactive web maps - zooming and panning should retrieve new data near-instantly. OpenStreetMap is an unwieldy but not "big" dataset - it fits easily on a consumer grade SSD. You need to index the dataset so that retrieving a specific slice is extremely fast - doing point in polygon tests, clipping operations on source geometries that have tens of thousands of vertices, etc should not happen at query time. This inevitably means pre-rendering as much as possible. In addition, this needs to work for every intermediate tile level when zooming out - each parent tile covers 4 child tiles, so you need some strategy for decimating the amount of data so that the tile sizes don't increase exponentially as you zoom out. This is beyond a pure computer programming problem and becomes a visual design problem as well - features such as roads or transit layers should form a sensible hierarchy with less important features removed. I've been working on this class of problems for a couple years now and have a related presentation day 2 of the conference mentioned in TFA - details in bio if you'd like to talk more

berkes 6y ago
> OpenStreetMap is an unwieldy but not "big" dataset Do note that while not BigData big, it _is_ large. Your PostgreSQL database (postgis) on your macbook won't be able to import `planet.pbf` in any reasonable time (think: weeks). Your digital-ocean VPS won't have enough diskspace to process a weekly planet.pbf and the free or cheap tier of your RDS won't be able to handle planet.pbf either. http://download.geofabrik.de/ http://download.geofabrik.de/ (also one of the StateOfTheMap conference sponsors) is a good place to find "chopped up" downloads to avoid that: just get only your country, a province or even just one city. On my developer env I always run everything through with `luxembourg.pbf`. or `iceland.pbf`.Luxembourg-latest does still contain nearly 2.5 million datapoints: `osm2geojson luxembourg-latest.osm | wc -l #=> 2417784`
- bdon 6y ago
  You are describing a problem inherent to using PostgreSQL, not OpenStreetMap itself. My presentation is specifically about my solution to this, which can easily import planet.pbf in 7-8 hours on a laptop with SSD, and can cut extracts for cities and countries like you describe based on minutely data: https://2020.stateofthemap.org/sessions/JDNTHK/ https://2020.stateofthemap.org/sessions/JDNTHK/ https://github.com/protomaps/OSMExpress https://github.com/protomaps/OSMExpress
  - berkes 6y ago
    Indeed, a lot is due to PostGIS. I'm using mimirsbrunn[1] a lot, lately, with Elasticsearch as storage; especially because it is fast. It does lack a processor that listens to changesets and imports those, though; so I still need to run a nightly rebuild of the entire database. Thanks for pointing me to your StateOfTheMap session. I'll "attend" it for sure. [1] https://github.com/CanalTP/mimirsbrunn https://github.com/CanalTP/mimirsbrunn
- aembleton 6y ago
  > Your macbook won't be able to import `planet.pbf` in any reasonable time (think: weeks) I guess it depends on specs. I recently did it, importing planet.pbf into this PostGis docker image [1]. It took 44 hours and consumed ~750GB of disk. OS: Ubuntu 20.04 CPU: Intel i7-8550U RAM: 16GB 1. https://hub.docker.com/r/postgis/postgis https://hub.docker.com/r/postgis/postgis

by bdon 6y ago

berkes 6y ago

> OpenStreetMap is an unwieldy but not "big" dataset Do note that while not BigData big, it _is_ large. Your PostgreSQL database (postgis) on your macbook won't be able to import `planet.pbf` in any reasonable time (think: weeks). Your digital-ocean VPS won't have enough diskspace to process a weekly planet.pbf and the free or cheap tier of your RDS won't be able to handle planet.pbf either. http://download.geofabrik.de/ http://download.geofabrik.de/ (also one of the StateOfTheMap conference sponsors) is a good place to find "chopped up" downloads to avoid that: just get only your country, a province or even just one city. On my developer env I always run everything through with `luxembourg.pbf`. or `iceland.pbf`.Luxembourg-latest does still contain nearly 2.5 million datapoints: `osm2geojson luxembourg-latest.osm | wc -l #=> 2417784`

bdon 6y ago
You are describing a problem inherent to using PostgreSQL, not OpenStreetMap itself. My presentation is specifically about my solution to this, which can easily import planet.pbf in 7-8 hours on a laptop with SSD, and can cut extracts for cities and countries like you describe based on minutely data: https://2020.stateofthemap.org/sessions/JDNTHK/ https://2020.stateofthemap.org/sessions/JDNTHK/ https://github.com/protomaps/OSMExpress https://github.com/protomaps/OSMExpress
- berkes 6y ago
  Indeed, a lot is due to PostGIS. I'm using mimirsbrunn[1] a lot, lately, with Elasticsearch as storage; especially because it is fast. It does lack a processor that listens to changesets and imports those, though; so I still need to run a nightly rebuild of the entire database. Thanks for pointing me to your StateOfTheMap session. I'll "attend" it for sure. [1] https://github.com/CanalTP/mimirsbrunn https://github.com/CanalTP/mimirsbrunn
aembleton 6y ago
> Your macbook won't be able to import `planet.pbf` in any reasonable time (think: weeks) I guess it depends on specs. I recently did it, importing planet.pbf into this PostGis docker image [1]. It took 44 hours and consumed ~750GB of disk. OS: Ubuntu 20.04 CPU: Intel i7-8550U RAM: 16GB 1. https://hub.docker.com/r/postgis/postgis https://hub.docker.com/r/postgis/postgis