3 ms·

> OpenStreetMap is an unwieldy but not "big" dataset Do note that while not BigData big, it _is_ large. Your PostgreSQL database (postgis) on your macbook won'

by berkes 6y ago

> OpenStreetMap is an unwieldy but not "big" dataset Do note that while not BigData big, it _is_ large. Your PostgreSQL database (postgis) on your macbook won't be able to import `planet.pbf` in any reasonable time (think: weeks). Your digital-ocean VPS won't have enough diskspace to process a weekly planet.pbf and the free or cheap tier of your RDS won't be able to handle planet.pbf either. http://download.geofabrik.de/ http://download.geofabrik.de/ (also one of the StateOfTheMap conference sponsors) is a good place to find "chopped up" downloads to avoid that: just get only your country, a province or even just one city. On my developer env I always run everything through with `luxembourg.pbf`. or `iceland.pbf`.Luxembourg-latest does still contain nearly 2.5 million datapoints: `osm2geojson luxembourg-latest.osm | wc -l #=> 2417784`

bdon 6y ago
You are describing a problem inherent to using PostgreSQL, not OpenStreetMap itself. My presentation is specifically about my solution to this, which can easily import planet.pbf in 7-8 hours on a laptop with SSD, and can cut extracts for cities and countries like you describe based on minutely data: https://2020.stateofthemap.org/sessions/JDNTHK/ https://2020.stateofthemap.org/sessions/JDNTHK/ https://github.com/protomaps/OSMExpress https://github.com/protomaps/OSMExpress
- berkes 6y ago
  Indeed, a lot is due to PostGIS. I'm using mimirsbrunn[1] a lot, lately, with Elasticsearch as storage; especially because it is fast. It does lack a processor that listens to changesets and imports those, though; so I still need to run a nightly rebuild of the entire database. Thanks for pointing me to your StateOfTheMap session. I'll "attend" it for sure. [1] https://github.com/CanalTP/mimirsbrunn https://github.com/CanalTP/mimirsbrunn
aembleton 6y ago
> Your macbook won't be able to import `planet.pbf` in any reasonable time (think: weeks) I guess it depends on specs. I recently did it, importing planet.pbf into this PostGis docker image [1]. It took 44 hours and consumed ~750GB of disk. OS: Ubuntu 20.04 CPU: Intel i7-8550U RAM: 16GB 1. https://hub.docker.com/r/postgis/postgis https://hub.docker.com/r/postgis/postgis

by berkes 6y ago

bdon 6y ago

You are describing a problem inherent to using PostgreSQL, not OpenStreetMap itself. My presentation is specifically about my solution to this, which can easily import planet.pbf in 7-8 hours on a laptop with SSD, and can cut extracts for cities and countries like you describe based on minutely data: https://2020.stateofthemap.org/sessions/JDNTHK/ https://2020.stateofthemap.org/sessions/JDNTHK/ https://github.com/protomaps/OSMExpress https://github.com/protomaps/OSMExpress

berkes 6y ago
Indeed, a lot is due to PostGIS. I'm using mimirsbrunn[1] a lot, lately, with Elasticsearch as storage; especially because it is fast. It does lack a processor that listens to changesets and imports those, though; so I still need to run a nightly rebuild of the entire database. Thanks for pointing me to your StateOfTheMap session. I'll "attend" it for sure. [1] https://github.com/CanalTP/mimirsbrunn https://github.com/CanalTP/mimirsbrunn

aembleton 6y ago

> Your macbook won't be able to import `planet.pbf` in any reasonable time (think: weeks) I guess it depends on specs. I recently did it, importing planet.pbf into this PostGis docker image [1]. It took 44 hours and consumed ~750GB of disk. OS: Ubuntu 20.04 CPU: Intel i7-8550U RAM: 16GB 1. https://hub.docker.com/r/postgis/postgis https://hub.docker.com/r/postgis/postgis