5 ms·

Overpass Turbo: A Web Based Data Mining Tool for OpenStreetMap

215 points by stefankuehnel 3y ago

rony989 3y ago
[dead]
dabber 3y ago
For anyone else curious: Source code for the backend here: https://github.com/drolbr/Overpass-API https://github.com/drolbr/Overpass-API Found from this link listed in the help modal: https://overpass-api.de/ https://overpass-api.de/ (also links to two additional frontends.)
shpx 3y ago
I've been asking ChatGPT to write Overpass Turbo queries and it does a good job.
- maelito 3y ago
  Yes. It was way harder with the other LLMs. Except Claude maybe.
- zarazas 3y ago
  There is also this: https://overpassnl.schumann.pub/ https://overpassnl.schumann.pub/
- Raphaellll 3y ago
  we systematically tested this: https://arxiv.org/pdf/2308.16060.pdf https://arxiv.org/pdf/2308.16060.pdf
  - maelito 3y ago
    Where can we use it ? I couldn't find a link. It is closed-source ? OK, I had a hard time understanding that the link given above is the demo of this paper. https://overpassnl.schumann.pub https://overpassnl.schumann.pub. Very interesting. Quite surprising too : it can find non trivial searches like "bars with darts" ("bars à fléchette" in French), but not easy ones ("fromagerie" or "dermato"). More complex queries did not work : "cafes close to the river" returned nothing, neither did "touristic site close to a pedestrian area" or "touristic streets".
    - Raphaellll 3y ago
      Yes, it's far from perfect. We currently work on augmenting the prompt with e.g. relevant key-value pairs extracted from https://wiki.openstreetmap.org/wiki/Map_features https://wiki.openstreetmap.org/wiki/Map_features
- e-v 3y ago
  Lucky you! I tried learning with examples generated by ChatGPT and none of them worked at all. I then learned the basics of the API by reading this excellent tutorial: https://osm-queries.ldodds.com/tutorial/ https://osm-queries.ldodds.com/tutorial/, and realised that, in hindsight, ChatGPT's answers weren't even close to being correct.
- stevage 3y ago
  Yeah, I've had that experience too. Or at least it gets close enough that I can make it work. The Overpass QL language is the most...bizarrely constructed language I've ever had to use. It's a bit hard to believe that the person who designed the language thought this was a perfectly reasonable statement: (._; >;);
  - Alifatisk 3y ago
    Gotta agree with you, the QL is strange. I tried using the wizard but it kept spinning. I tried asking Bard and ChatGPT but they provided invalid syntax. This is sad because overpass is extremely powerful tool. I’ll give the tutorials other commenters linked a try.
  - Aachen 3y ago
    I read up on the syntax a few years ago and, by learning only a few of the symbols, it actually starts to make a lot of sense and seemed like a nice shorthand. I then promptly forgot everything and find the syntax very annoying again nowadays... seriously, it's weird remembering having knowledge but not having the knowledge itself anymore
    - flexagoon 3y ago
      Overpass syntax is one of those things that I just read the documentation for every single time I have to do it. Last time I used it things like ChatGPT didn't exist yet though, so I'll probably use it from now on if I ever have to write an overpass query
      - stevage 3y ago
        Yeah me too. No way is my brain going to remember any of that between times that I use it.
  - vloewe 3y ago
    I scraped the these websites https://wiki.openstreetmap.org/wiki/Map_features https://wiki.openstreetmap.org/wiki/Map_features and gave it to OpenAI's Assistant maker -> This made great results in making valid queries Could even make prompts like "Can you find the buildings closer than 500m to a café?"
- webel0 3y ago
  It does. The problem is that I have a hard time determining if the output is, indeed, what I was trying to query in the first place!
ImaCake 3y ago
There is a QGIS plugin for this API as well which is excellent. Note that in openstreetmap.com you can zoom in and query items (cursor with question mark icon) to get the feature information that you then use in Overpass. Very useful!
- Freak_NL 3y ago
  openstreetmap.org not .com (!)
  - flexagoon 3y ago
    openstreetmap.com just redirects to the .org domain. There's also just osm.org, which is much easier to type.
curtisblaine 3y ago
The wizard isn't working for me. Just keeps spinning.
- a_gnostic 3y ago
  Spinning is a nice trick.
marklit 3y ago
I built osm_split to extract OSM features into named GPKG files. It makes it easy to pick features by name and drop them into QGIS. https://github.com/marklit/osm_split https://github.com/marklit/osm_split
- RicoElectrico 3y ago
  Isn't the point of GPKG that you can bundle multiple named layers in one file?
  - marklit 3y ago
    Yeah, you can. The file will be huge though and QGIS won't be performant. Somewhere like Tokyo will be ~400 MB in OSM's PBF format but will contain ~1,300 different types of features. Seeing these listed in alphabetical order in a file explorer makes it easier to track down only what you need to build your map. It could be you only end up using 150 MB of GPKG files and QGIS, even on an old laptop, would be performant.
vloewe 3y ago
Overpass is great! Published a ShowHN about Atlas.co yesterday, but I also just want to mention that we have a Overpass Turbo OSM integration that enables you to query data directly in a map and then download it as geojson, kml or shp (obv you can also style it and use it in analysis). Might be useful for some
lmeyerov 3y ago
Related: If any OSM GIS engineering enthusiasts, we have a small OSS repo we are looking for an assist on that should be fun for the right person: Spin up a big box, quickly render planet-scale tiles at a decent level (maptiler), and restart as a more reasonably-sized zoom level for serving. We did the core infra/devops, and are looking for someone to help tweak the perf knobs etc. See my profile on how to reach.
- mistrial9 3y ago
  please disclose if you are asking for an unpaid intern or volunteer with certain qualifications
  - lmeyerov 3y ago
    Sponsor, even!
- Doctor_Fegg 3y ago
  Do you really mean maptiler? tilemaker or planetiler will do a better job of “quickly rendering planet-scale tiles”.
the_g0d_f4ther 3y ago
Actually saved me in an internship where i needed a complete dataset of adresses. This post remindes me that i had a gdoc where i had i lot of ideas for side projects, thanks !
Tempest1981 3y ago
Mentioned here a few months ago: (was trying to remember where I saw it) How to find a street in 2 minutes [video] (youtube.com) https://news.ycombinator.com/item?id=36912880 https://news.ycombinator.com/item?id=36912880
guwop 3y ago
Hmmmm. If i want to do larger scale extractions of OSM data into a layer - what's the move? I remember someone on here posted a way to do it
- twp 3y ago
  I wrote https://github.com/twpayne/osm-extract https://github.com/twpayne/osm-extract for exactly this.
- ikawe 3y ago
  I typically download a pbf extract, and then use the osmium cli for local filtering. Then I use ogr2ogr to convert to whatever output format I need.
HumblyTossed 3y ago
Slightly off topic and maybe better as an Ask HN, but here goes. There are apps for updating OSM that will let me put in the information for the houses in my neighborhood, for example, but that gets tedious. Are there any scripts (python, shell, ?) that would make that easier?
- toomuchtodo 3y ago
  https://wiki.openstreetmap.org/wiki/Import https://wiki.openstreetmap.org/wiki/Import > Imports and automated edits should only be carried out by those with experience and understanding of the way the OpenStreetMap community creates maps, and only with careful planning and consultation with the local community. See Import/Guidelines and Automated Edits code of conduct for more information. Imports/automated edits which do not follow these guidelines might be reverted!
- Quot 3y ago
  I used StreetComplete when I had an Android phone. It looks for missing data near you, and gives you a really nice interface to input that data. They are working on an iOS build, and I can't wait to start using it again. https://github.com/streetcomplete/StreetComplete/ https://github.com/streetcomplete/StreetComplete/ Edit: Also, OSM has a few wiki pages for editing software on different platforms. - Android_apps_that_can_upload_changes_to_OSM - https://wiki.openstreetmap.org/wiki/Category:Android_apps_that_can_upload_changes_to_OSM https://wiki.openstreetmap.org/wiki/Category:Android_apps_th... - Android_apps_that_can_record_GPS_tracks - https://wiki.openstreetmap.org/wiki/Category:Android_apps_that_can_record_GPS_tracks https://wiki.openstreetmap.org/wiki/Category:Android_apps_th... - IOS_software - https://wiki.openstreetmap.org/wiki/Category:IOS_software https://wiki.openstreetmap.org/wiki/Category:IOS_software - Mobile_editors - https://wiki.openstreetmap.org/wiki/Category:Mobile_editors https://wiki.openstreetmap.org/wiki/Category:Mobile_editors
jjwiseman 3y ago
I've played around with Overpass and GPT a bit. Here are a few examples that I think demonstrate the potential: Find all buildings that straddle the boundary between Glendale and Burbank in California: https://twitter.com/lemonodor/status/1636849040548675584 https://twitter.com/lemonodor/status/1636849040548675584 Finding the suspected origin of an explosion, by using the time from seeing the explosion on video to the time the sound of the explosion reaches the microphone: https://twitter.com/lemonodor/status/1636859983223734273 https://twitter.com/lemonodor/status/1636859983223734273 Someone on twitter asked "Is anyone aware of any 4 way stop intersections in Australia?" https://twitter.com/lemonodor/status/1516239321094713346 https://twitter.com/lemonodor/status/1516239321094713346 Finding examples of airport runways that cross highways: https://twitter.com/lemonodor/status/1709598048459132976 https://twitter.com/lemonodor/status/1709598048459132976 Finding banks that might be at risk of robbery: https://twitter.com/lemonodor/status/1716153200750051332 https://twitter.com/lemonodor/status/1716153200750051332
- jjwiseman 3y ago
  Bellingcat's Bellingcat OpenStreetMap search is a tool that wraps Overpass Turbo in an easier-to-use interface for the specific task of geolocation based on image and videos: https://osm-search.bellingcat.com https://osm-search.bellingcat.com
  - jjwiseman 3y ago
    Here’s a video showing the sort of geolocation task the Bellingcat tool is intended for: https://youtu.be/GqKNKQ02pjY?si=DyAB3YZzl3gJUzT9 https://youtu.be/GqKNKQ02pjY?si=DyAB3YZzl3gJUzT9
- WaxProlix 3y ago
  This is awesome, but a lot of the queries I've tried to make seem to fail on OSM's data itself, which is sad. For instance, "coffee shops within a certain distance of fast EV chargers" would be really valuable, but the underlying data just doesn't have that EV data. So it's cool for a lot of stuff, but mostly explicitly streets; the other kinds of nodes all exist but aren't as well fleshed out as they could be (totally understandable).
westnordost 3y ago
Regarding data mining tools for OpenStreetMap, there is also a tool with which one can use SPARQL syntax. This means that one could conflate data with wikidata or other data sources easily, too: QLever: https://qlever.cs.uni-freiburg.de/osm-planet https://qlever.cs.uni-freiburg.de/osm-planet Actually, there are two. Sophox is around for longer: Sophox: https://sophox.org/ https://sophox.org/

215 points by stefankuehnel 3y ago

[dead]

For anyone else curious: Source code for the backend here: https://github.com/drolbr/Overpass-API https://github.com/drolbr/Overpass-API Found from this link listed in the help modal: https://overpass-api.de/ https://overpass-api.de/ (also links to two additional frontends.)

shpx 3y ago

I've been asking ChatGPT to write Overpass Turbo queries and it does a good job.

maelito 3y ago
Yes. It was way harder with the other LLMs. Except Claude maybe.
zarazas 3y ago
There is also this: https://overpassnl.schumann.pub/ https://overpassnl.schumann.pub/
Raphaellll 3y ago
we systematically tested this: https://arxiv.org/pdf/2308.16060.pdf https://arxiv.org/pdf/2308.16060.pdf
- maelito 3y ago
  Where can we use it ? I couldn't find a link. It is closed-source ? OK, I had a hard time understanding that the link given above is the demo of this paper. https://overpassnl.schumann.pub https://overpassnl.schumann.pub. Very interesting. Quite surprising too : it can find non trivial searches like "bars with darts" ("bars à fléchette" in French), but not easy ones ("fromagerie" or "dermato"). More complex queries did not work : "cafes close to the river" returned nothing, neither did "touristic site close to a pedestrian area" or "touristic streets".
  - Raphaellll 3y ago
    Yes, it's far from perfect. We currently work on augmenting the prompt with e.g. relevant key-value pairs extracted from https://wiki.openstreetmap.org/wiki/Map_features https://wiki.openstreetmap.org/wiki/Map_features
e-v 3y ago
Lucky you! I tried learning with examples generated by ChatGPT and none of them worked at all. I then learned the basics of the API by reading this excellent tutorial: https://osm-queries.ldodds.com/tutorial/ https://osm-queries.ldodds.com/tutorial/, and realised that, in hindsight, ChatGPT's answers weren't even close to being correct.
stevage 3y ago
Yeah, I've had that experience too. Or at least it gets close enough that I can make it work. The Overpass QL language is the most...bizarrely constructed language I've ever had to use. It's a bit hard to believe that the person who designed the language thought this was a perfectly reasonable statement: (._; >;);
- Alifatisk 3y ago
  Gotta agree with you, the QL is strange. I tried using the wizard but it kept spinning. I tried asking Bard and ChatGPT but they provided invalid syntax. This is sad because overpass is extremely powerful tool. I’ll give the tutorials other commenters linked a try.
- Aachen 3y ago
  I read up on the syntax a few years ago and, by learning only a few of the symbols, it actually starts to make a lot of sense and seemed like a nice shorthand. I then promptly forgot everything and find the syntax very annoying again nowadays... seriously, it's weird remembering having knowledge but not having the knowledge itself anymore
  - flexagoon 3y ago
    Overpass syntax is one of those things that I just read the documentation for every single time I have to do it. Last time I used it things like ChatGPT didn't exist yet though, so I'll probably use it from now on if I ever have to write an overpass query
    - stevage 3y ago
      Yeah me too. No way is my brain going to remember any of that between times that I use it.
- vloewe 3y ago
  I scraped the these websites https://wiki.openstreetmap.org/wiki/Map_features https://wiki.openstreetmap.org/wiki/Map_features and gave it to OpenAI's Assistant maker -> This made great results in making valid queries Could even make prompts like "Can you find the buildings closer than 500m to a café?"
webel0 3y ago
It does. The problem is that I have a hard time determining if the output is, indeed, what I was trying to query in the first place!

ImaCake 3y ago

There is a QGIS plugin for this API as well which is excellent. Note that in openstreetmap.com you can zoom in and query items (cursor with question mark icon) to get the feature information that you then use in Overpass. Very useful!

Freak_NL 3y ago
openstreetmap.org not .com (!)
- flexagoon 3y ago
  openstreetmap.com just redirects to the .org domain. There's also just osm.org, which is much easier to type.

curtisblaine 3y ago

The wizard isn't working for me. Just keeps spinning.

a_gnostic 3y ago
Spinning is a nice trick.

marklit 3y ago

I built osm_split to extract OSM features into named GPKG files. It makes it easy to pick features by name and drop them into QGIS. https://github.com/marklit/osm_split https://github.com/marklit/osm_split

RicoElectrico 3y ago
Isn't the point of GPKG that you can bundle multiple named layers in one file?
- marklit 3y ago
  Yeah, you can. The file will be huge though and QGIS won't be performant. Somewhere like Tokyo will be ~400 MB in OSM's PBF format but will contain ~1,300 different types of features. Seeing these listed in alphabetical order in a file explorer makes it easier to track down only what you need to build your map. It could be you only end up using 150 MB of GPKG files and QGIS, even on an old laptop, would be performant.

vloewe 3y ago

Overpass is great! Published a ShowHN about Atlas.co yesterday, but I also just want to mention that we have a Overpass Turbo OSM integration that enables you to query data directly in a map and then download it as geojson, kml or shp (obv you can also style it and use it in analysis). Might be useful for some

lmeyerov 3y ago

Related: If any OSM GIS engineering enthusiasts, we have a small OSS repo we are looking for an assist on that should be fun for the right person: Spin up a big box, quickly render planet-scale tiles at a decent level (maptiler), and restart as a more reasonably-sized zoom level for serving. We did the core infra/devops, and are looking for someone to help tweak the perf knobs etc. See my profile on how to reach.

mistrial9 3y ago
please disclose if you are asking for an unpaid intern or volunteer with certain qualifications
- lmeyerov 3y ago
  Sponsor, even!
Doctor_Fegg 3y ago
Do you really mean maptiler? tilemaker or planetiler will do a better job of “quickly rendering planet-scale tiles”.

the_g0d_f4ther 3y ago

Actually saved me in an internship where i needed a complete dataset of adresses. This post remindes me that i had a gdoc where i had i lot of ideas for side projects, thanks !

Tempest1981 3y ago

Mentioned here a few months ago: (was trying to remember where I saw it) How to find a street in 2 minutes [video] (youtube.com) https://news.ycombinator.com/item?id=36912880 https://news.ycombinator.com/item?id=36912880

guwop 3y ago

Hmmmm. If i want to do larger scale extractions of OSM data into a layer - what's the move? I remember someone on here posted a way to do it

twp 3y ago
I wrote https://github.com/twpayne/osm-extract https://github.com/twpayne/osm-extract for exactly this.
ikawe 3y ago
I typically download a pbf extract, and then use the osmium cli for local filtering. Then I use ogr2ogr to convert to whatever output format I need.

HumblyTossed 3y ago

Slightly off topic and maybe better as an Ask HN, but here goes. There are apps for updating OSM that will let me put in the information for the houses in my neighborhood, for example, but that gets tedious. Are there any scripts (python, shell, ?) that would make that easier?

toomuchtodo 3y ago
https://wiki.openstreetmap.org/wiki/Import https://wiki.openstreetmap.org/wiki/Import > Imports and automated edits should only be carried out by those with experience and understanding of the way the OpenStreetMap community creates maps, and only with careful planning and consultation with the local community. See Import/Guidelines and Automated Edits code of conduct for more information. Imports/automated edits which do not follow these guidelines might be reverted!
Quot 3y ago
I used StreetComplete when I had an Android phone. It looks for missing data near you, and gives you a really nice interface to input that data. They are working on an iOS build, and I can't wait to start using it again. https://github.com/streetcomplete/StreetComplete/ https://github.com/streetcomplete/StreetComplete/ Edit: Also, OSM has a few wiki pages for editing software on different platforms. - Android_apps_that_can_upload_changes_to_OSM - https://wiki.openstreetmap.org/wiki/Category:Android_apps_that_can_upload_changes_to_OSM https://wiki.openstreetmap.org/wiki/Category:Android_apps_th... - Android_apps_that_can_record_GPS_tracks - https://wiki.openstreetmap.org/wiki/Category:Android_apps_that_can_record_GPS_tracks https://wiki.openstreetmap.org/wiki/Category:Android_apps_th... - IOS_software - https://wiki.openstreetmap.org/wiki/Category:IOS_software https://wiki.openstreetmap.org/wiki/Category:IOS_software - Mobile_editors - https://wiki.openstreetmap.org/wiki/Category:Mobile_editors https://wiki.openstreetmap.org/wiki/Category:Mobile_editors

jjwiseman 3y ago

I've played around with Overpass and GPT a bit. Here are a few examples that I think demonstrate the potential: Find all buildings that straddle the boundary between Glendale and Burbank in California: https://twitter.com/lemonodor/status/1636849040548675584 https://twitter.com/lemonodor/status/1636849040548675584 Finding the suspected origin of an explosion, by using the time from seeing the explosion on video to the time the sound of the explosion reaches the microphone: https://twitter.com/lemonodor/status/1636859983223734273 https://twitter.com/lemonodor/status/1636859983223734273 Someone on twitter asked "Is anyone aware of any 4 way stop intersections in Australia?" https://twitter.com/lemonodor/status/1516239321094713346 https://twitter.com/lemonodor/status/1516239321094713346 Finding examples of airport runways that cross highways: https://twitter.com/lemonodor/status/1709598048459132976 https://twitter.com/lemonodor/status/1709598048459132976 Finding banks that might be at risk of robbery: https://twitter.com/lemonodor/status/1716153200750051332 https://twitter.com/lemonodor/status/1716153200750051332

jjwiseman 3y ago
Bellingcat's Bellingcat OpenStreetMap search is a tool that wraps Overpass Turbo in an easier-to-use interface for the specific task of geolocation based on image and videos: https://osm-search.bellingcat.com https://osm-search.bellingcat.com
- jjwiseman 3y ago
  Here’s a video showing the sort of geolocation task the Bellingcat tool is intended for: https://youtu.be/GqKNKQ02pjY?si=DyAB3YZzl3gJUzT9 https://youtu.be/GqKNKQ02pjY?si=DyAB3YZzl3gJUzT9
WaxProlix 3y ago
This is awesome, but a lot of the queries I've tried to make seem to fail on OSM's data itself, which is sad. For instance, "coffee shops within a certain distance of fast EV chargers" would be really valuable, but the underlying data just doesn't have that EV data. So it's cool for a lot of stuff, but mostly explicitly streets; the other kinds of nodes all exist but aren't as well fleshed out as they could be (totally understandable).

westnordost 3y ago

Regarding data mining tools for OpenStreetMap, there is also a tool with which one can use SPARQL syntax. This means that one could conflate data with wikidata or other data sources easily, too: QLever: https://qlever.cs.uni-freiburg.de/osm-planet https://qlever.cs.uni-freiburg.de/osm-planet Actually, there are two. Sophox is around for longer: Sophox: https://sophox.org/ https://sophox.org/