6 ms·
Satellites auto detect buildings in OpenStreetMap
- bpchaps 10y agoNice! OSM data unfortunately wasn't comprehensive enough for a project of mine, but I still love what they do. It's great seeing these kinds of improvements!
- beering 10y agoWhat was your project, and what did you end up using?
- bpchaps 10y agounstructured blob: I'm working with data from a FOIA request for ~5yrs worth of parking tickets in Chicago. About 18m rows in total. The end goal is to be able to say whether a parking ticket was valid or not. It's been ongoing for about two years now, but when I have chunks of time I find myself getting back into the old code or just rewriting chunks. Part of the reason it's taking so long is that a personal requirement of mine is to treat every ticket on the same footing, so throwing rows out during cleanups etc isn't something I'm necessarily comfortable with. It's kinda frustrating but it's a hopeful goal. Since ticketers don't record lat/long and only [badly typo'd] addresses, I'm stuck doing the address to lat/long translation myself. It's not easy by any means since there are many, many many edge cases. The rate limits on common street address resolution tends to be pretty low (~10k/day) and the paid APIs are ungodly expensive. SM has been a great tool to help things go a bit forward, but when an address is missing, so is its lat/long which in many cases means rows get thrown out - which I'm not comfortable with. Interpolation using shapefile data would work, but the number of edge cases makes that difficult. I believe the data I have right now uses a combination of census data, OSM data and shapefiles from data.cityofchicago.org. Someone in the local data civics group put together a cleaned up copy of that where I was originally using the sources separately. I can pull it if you're curious. To complicate things, having missing data adds noise and false positives when correcting typos. There don't really seem to be many address sources that are consistent with address types - eg, "pl" vs "plaza" vs "place". I threw together some pretty decent python code with recursive difflib to clean typos using other typo'd addresses as a path to the correct address and it does pretty well to correct much of that. For example, in a few steps 'nrragansitt' becomes 'narragansett'.
- nl 10y agoGiven that you have a fixed area the tickets can be in, and you know every possible address, it seems like you could investigate string distance measures. It would solve your pl vs plaza vs place problem.
- bpchaps 10y agoThat's what using difflib does. Plaza vs pl vs place is just one example of many, though. There are about 1k different street names in chicago and about 18k different street names in the data. Many are missing street type and there are many street names which have different street types. So that tends to open up cans of worms. That, and street names that have 'street' in their names. Also, 'avenue' is a street name which screws things up left and right. It's not a simple problem, and none of the popular machine learning libraries which use string distance for this sort of problem have been very effective.
- maxerickson 10y agoHave you tried https://github.com/openvenues/libpostal https://github.com/openvenues/libpostal?
- bpchaps 10y agoYep - it's actually been pretty invaluable in getting this project to where it is now. :) It definitely has its own flaws, but those are easy to work around. Edit - looks like they have a similar problem from dealing with hand typed data. From their road map: Ambiguous token classification (coming soon): e.g. "dr" => "doctor" or "drive" for an English address depending on the context. Multiclass logistic regression trained on OSM addresses, where abbreviations are discouraged, giving us many examples of fully qualified addresses on which to train
- eloy 10y agoI've drawn a few houses on OSM a few years ago. Great to see projects like this that are automating this process!
- andrewljohnson 10y agoThis has been heating up. Here's the seminal paper that kicked off the Cambrian Explosion: https://www.cs.toronto.edu/~vmnih/docs/Mnih_Volodymyr_PhD_Thesis.pdf https://www.cs.toronto.edu/~vmnih/docs/Mnih_Volodymyr_PhD_Th... And I saw two presentations about this at State of the Map US. * Anand Thakker of Development Seed had a cool live demo: https://www.youtube.com/watch?v=BftllmqXSpA&list=PLqjPa29lMiE3eR-gK80irr3xdUiRbIMeg&index=2 https://www.youtube.com/watch?v=BftllmqXSpA&list=PLqjPa29lMi... * Facebook mapped a bunch of roads in Egypt, and has an ongoing effort. Facebook apparently didn't allow their's to be taped. I have a similar project called DeepOSM: https://github.com/trailbehind/deeposm https://github.com/trailbehind/deeposm There is starting to be a conversation in the OSM community about allowing robot edits into the map, I'm definitely pro robot. Google has been doing it for years. Mike Migurski touched off debate on the OSM Slack and blog comments with his post: http://mike.teczno.com/notes/openstreetmap-at-a-crossroads.html http://mike.teczno.com/notes/openstreetmap-at-a-crossroads.h... (added this link after maxerickson's comment reminded me what touched of the debate). http://terrapattern.com http://terrapattern.com is also a related project from some CMU people.
- manarth 10y ago> "There is starting to be a debate in the OSM community about allowing robot edits into the map" What do you think about a hybrid: bot detects and recommends changes, which are then queued or tagged for human verification? I've been looking at scuba data in the OSM dataset, and there are a number of bot-added entries, with a tag "verified": "no, please check and del this tag" (e.g. Maaya Thila dive site [1]). [1] http://www.openstreetmap.org/node/663869880 http://www.openstreetmap.org/node/663869880
- andrewljohnson 10y agoThat's the status quo. I'm in favor of a regime where there is a benchmark for the recall and precision rates for the automated analysis, and anything that hits the benchmarks is acceptable. I don't regard this as a philosophical matter - let's accept any bot analysis that hits a certain threshold. 1) We can certainly set the threshold above the TIGER import in the US, which most people agree is positive, but very crufty. 2) It's the developing parts of the world that need automated road creation the most. The opposition seems to emanate from rich, European nations.
- deleted 10y ago[deleted]
- maxerickson 10y agoThis space is heating up a bit. Facebook (!) has developed a system for mapping roads and is working towards using it to add data to OpenStreetMap: http://forum.openstreetmap.org/viewtopic.php?id=55220 http://forum.openstreetmap.org/viewtopic.php?id=55220 https://news.ycombinator.com/item?id=12150862 https://news.ycombinator.com/item?id=12150862 (no comments) There's several other people/groups working on similar projects, like DeepOSM: http://www.deeposm.org/ http://www.deeposm.org/ https://news.ycombinator.com/item?id=11808639 https://news.ycombinator.com/item?id=11808639 (some discussion) And no shortage of feels about how all the parts need to work together: http://mike.teczno.com/notes/openstreetmap-at-a-crossroads.html http://mike.teczno.com/notes/openstreetmap-at-a-crossroads.h... https://news.ycombinator.com/item?id=12187782 https://news.ycombinator.com/item?id=12187782 (no comments) http://tomlee.wtf/2016/07/31/introductory-openstreetmap-politics/ http://tomlee.wtf/2016/07/31/introductory-openstreetmap-poli... http://www.maproomblog.com/2016/08/openstreetmap-at-the-crossroads/ http://www.maproomblog.com/2016/08/openstreetmap-at-the-cros... There have been earlier efforts to do automated feature extraction, but it seems that improvements in AI and better, more liberally licensed imagery are moving things towards some sort of tipping point.
- MRPockets 10y agoAs an infrequent but repeat contributor to OSM, this is fantastic! While there is an historic bias against automated edits (sometimes for good reasons but I do often wonder...), automating the tedious parts of mapping is, imho, a very good thing. That said, I don't see much detail here regarding what happens when a missing building is detected. Does it get automatically added to the map? Or maybe a note is made indicating a building needs to be added? Maybe the building is auto-added but with a FixMe note for manual mappers to examine it for accuracy?
- joshvm 10y agoAutomated building detection isn't exactly novel in remote sensing. People have been doing it for years so they can measure how fast cities develop, monitor inhabited areas after natural disasters and so on. This kind of work is never 100% correct. It usually relies on multi-band imagery so you can figure out what's vegetation and what's roof tile. As far as I know Google does something similar to automate mapping of roads, and then they have a huge team of people cleaning up afterwards. The best solution here (IMO) would be to add a list of potential buildings that should be checked by people either by cross-referencing satellite imagery or by people who actually live in the area.
- treigerm 10y agoHere's a demo of the project: http://ryfeus.github.io/urban_osm_sentinel/demo/ http://ryfeus.github.io/urban_osm_sentinel/demo/ Also does anyone know if it's possible to get the code for this? Haven't found a github or bitbucket repo for it yet.
- lucb1e 10y ago> The green buildings are existing structures captured in OpenStreetMap. The red buildings are the ones we detected as missing from the map. And which color is existing structures not identified by the software? Because that would be a very quick and easy way to spot how well it's actually performing.
- autokad 10y agoor which red buildings aren't buildings. It'd be nice to verify on a city such as Philadelphia that has an open parcel layer, but then again cities with parcel layers are probably well mapped out in OSM
- Freak_NL 10y agoYeah, usually there are scripts that can update and import building outlines from open government data. Usually this is done with human guidance and by request. So if a mapper in a certain area notices missing or outdated building, he or she might request that someone in the community familiar with those import tools import the outlines for that area. Ideally the local mapper can then verify and enrich this data. In a lot of developed countries almost any building barring small garden sheds is meticulously described in government parcel databases with decimetre precision. In the past few years these outlines have been imported in the Netherlands in OpenStreetMap; it really looks a lot better than just a bunch of rectangles.
- protomikron 10y agoThis is an interesting area and the most important recent change is the opening of the data. More and more municipalities provide web services where you can download high-quality imagery, e.g. have a look at https://www.wien.gv.at/ma41datenviewer/public/ https://www.wien.gv.at/ma41datenviewer/public/ in the left menu you can select "Luftaufnahmen" (aerial imagery) or DOM (Digitales Oberflaechenmodell == Digital Surface Model) and the resolution is quite high (unfortunately the imagery is watermark-raped, but most training methods are robust against that kind of errors). One might think aerial imagery is not a problem, as we have Google Maps, Bing, etc., but you reach there (free) API limits quite fast, if the goal is to get bulk data to train classifiers. In particular laser data is a game changer and makes building footprint detection much simpler (you can make pretty training masks from the provided vector layers and train e.g. a NN for segmentation). Current state-of-the art in building and street detection from photos only is around 85% to 95%, which is quite high, but a bit too low for fully automatic usage in e.g. urban development or environmental engineering. In particular there are images where it is nearly impossible to distinguish the roof from the nearby place, if the roof is not enclosed by a green area or a garden. At the moment classifications/segmentations are polished by humans (and providing accurate (satellite or aerial) imagery classification is a big business), but DSM data is so good, that it makes fully automatic approaches feasible and I would say one could reach ~99% there.
- sp8962 10y agoThe fallacy in the concept we're discussing here is assuming that OpenStreetMap is being held back by the speed with which we can add infrastructure geometry (I'm specifically excluding mass doodling of buildings as that is a topic on its own, and, hey, google does just fine without buildings nearly everywhere). Given reasonable quality aerial imagery (on which automatic detection a la Facebook depends too) the limiting factor for years in OpenStreetMap has been meta data collection (names, road attributes, POIs, addresses and so on). Humans are quite fast and good at tracing roads and the couple of minutes it takes to add a few dozens of streets is just inconsequential compared to the time and effort it takes to go there and actually survey and add the meta data. Now despite some bloggers, that themselves haven't noticed that OpenStreetMap is very different than in 2007 (but you know xkcd 386), claiming that OSM is standing still, the OpenstreetMap community is quite welcoming to new and better ways of surveying and obtaining data. Mapillary and now OSV have made lots of aspects of meta data collection easier and faster and have found wide spread adoption in OSM including automatic sign detection and so on. But even with such advances, gathering and entering the meta data is still by the order of magnitudes the dominating time consuming process in OpenStreetMap (and for what it is worth in any mapping activity).
- legulere 10y agoTracing buildings by hand is quite time-intensive even with stuff like the buildings plugin for JOSM. And it is a prerequisite for going out and mapping addresses. In cities metadata acquisition might be a much larger thing, but it isn't here on the countryside where I live. Tracing landuse form aerial imagery is probably 90% of what I do here.
- sp8962 10y agoHaving buildings is definitely not a prerequisite for mapping addresses (when I go mapping I will typically and them before too, but again that is only a minor amount of time compared to actually gathering the data). Mass mapping buildings is a rather recent phenomenon in OpenStreetMap and you can have a high quality, fully functional map with not a single building outline (as google shows too). Not to mention that we have had a country with full address coverage since ages that only has a small number of buildings mapped (DK). Now there are certain high detail tasks for which having building outlines make sense: for example adding entrances with addresses, but that is fairly far along on the path of iteratively adding more detail to OSM.
- plq 10y agoDoesn't this mean the OSM repository will be used as a cache of the output of someone's version of some ml algorithm? Why not distribute the data as well as the tagger and let the users tag the data themselves? This will give more freedom to the user and arguably lead to a better quality data (e.g. some additional tuning could give better results than the general case for a given region, which can be best performed by a team more familiar with that particual area) as way more eyeballs are now on the tagger itself, not just its output.
- rmc 10y agoIn places with an active OSM community, like Europe, the community usually maps the new buildings faster than the satillite imagery is updated.