4 ms·
I'm surprised I hadn't seen this project before. I had been dabbling with the idea of a geocoding service that is highly accurate based on open data, but gave
by Uberphallus 5y ago
I'm surprised I hadn't seen this project before.
I had been dabbling with the idea of a geocoding service that is highly accurate based on open data, but gave up because data quality is all over the place even for countries with excellent open data access (France being the gold standard in my book). I found that to have accurate geocoding I had to have an ETL stack that pulled data from 2-5 sources per country, with country-specific code for basically every country, and that's just to store usable data. Even keeping it to the major economic powers felt too daunting, nobody wants a partial coverage geocoder and I don't want to be a one man data janitor gig, which is what the whole thing felt like. And still large swaths of countries are totally uncoded for all sorts of reasons.
A quick search shows some of the problems I found in my quest, and some new.
1. Wrong postal codes for certain addresses, e.g. Place de Gaulle, Antibes, shows 06160 but it's 06600. It's probably taken from cadastre or nearest OSM tags, but it's wrong, different parts of the town have either of the postal codes. For France, the postal code relations/polygons from OSM are accurate, just don't take postcode info from anywhere else, really, unless for some reason your point falls within France but somehow out of any postal code polygons.
2. Missing addresses. For France, I did some analysis on open data to find what street names were the most common, and found ~15k steets/squares/etc named "de l'Église", due to basically every town having at least a church and naming their street/square after them. Not sure if a problem with the data or with the parsing, but it only shows a few dozen.
3. While OpenAddresses is doing an amazing job, I only found the coordinate data to be accurate, and not always. Most of the fields in their own JSON are empty (postcode, district, region, and sometimes even city), but Gisgraphy takes them at face value, e.g. first result for Улица Деревянко is just that, it only shows street name. Also the coordinates are broken: 0.140642,34.6784995 falls either in Kenya or Algeria (coordinate swap is also rampant in many sources, which is a another topic). That is the official data from Kharkiv, Ukraine, which probably geocodes the locations in a different system than WSG84 for, you know, reasons. OA tends to focus on easy to process data, and can't blame them, since the state of the whole geocoding world is an absolute clusterfuck.
4. Related to 3, OpenAddresses only has (had?) two towns from all of Ukraine (Kharkiv and Dnipro) with really terrible data. But OSM data is also fed to their model, so data overlaps, sometimes with different scripts, or languages. So the street above, "Улица Деревянко", which is in Russian btw, appears again as "Дерев’янка вулиця" which is Ukrainian. The whole l10n/i18n part of geocoding is another tough problem to solve, because letters written to Derevyanka Vulitsa will arrive as well, transliteration data is spotty at best, but for many languages it can be transliterated through software, again, depending on many factors.
That said, I'd love to work on a project like this as part of a team.
- timita 5y agoThe guys at OpenCage Geocoder[0] are doing a great job using only open data. But they are a team with over a decade of experience parsing an deduplicating address data from dozens of countries. That said, their jobs page doesn't have much for now, but you may want to keep an eye on it. Disclaimer: the founder of the company and I are acquaintances, but my assessment is only based on the quality of their service. I've been using it in production for reverse geocoding for a few years now. [0] https://opencagedata.com/ https://opencagedata.com/
- freyfogle 5y agoHi, Ed from OpenCage here, thanks for the kinds words! It's true we don't have any open positions right now. But anyone who is into geo stuff in general and geocoding specifically can dive in to OpenStreetMap and the open source libraries we (and many others) rely on and contribute to. Most notably Nominatim https://nominatim.org https://nominatim.org Here's a podcast interview I did last summer with Sarah Hoffmann, the lead maintainer. https://thegeomob.com/podcast/episode-35 https://thegeomob.com/podcast/episode-35
- pabs3 5y agoIf you're ever hiring folks to work on open source geo stuff, please post them on FOSSjobs and the other aggregators linked from the wiki: https://www.fossjobs.net/ https://www.fossjobs.net/ https://github.com/fossjobs/fossjobs/wiki/resources https://github.com/fossjobs/fossjobs/wiki/resources
- Uberphallus 5y agoHi! Shame you're not hiring. While your API service is stellar, if not the best with open data, unfortunately the data quality is always the limit and one can only extract so much from it. While I noticed a lot of sanitization when running some queries, it didn't take a long time to find hiccups, mainly because I know the types of warts open geo data has. But from my quick tests, there are two issues. 1. Spain. Like, the whole of it. OSM Spain is lacking a lot of number information. Even Madrid (city) alone is missing a lot, and some reasonably large towns are basically unnumbered. E.g. 40.309452, -3.730451, the whole of Getafe (180k people) lacks numbering. All that information is available in the catastro, but names are often shortened, missing prepositions, lacking accents ("Calle de la Pasión" becomes "CL PASION" in the catastro) and is a horrible mess overall with no 100% proof way to cross correlate data, but here I don't see any cross correlation happening at all. 2. Searching for "Place de Gaulle", because it's a solid no-strange-characters way to obtain an endless supply of points within France, shows a mysterious result at rank 10: 47.63341, -83.04979, in the middle of nowhere, ON, Canada. No info whatsoever. Why would that rank that high, vs thousands of French counterparts? It doesn't appear in Nominatim either, nor in any of the datasets I've worked with; not sure where that comes from. Now I am curious, what's that?