26 ms·

OpenStreetMap Is in Trouble

1062 points by emacsen 9y ago

MonkeyDan 9y ago
I wouldn't mind seeing a two-tier approach, with authoritative municipal data as a base (see https://openaddresses.io/ https://openaddresses.io/ for an example of collecting a large number of sources and normalizing it) with user-submitted edits on top.
IntronExon 9y ago
Re: the issues with moderation, I sometimes wonder if Wikipedia were trying to start from scratch in the last few years... would it work? With all of the trolls and bots, easy access it might have died hard and early. One of the most significant technical problems with OSM is the lack of a review model, that is for a change to the map to be staged and then reviewed before being applied. Not having this functionality caused ripples of problems throughout the system, some of which I'll discuss here. Moderation is expensive, and Wikipedia got by initially with a pretty minimal degree of it, then ramped up as they got bigger. What if they’d been slammed by modern trolling at the start?
- dalbasal 9y ago
  Well... Nothing similar to Wikipedia has really happened in the last ten years. It's a fairly unique thing, still as relevant and important as ever. Hard to say if it's because Wikipedia was a long shot at any time, or because of the 90s web vs the current one, culturally or technically.
  - twblalock 9y ago
    I think its because anything similar just gets implemented as part of Wikipedia/Wikimedia at this point.
    - freneticfox 9y ago
      Just to meta-meta this conversation and the article in new and confusing ways, Wikimedia also hosts a map server, using OSM's data, @ https://maps.wikimedia.org/ https://maps.wikimedia.org/ :)
    - dalbasal 9y ago
      True. I sort of meant "similar"in a wider sense: large scale, community effort to do something useful. Wikipedia represents a lot about what we that the internet, and internet culture was.^ Remember how big a deal Wikipedia is. A lot of that "human knowledge at your fingertips" is Wikipedia. Maybe that applies at a bigger level, with "similar"things being implemented as part of something. I suspect that a big part of the reason Wikipedias don't get started now is that they get implemented as startups. Anything showing promise on that scale is a "unicorn" and in 2018 Jimmy Wales would have had a choice between private/startup and internet-owned. That's a billion dollar choice. For example, stack overflow could have been built like a Wikipedia. If it was done in 1996, that would be a sensible choice. In 2018, a founder would be forgoing an unthinkable sum of money, and also resources to make it happen with. ^I guess all this needs to be caveated with "other than OSS."
- emacsen 9y ago
  It's an interesting question of how much flexibility you give people at the beginning vs as a project matures, but I'm looking at the situation with OSM pragmatically, that is to say "What OSM needs to do now", rather than "What they should have done". The problem of both bad edits and vandalism are very large (much like Google Mapmaker) and other measures we've put in place, such as changeset comments (which I helped develop) and vandalism tools don't seem to cut it.
  - RosanaAnaDana 9y ago
    What about some kind of automated approach to validation? A major difference between OSM data and something like wikipedia is that, unlike wikipedia, at least some of the OSM data verifiable. A street somewhere, does exist, in a particular spot. There must be some way to croudsource validation.
    - spectre256 9y ago
      Ultimately it comes down to the resources to create such a thing, right? There have been some successes with crowdsourced validations such as Facebook's translation efforts (although I think in that case it was merely a component of the process). And it would rely on a volume of edits that OSM may not have. There may be 1000 people ready to verify the location of the Statue of Liberty, but probably not the location of every building in a remote area being mapped for HOT. Mapbox has built the start of a validation framework, and they have done very cool stuff. Of course then they get criticized for "forking" OSM. There are definitely valid criticisms there, but it seems like there's no way to win.
  - IntronExon 9y ago
    I didn’t mean to distract from the main point of your piece (which was a good read btw), it just made me curious in general. Maybe the only way OSM can work is by treating moderation/curation as the problem to solve?
- swang 9y ago
  that kinda overestimates how long it took wikipedia to get to where it is today. when wikipedia launched, no one knew what it would grow into. why would large amounts of trolls bother harassing wikipedia when in all likelihood it would have just gone nowhere.
- rspeer 9y ago
  It appears to me that there was a golden age of crowdsourcing that lasted from maybe 1999 to 2010. If you started a project in this time, you had the chance to get lots of quality contributions, for free, from people whose motivation is just to be part of something. Now, you will get: - Modern trolling, as you point out. Well-organized armies of trolls, some working for political purposes, some just for the lulz, can subvert anything. - Disruptive bots (I don't know why this is a change since the golden age, but maybe it's simply that more people are better at programming) - Spam that's a full-time job to moderate - Lack of motivation because there are so many other fascinating things on the Internet - A change in expectations because money has entered the equation -- crowdsourcing is now something that people do as a menial job, and they aren't paid enough to do it well The sabotage by 4chan of the crowdsourcing team in the Pentagon shredder challenge [1; sorry about the stupid title] makes a good indicator of the end of the crowdsourcing era. That happened in 2011. I run a project, ConceptNet [2], that has obtained a lot of its data from crowdsourcing. Years ago, I stopped supporting the ability for the public to edit ConceptNet directly. (This does not mean the project shut down. Far from it. It currently produces state-of-the-art NLP models.) Supporting public contributions became obviously not worth the effort. Some projects that started at the right time and maintained excellent communities are still working, though -- Wiktionary would be an example, and it's one of the places that still provides new data to ConceptNet. [1] https://www.wired.com/2015/02/how-a-lone-hacker-shredded-the-myth-of-crowdsourcing/ https://www.wired.com/2015/02/how-a-lone-hacker-shredded-the... [2] http://conceptnet.io http://conceptnet.io
RosanaAnaDana 9y ago
I've found that in terms of supporting any kind of additional analysis, OSM data to be unreliable to the point of uselessness, which isn't to say there is a better alternative. I think this quote from your article: "One of the most significant technical problems with OSM is the lack of a review model, that is for a change to the map to be staged and then reviewed before being applied. Not having this functionality caused ripples of problems throughout the system, some of which I'll discuss here." is the primary reason I can't regularly use OSM data. Trust me when I say there are millions of instance where having reliable data regarding streets could save me time/ stress. Do you have any thoughts on how to improve the quality of OSMaps? Conceptually, I feel like the product is indispensable. In practice, I feel like the product is utterly unusable.
- emacsen 9y ago
  This was a hard post to make. OSM was a big part of my life for many years, and I have friends in the project. I like my friends (even the ones I disagree with). But where OSM has failed most in my view (and maybe I need to add this to the article) is that there are now free geographic datasets from local governments and without the ability to integrate them into OSM easily, and then update them (which we basically can't do), I fear OSM is going to ultimately fail at its mission
  - toomuchtodo 9y ago
    Is this a technology issue? Or a funding issue?
    - croshan 9y ago
      Seems more like a political issue
    - maxerickson 9y ago
      It's frequently a licensing issue and then also a resources issue. There's relatively generalized tools for diffing data against OSM and outputting a change file, but lots of data is released using ambiguous or slightly incompatible licensing. One example is http://blog.improve-osm.org/en/2018/01/new-features-and-enhancements-in-cygnus/ http://blog.improve-osm.org/en/2018/01/new-features-and-enha...
amygdyl 9y ago
Is there a real hurdle in patents on map graphics held by the major mapping companies and national organisations? Could a goal of being truly open and complete, be incompatible with the interests of privacy, national security even? If community involvement is lacking, I immediately thought about the cimnunies who have real interest in mapping and particularly free open mapping, such as conservationists, residents of small towns concerned about planning and in my home Sussex county, hikers always fighting losing out to farming interests over ancient byways. But these constituencies need tools. I would immediately donate to such efforts if I knew they existed. Or a corporate like ACDSystems, who publish Deneba Canvas which while sadly lost adrift far from the software I remember, incorporated superb GIS and technical drawing capabilities. This would be the kind of marketing that traditionally worked very well, tools and support for local news stories, which use to run a long while anyway.
acabal 9y ago
I remember just a few years ago, I got a new phone and was interested in replacing as many Google apps with OSS maps that I could. (K9 instead of Google's mail app, etc.) Naturally I turned to OSM instead of Google Maps. But I was really surprised by how strange and difficult it was to install and use. As the author mentioned, OSM itself doesn't have any apps, which is very confusing. After some research I figured out that you have to download a 3rd party app. The 3rd party apps that do exist all use different terminologies, require downloading huge amounts of data and picking what maps you want ahead of time, and had different levels of functionality. The UIs were universally terrible (at the time at least). Needless to say I quickly gave up on using OSM and turned back to Google Maps--and I'm their perfect use case, a skilled techie with a deep interest in using free software! I can't imagine trying to explain how to use OSM to my mom, even if I somehow managed to install it for her. I can't comment on the rest of the article but its usability worries are on point.
- leokennis 9y ago
  Exactly. If the alternative is Google Maps (or even Apple Maps), it’s not good enough to make me pay $5 to download a map of 1 country and forcing me to keep 500mb of data on my device permanently.
- striking 9y ago
  Maps.me is pretty killer, though. The only issue I see with OSM is lack of traffic data. Generally when I drive with Google Maps, I already know the route but want Google to step in when there's traffic.
  - jedberg 9y ago
    > Generally when I drive with Google Maps, I already know the route but want Google to step in when there's traffic. You should try Waze. I too mostly use Google to route me around traffic, but Waze is an order of magnitude better at it. It has more frequently updated traffic data and is willing to take you on side streets. For example the other day I was driving down a major road, and Google said to just stay on it, but Waze took me down a side street for two blocks and then back to the main road, saving me four minutes, which was 25% of the total trip time. Also, when making a long drive, Google chooses a route that is optimal right now for the entire route. Waze accounts for the fact that it will be an hour later when you get to the second half of your trip, and uses that to make better choices.
duozerk 9y ago
OSM gives people the tools to create their own map rather than offering them a simple, out of the box solution On this point and the main part of the first argument, I kinda disagree; the OSM project should specifically aim to do that: providing means for one to create a service. However, providing an actual map service would cost much, much more in bandwidth and I doubt a free software project like this has that kind of funds - actually serving tiles can add up to large costs pretty fast if enough people use the service. So if you want an actual service, you either do the hosting yourself or go see mapbox/whatever.
- Avamander 9y ago
  Time to move away from awful raster maps and to vector maps?
  - duozerk 9y ago
    That would definitely help a lot (and osmdroid at least does support vector map sources); I imagine the traffic would still be significant though.
    - Avamander 9y ago
      Locus (not FOSS) supports vector maps and it's price is way cheaper than an external 1TB drive would cost, just for good map coverage of a few countries.
  - sydd 9y ago
    There are some, e.g. OSMAnd on Android, they are awfully slow. I wonder why no one has written a proper, fast Opengl renderer for osm
    - lucb1e 9y ago
      > [like] OSMAnd on Android, they are awfully slow On my Galaxy Note 2 (2012), you're right. I've ordered a new phone yesterday because it's getting quite bad after five years of daily use. My girlfriend bought a phone for €300, and the one I ordered (now, a year later) for €250 actually has the same chipset. It runs great on that. I consider ~250eur to be the bottom of the smartphone market unless you're going for a potato to call and text with (not even browse the web). Given that it runs smooth on that... I'm not sure what you're running OsmAnd on with expectations of fast rendering.
andrewljohnson 9y ago
This post seems half right, half wrong. Wrong that OSM should be a service provider and not a database - I think it should be a database, and corporations are good for OSM. The post calls for OSM to essentially become Mapbox. It's insanely expensive to be Mapbox, and Mapzen just failed trying to be a less corporate version of Mapbox. And then the usage policy part of the article is moot or at least not that consequential if you don't believe OSM should be a service provider. Right: Moderation tools need to be created/fixed. New mappers need to be encouraged/onboarded better. Vandalism is a problem. OSMF culture is toxic. Hidden gatekeepers are toxic. Imports need to be supported. Bots need to be enabled, and we need to consider people who contribute code/tech/imports first class OSM contributors and not just glorify the individual mapper. Probably right: OSM needs layers, or even a more sweeping re-architecture to allow better versioning, moderation, better tooling, and easier understanding of the data. Probably wrong: Expanding the scope of OpenStreetMap to include transient data. Too much other work to do, maybe later for this.
- emacsen 9y ago
  > The post calls for OSM to essentially become Mapbox The problem is that most companies using OSM are using Mapbox. This leads to a few problems, including attribution, which I decided not to touch on, but is a serious problem that Mapbox doesn't address sufficiently. If they could use OSM directly, they'd be engaged with OSM directly. They'd care about OSM. Right now OSM shoves these users away and makes them into customers of some third party (maybe Mapbox, maybe someone else) instead of caring about OSM itself. > Expanding the scope of OpenStreetMap to include transient data. Too much other work to do, maybe later for this. How about letting users experiment and seeing what cool stuff they can come up with?
  - andrewljohnson 9y ago
    > The problem is that most companies using OSM are using Mapbox. Is that even true? And is number of companies even a good metric? Based on revenue/market cap, the bulk of companies use OSM directly. The world's biggest company (Apple) uses OSM directly. Telenav uses it directly. Don't Tesla and Uber use OSM directly too? And we use Mapbox at Gaia GPS, but we also use OSM data directly. > How about letting users experiment and seeing what cool stuff they can come up with? In my experience building a company, if you focus on too many things, you get everything wrong. This smells to me of being not on target for making OSM as strong as possible, even though it has alluring upside.
tobltobs 9y ago
I can understand the frustration of OSM users that they have to create their own maps, but providing free tile servers would never be possible with the restricted funding for OSM.
jimmyrocks 9y ago
Thanks for posting this here! I feel like this is mostly good constructive criticism. I have given a few talks about using OSM data in the US government and it comes down to authoritative data sources and validation. It's a great starting point, but there needs to be another layer (or two) of validation before it can be used at the enterprise level. The National Map Corps has some good examples of how this can be implemented with data stewardship. I am also working on a few projects to take public domain data and put it into OSM. While it's useful and mostly open, it's definitely less open than the original data sources. Another issues is that as soon as someone edits the imported data, it will become less authoritative as well.
- emacsen 9y ago
  You're welcome this Jimmy. And you rock. You're one of the people who got me into OSM in the first place. And I still love OSM. I want to see it succeed! Go Igles!
  - spectre256 9y ago
    As soon as I started reading your article I actually started thinking of Jimmy and how he's been a part of a real government entity (the national park service) sending data directly into OSM.
- lucb1e 9y ago
  > it will become less authoritative Or better than the authoritative original. Like with Wikipedia, where you often hear that people prefer to cite a single person as source (because "anyone could edit Wikipedia", as if no single person can setup a website or write a paper), at least Wikipedia has had thousands of reads and often dozens or hundreds of reviews. People are scared of the idea "it could be anyone", while the most likely editor is just a peer with good intentions.
  - jimmyrocks 9y ago
    In many (most?) cases it can be better than the authoritative sources. But there is also the possibility for vandalism or sloppy work. There are established ways to combat this (peer review, data stewards, reputation scoring) and even OSM has a checkbox for further validation. OSM will never fit every use case, but it's good to understand where it falls short and can improve.
jedberg 9y ago
Open maps used to be the job of the USGS. They stopped doing it because of Google. I was touring the USGS in Palo Alto about 10 years ago, and the librarian there was lamenting the existence of Google. He said that while it was nice that Google was doing all this work, it was causing the USGS to stop doing it, since they could get the data from Google. He predicted that the USGS would get severely cut in their funding for mapping activities because of it, and he was right. He was worried that all the best data would be locked up in Google's servers, and he was right about that too.
- jimmyrocks 9y ago
  I used to work in the building with the USGS map store. It was very sad to see it close. It wasn't commercial map providers that caused it to close, it was the avilablilty of USGS topo maps online, and a faster data collection to map pipeline. You can still order maps printed on demand directly from the USGS.
- maxerickson 9y ago
  They weren't ever really in the business of publishing data. (A challenge: download a routable road network for the US from a USGS source)
reaperducer 9y ago
He's not wrong. OSM is too complex for the majority of businesses. I know two that chose to pay Google for their mapping needs rather than jump through OSM's hoops. Imagine if the only way to get Linux on your computer was to download and compile the whole thing from source yourself, instead of buying or downloading a distro. If OSM had a mapping solution like Google's, it would thrive. And just for fun, it should undercut Google's pricing.
- spectre256 9y ago
  In that sense, perhaps the linux distro model actually makes more sense than centralizing power into one entity? Ubuntu, Redhat, and many others produce their own polished, supported products, sell it, and then give back to the central parts of the community in the process. Mapbox and others certainly seem like they essentially play the part of "OSM Distro" in that regard. And maybe its okay for multiple entities to be able to contribute in different ways.
- Doctor_Fegg 9y ago
  > If OSM had a mapping solution like Google's, it would thrive. It does, it’s called Mapbox. Plus a number of excellent smaller providers - Thunderforest, Geofabrik, Graphhopper etc.
  - beering 9y ago
    And kind of like Linux, you have a patchwork of unsatisfying solutions. Mapbox isn't always cheaper than Google. They have not lowered their tile prices in years, for example. And on geocoding and some other APIs they give out poor results compared to Google. Do they have competition? Cloudmade went in some car-related direction. Mapzen never managed to have a real business plan. And your "excellent small providers"? Yeah, Andy may be able to create some bespoke solution for you if he has time, and Geofabrik can wrangle some host-it-yourself setup onto AWS, but it's still very true that OSM themselves don't have a good story on a lot of mapping needs. You can either browse their web map with unusable search or download the 38GB database, and that's about it. No wonder many businesses just pay Google rather than dive into the rabbit hole of OSM.
aspirin 9y ago
Im my opinion the biggest problem is the data model (or lack of). Tags i.e. key-value pairs are used to add properties to geometry. Mappers use different combinations of keys/values to describe the same thing. The post touches on this, but does not offer solutions other than adding "layers". There should be a strict schema that would be community managed through some process.
- maxerickson 9y ago
  A strict schema won't help when people still use it incorrectly. Improvement over what exists now would require that the schema be applied strictly and that isn't going to happen with a huge crowdsourced project.
- lucb1e 9y ago
  Like what, tag proposals where people vote yay or nay? We have those. As an active mapper, I very rarely find two ways of tagging the same thing, so the process seems to work fine. And if you do find a situation where there are multiple options, it takes only a quick check on taginfo to find the more commonly used one. If you go with that, it should converge soon.
  - aspirin 9y ago
    Currently there is nothing to stop mappers using tags in wrong ways. It should just be flat out impossible. If a field/tag isn't defined in the schema, it could not be used. The range of values should be limited in the same way, too.
    - maxerickson 9y ago
      How does a schema stop someone from marking a building as a pond?
    - lucb1e 9y ago
      Ah, I see where you're going with this. It would slow innovation down to a trickle, though. Anyone who wants to do something that's not yet standardized would have to stash their edits somewhere and wait for official approval. And that would likely take weeks or months at best. Currently, the best idea wins automatically because good ideas are used by both mapping applications and mappers. Maybe something in the middle, like the current wiki system but with an enforcement rule of whatever is on the wiki. Anyone could freely edit it because it's a wiki (adding allowed options, and removing/changing any that have extremely few uses (zero, or perhaps a few which are then removed)), but at least there is a correct spec. If people really had to adhere to a strict, slow-moving standard with pre-approval, the spec would also become extremely complex and very difficult to implement, because the spec-makers will want to think of every eventuality. See any other large specification ever, like HTML or something. I can't see that ending well. I think a hybrid system might be better than a strict one, given the choice between those two. I like thinking about it though! Always good to have hear ideas.
RcouF1uZ4gsC 9y ago
Even Apple the most valuable company in the world with hundreds of billions of dollars of cash on hand is way behind Google in terms of mapping. https://www.justinobeirne.com/google-maps-moat/ https://www.justinobeirne.com/google-maps-moat/ If Apple has trouble keeping feature parity with Google Maps, there is no way that an endeavor like Open Street Maps has any real chance of making something with the breadth, features, and usability of Google Maps.
- reaperducer 9y ago
  > there is no way that an endeavor like Open Street Maps has any real chance of making something with the breadth, features, and usability of Google Maps But does it need to? Isn't there a market for basic mapping, without all the needless feature bloat? I'd rather a mapping service concentrate on getting roads right, and not whether there is a Howard Johnson's nearby.
  - zbentley 9y ago
    Most people are using those roads to get to somewhere, be it a HoJo or a street address. So the geotagging ability is very, very important. I suspect a map system with vastly superior routing information (just roads and coordinates) and slightly inferior geotagging would still lose to one with the opposite balance.
- spectre256 9y ago
  True, at least for the moment, although that doesn't mean we shouldn't try. One nice thing about open data is it can outlast. What do you think will survive and be readily accessible on the internet longer, Wikipedia or Quora? My money's on Wikipedia 100%. With OSM things are similar. Obviously geo data has additional ways it can grow stale, but still. Google is big today but there's no guarantee for the future.
- detaro 9y ago
  Goes both ways: Openstreetmap is better for some things already, and I don't see Google catching up to it at them. A big issue is consistency though: OpenStreetMap varies way more in quality between regions, to the point that detailed conversations almost make no sense without disclaimers about where you are talking. OpenStreetMap is a really good map where a strong community exists (either locally or dedicated remote interest) and a pretty bad map if no large effort has touched the area.
pornel 9y ago
I've tried to build an open check-in app based on OSM data, and I've ran into many of the problems mentioned in the article: • Tags and relations are a total mess. There are many different and overlapping schemes. There is obviously outdated/sub-standard data, but automated fixes are forbidden. There are some neat tools to make sense of the tag data, but ultimately I had to deal with all that mess and complexity on my end. • The lack of moderation or a way to officially act as an intermediary for edits (other than clunky oAuth flow) meant I couldn't expose an Add/Edit/Fix functionality for my users. Similarly, if I wanted to let my users upload a photo of a venue, there was no service to store it, no identifier or tag to associate it. • The API returns only nodes entirely within an area, rather than everything intersecting a given area. If you fetch small lat/long area, you won't get a node for the country/city/district it's in, unless you stumble upon a point in its border or label. That meant I couldn't fetch things from OSM API as needed, and had to have my own copy of the data and the DB and custom search. • The map looks ugly without buildings. There was free building data available for areas I was interested in, but I was told to copy them by hand.
- maxerickson 9y ago
  Were you doing your own map rendering? In that case you wouldn't have to add the buildings to OSM to show them, just merge the data during your map preparation. Nominatim does have a reverse geocode api which returns the sort of place information you want in your third point (given a coordinate, returns an address).
- ris 9y ago
  > There is obviously outdated/sub-standard data, but automated fixes are forbidden. Please consider arms race encouraging automated edits would begin. A lot of people have a lot of awfully different views about how map data should be recorded (you thought source code formatting was contentious?) > I couldn't expose an Add/Edit/Fix functionality for my users. Lots of tools have managed this. > if I wanted to let my users upload a photo of a venue, there was no service to store it No. That's not what OpenStreetMap is. They aren't there to give out free image storage. > The API returns only nodes entirely within an area Yes, it's meant to be a low-level API which makes few assumptions about the meaning of any of the data. For the ability to make more complex queries you could have looked at Overpass. And possibly there is a space for a third party service to provide an API with such an ability. Again, I don't see that as OSM's business - OSM here should be considered more like the UN. > There was free building data available for areas I was interested in, but I was told to copy them by hand. And yet NYC have managed somehow to get most city building data into OSM, so perhaps you haven't investigated every possible option for getting what you want here...?
gpsx 9y ago
Previously I really like the flexibility of the tagging system in open street map. I used something like it for a unrelated commercial product and we have found we have to switch away from it. And we had very tight control over the definitions of the tagging we were using. The tools were difficult, as is mentioned in the main post. That is certainly an inconvenience but I figured we could muddle through that. The bigger problem is that the customers just never got it. They wanted a single, authoritative classification for an object. Our plan is to move to a system inspired by the model used in BIM for classification. The princicple difference is that in open street map all data identifying an object has to lie within the tags. In the alternate system, a single identifier for the object type is used, and this references external metadata that gives you additional information which does not have to be embedded in the tags on the object. An example of additional information held external to the map and tags is that a bathroom is a type of room. I don't have to include in the map data that the object is both a bathroom and a room. I should add, the external meta data also gives a strict definition of what types of tags (and maybe values) are allowed on the given type of object. If you need something a little different, you can make a new type. I am not sure how well this type of system could work in open street map. Is it too rigid? Or would there be too much proliferation of types? Classification is always hard and I am not sure people have solved that problem in any large context. And of course, changes like this would mean a new database. If people ever do think about making an OSM 2.0 maybe this, or another tagging alternative, is something to consider. (edit: typos)
- pbowyer 9y ago
  I like the sound of this, like an object graph of data from OSM. > I am not sure how well this type of system could work in open street map. Is it too rigid? Or would there be too much proliferation of types? Classification is always hard and I am not sure people have solved that problem in any large context. Ontologies are well-researched. Unfortunately they're not my area of expertise as I turned away from doing a PhD in them and went into business. I have been playing with a controlled vocabulary since Christmas for a personal project and the complexity of definition has made me think this is why Google et al have gone for a more flexible, machine-learned typing, and why the types for schema.org are so broad. Having been negative, I think this is exactly what OSM needs to organise the world's objects.
just_testing 9y ago
I'm also interested in the development of OSM (being an editor and a data consumer) and I've witnessed first hand some of your issues. For instance, once I've had to create my own "geocoder" for transit fines in São Paulo using OSM data, because no geocoder commercially available would be able to deal with the data. So, now that we clarified I'm not a troll, nor a bot, I ask you: how should we deal with those issues? Join en masse the OSMF so we can vote for a change? Begin the "New-OSM-But-Better Foundation"? I also think having layers (particularly, layers vouched by governments) would be awesome. I saw several governmental branches in Brazil wanting to be more integrated with OSM - this would allow them to stay "inside" OSM (and being an accredited citizen), instead of creating their own silos, importing data from time to time.
- emacsen 9y ago
  Honestly, I have no idea.
SteveCoast 9y ago
I'm the DRI for most of these problems. Random thoughts: I think David Ogilvy said something like "Search all the parks in your city, you'll find no statues of committees". Everyone was super keen to make OSM in to a web of committees, and this is one root of the problem from an organizational point of view. It's not super clear to me what the OSMF has done in the last (to pick a number) 5 years. (I don't regard "doing the same thing as last year" as relevant to anything, I just have a personal bias towards new things). From a technical point of view we merged sysadmin (who inherently want stability) with development (who inherently want change) and stability won (it doesn't involve arguments or as much work, I guess). That's why the API looks the same way I designed it X number of years ago. It's still a wonderful project, but more of a work of art than anything else. It's not like wikipedia didn't have all the same problems. Making a project that really evolves and grows over large periods of time is really hard. Like - most of the things on the S&P 500 won't be there in 10 or 20 years or something. I think what really happens is that projects go through a lifecycle and something new comes along to replace them, rather than anything getting fixed. I'm not sure if this is a good example, but, you could try to "fix" Apache, or you could just do this new thing, "nginx". For all kinds of reasons, the new thing is more efficient. One of the few counter examples I think is going to be Amazon. The world has moved on from making vector maps and OSM solved almost all the related problems (just not geocoding, which annoys me no end). It moved on, because of OSM! All the money and interest in maps is now around autonomous navigation, which means 3D maps and other things outside of OSMs area.
- lucb1e 9y ago
  > I'm the DRI All I can find is "Dietary Reference Intake" on https://www.merriam-webster.com/medical/DRI https://www.merriam-webster.com/medical/DRI -- what do you mean? (I agree with most of your post by the way (except for the bit about it being art, which would mean it has no function except to be art), I'm just unclear on what you meant here.)
  - SteveCoast 9y ago
    Directly Responsible Individual https://en.wikipedia.org/wiki/Steve_Coast https://en.wikipedia.org/wiki/Steve_Coast
WhitneyLand 9y ago
contact email below
- emacsen 9y ago
  You can email me at emacsen at emacsen dot net
btilly 9y ago
Here is my experience. I work for a company that syndicates location data for businesses. We had a couple of businesses that wanted to syndicate to OSM. It. Was. A. Nightmare. We have addresses for every store. We don't know where the building is. Any store we place is going to be easy to place on a street near the building. If you've not placed it inside of the strip mall on the right spot, other people delete it. And what do you do with NYC? Suppose that you have a store that is somewhere on the second floor of a skyscraper. We don't know where it is in that building. Have you tried to map out thousands of department stores? We know hours, phone numbers, and addresses for each department. We have NO idea what the actual store layout is. Without the layout, OSM basically didn't seem to want the rest of what we can give them. If a customer is willing to pay us to deal with OSM, we're going to discourage them and ask for top dollar if they insist. It is that painful.
- juliendorra 9y ago
  Are you complaining that OSM contributors rejected wrong or imprecise location data about your client’s stores? If you are, I would like you to also consider this from the point of view of (maybe volunteer) contributors who prefer less data but correct data in OSM. Maybe draw a parallel with Wikipedia where a page that is not up to the standards might be deleted. (Not a contributor to OSM myself, but I have helped the French Drupal community and cofounded the Museomix community. Also, I appreciate precise data in OSM)
  - crooked-v 9y ago
    The broader point is that there's a structural issue here, as there's no ability to link data to addresses in a form that doesn't require literally knowing the location of a store within a building ahead of time.
  - btilly 9y ago
    No. I am stating that OSM has no setup for, or desire to, accept useful but incomplete data. And unless they do, they will not have data available that other services have. I know where Google, Bing, Facebook, Yelp, Foursquare, etc get phone numbers and current business hours for our clients. They are straightforward for us to work with, and we are happy to give it to them. OSM is not easy to work with us, so don't get that data. OSM has a choice. Be easy to work with, or have limited data. OSM has chosen to be hard to work with. That is a valid choice, but then you have to live with the result.
ris 9y ago
FWIW I don't really think any of these problems are "new" and don't think OSM is truly in any more trouble than it's ever been. And as many tensions as there obviously are in OSM, it hasn't stopped us getting pretty damn far. And if anyone thinks that OSM may be stagnating I would suggest getting involved in improving their local area (it's not hard!) and I think they will quickly find out for themselves.
nottorp 9y ago
I attempted to do an offline map with OSM/Mapbox/etc. I found out that it's a major bother even if you're willing to pay someone for the data. Project got canceled so I don't have a final opinion, but i'll avoid OSM next time...
- 6487698hhhh 9y ago
  We use Esri for rendering vector tiles and it works well.
madez 9y ago
I would love to mirror OSM including all the data, the rendering and the website. Is there a way to do that? I can get the data[0], but is there some guide or container image that includes all the rest that is needed in a working condition? [0] https://wiki.openstreetmap.org/wiki/Planet.osm https://wiki.openstreetmap.org/wiki/Planet.osm
- maxerickson 9y ago
  There's some guides for setting up tile rendering at https://switch2osm.org/ https://switch2osm.org/ It doesn't entirely make sense to mirror the editing api, unless you want to fork OSM. And I guess a lot of the user data isn't really readily available. If your goal is to help the project you might consider running a tile node https://wiki.openstreetmap.org/wiki/Servers/Tile_CDN https://wiki.openstreetmap.org/wiki/Servers/Tile_CDN
  - deleted 9y ago
    [deleted]
- maxerickson 9y ago
  More directly answering your question, most of the configuration of the various servers backing OSM is available here: https://github.com/openstreetmap/chef https://github.com/openstreetmap/chef
sorenjan 9y ago
One thing that made me a bit nervous to start editing is the lack of an easy way to roll back changes, at least I couldn't find one using the iD editor. If I make a mistake and save it I need to do new edits to try and fix it, possibly making things worse, instead of just removing the entire changeset. I had some issues with multipolygons and had to edit the same things maybe four or five times before I got it right. And why isn't the Strava heatmap easier to use? They allow OpenStreetMap to use the heatmap for tracing [1], but doing so in the iD editor requires you to copy a long url from the wiki and use a custom layer. This only shows the heatmap, no aerial photos or other mapping data, and it disappears when zoomed in too much. [0] https://i.imgur.com/V4Tn96F.png https://i.imgur.com/V4Tn96F.png [1] https://wiki.openstreetmap.org/wiki/Strava#Data_Permission_-_Allowed_for_tracing.21 https://wiki.openstreetmap.org/wiki/Strava#Data_Permission_-...
- bhousel 9y ago
  Hey sorenjan - The OSM Wiki is not correct. We've received no license from Strava that allows tracing outside of their iD fork. Current progress on securing permission to use the heatmap for OSM tracing is happening here: https://github.com/osmlab/editor-layer-index/pull/373 https://github.com/osmlab/editor-layer-index/pull/373
  - sorenjan 9y ago
    Ok, good to know. It seems mostly academic though, since using their iD fork with the slide add-on [0] is a better user experience, and someone from Strava mentions in the Github issue that it's approved internally. Of course I'm no lawyer and I understand that proper licensing is important, I'll just use their fork for now when tracing their heatmap. I see that you're the iD maintainer, are there any plans on adding the slide functionality? Or b-splines which could get tessellated client side before saving to the database? I can take this opportunity to thank you and the other contributors, I probably wouldn't edit OSM if it weren't for iD. [0] https://labs.strava.com/slide/ https://labs.strava.com/slide/
aluhut 9y ago
CTRL+F street complete Nothing. Even though you've commented on my post. Guess, it would have got in the way of the flow of negativity. Same goes for maps.me.
- rspeer 9y ago
  This sounds like it was meant as some kind of direct message to emacsen. Do you want to explain to anyone else what you mean?
  - aluhut 9y ago
    He already commented recently under a repost of his former article. I've pointed out the good things about OSM. He proceeded with answering to my comment but completely ignoring the content of it while advertising this piece to come. I'm still unable to find those good things in this article which supports my theory of overdrawn and almost(?) total negativity and ignorance of the good aspects of OSM.
NelsonMinar 9y ago
This is an extremely thoughtful and constructive post. Thanks to Serge for making it. I looked closely at getting deeply involved with OSM a few years back. And bounced off hard because the community is infested with assholes. There's a lot of very rude people who are active on the mailing lists that set the culture. It made me uninterested in trying to work through those cultural problems to improve the technical project. I'm not alone in this feeling.
altitudinous 9y ago
That is tl;dr I got criticised when I was a young coder for writing long rambling emails and articles. Now I am older I see why. Too much info in the world, I don't have time to read that although I am interested in why this person thinks openstreetmap is in trouble, but man a wall of text. This is why scientific articles have extracts at the top, bulletpoints work well too.
tomkinson 9y ago
Thank you for your contributions and writing this. Just spent a week of development time 'fixing' Nomination reverse geo-code issues and creating a pay for RGC backup because we exceeded the 1per sec max and had NO idea when it would turn back on. To boot, no help whatsoever determining why when we used it successfully for over a year. All of a sudden NOPE, we're cut off with foggy reasoning and no explanation of why or when we'll be back up. I've spent several months on maps. All of them. I know them through and through. I could sleep speak API docs and pricing for mapping services. We need OSM. OSM needs many of these 'fixes'. ASAP. No idea who was running things there. You gotta work on your analogies ;), but yeah if a user doesn't know how much the total usage is and can only use a percentage of that unknown total, it's ridiculous. Let's get a premium account going at resonable rates. Same for reverse geocode. No more than a 1ps max. Ok so what if we have a surge. Error 3. Give me a break. And so brashly the docs state this almost warning deterring any user from ever wanting to integrate because there is a big ol' flag saying 'we DO NOT want to scale with you!' At a bare minimum how about you average per second rates over a 10min aggregate. Soooo much room for overhaul and improvement here. This is a big start. Number one on HN! Let's fix/upgrade the antiquated OSM tech, logic and direction. It's a new world and the world needs OSM.
- rmc 9y ago
  You can install Nominatim om your own hardware and then make as many queries as you want. There are also companies offering hosted Nominatim (e.g. geofabrik), which will come with a limit and a way to see how much you have used.
jokoon 9y ago
I believe that OSM should deliver data, and not a service because it's extremely data-demanding. Although I have to say that since OSM offers data, they could make an effort into providing good exports of that data, with better segmentation and choice of format. You should not have to download planet.osm or use some overpass API. The choices are very limited, and offering data through XML is insufficient.
qwerty456127 9y ago
IMHO internet projects, especially free data-centric projects shold just be made as granular as possible. There should be a project to maintain the actual data base and independent projects to display and integrate this data. And as far as I know this is how it actually is in the world of OSM though the actual projects are not being developed and marketed actively enough. I may be wrong - I have never really used OSM (I prefer the proprietary map project native to the country I live in - it does the local job much better than both OSM and Google), just read about it a number of times.
ultim8k 9y ago
I personally believe that world needs a "wikipedia" for maps, not another hard to understand tool or technology. Yes. Technology matters of course, but not everyone is an engineer or entrepreneur.
donpdonp 9y ago
This was a wonderful post, thanks for the honesty and detail, emacsen. What I'd like to see as far as a technical overhaul, is reimagining the entire thing as a 3D dataset. Needing to model a paper map is no longer the goal, modeling space in 3 dimentions seems to be the more exciting/useful way forward. I'd like to have a model of my town in 3D, either at mostly manual low-poly resolution (aka minecraft) thats done by hand, or (though much more data intensive) with point cloud/lidar data. Web pages can receive the 3d data and render a 2d model, or the server can project 3d onto 2d tiles before delivery.

1062 points by emacsen 9y ago

I wouldn't mind seeing a two-tier approach, with authoritative municipal data as a base (see https://openaddresses.io/ https://openaddresses.io/ for an example of collecting a large number of sources and normalizing it) with user-submitted edits on top.

IntronExon 9y ago

Re: the issues with moderation, I sometimes wonder if Wikipedia were trying to start from scratch in the last few years... would it work? With all of the trolls and bots, easy access it might have died hard and early. One of the most significant technical problems with OSM is the lack of a review model, that is for a change to the map to be staged and then reviewed before being applied. Not having this functionality caused ripples of problems throughout the system, some of which I'll discuss here. Moderation is expensive, and Wikipedia got by initially with a pretty minimal degree of it, then ramped up as they got bigger. What if they’d been slammed by modern trolling at the start?

dalbasal 9y ago
Well... Nothing similar to Wikipedia has really happened in the last ten years. It's a fairly unique thing, still as relevant and important as ever. Hard to say if it's because Wikipedia was a long shot at any time, or because of the 90s web vs the current one, culturally or technically.
- twblalock 9y ago
  I think its because anything similar just gets implemented as part of Wikipedia/Wikimedia at this point.
  - freneticfox 9y ago
    Just to meta-meta this conversation and the article in new and confusing ways, Wikimedia also hosts a map server, using OSM's data, @ https://maps.wikimedia.org/ https://maps.wikimedia.org/ :)
  - dalbasal 9y ago
    True. I sort of meant "similar"in a wider sense: large scale, community effort to do something useful. Wikipedia represents a lot about what we that the internet, and internet culture was.^ Remember how big a deal Wikipedia is. A lot of that "human knowledge at your fingertips" is Wikipedia. Maybe that applies at a bigger level, with "similar"things being implemented as part of something. I suspect that a big part of the reason Wikipedias don't get started now is that they get implemented as startups. Anything showing promise on that scale is a "unicorn" and in 2018 Jimmy Wales would have had a choice between private/startup and internet-owned. That's a billion dollar choice. For example, stack overflow could have been built like a Wikipedia. If it was done in 1996, that would be a sensible choice. In 2018, a founder would be forgoing an unthinkable sum of money, and also resources to make it happen with. ^I guess all this needs to be caveated with "other than OSS."
emacsen 9y ago
It's an interesting question of how much flexibility you give people at the beginning vs as a project matures, but I'm looking at the situation with OSM pragmatically, that is to say "What OSM needs to do now", rather than "What they should have done". The problem of both bad edits and vandalism are very large (much like Google Mapmaker) and other measures we've put in place, such as changeset comments (which I helped develop) and vandalism tools don't seem to cut it.
- RosanaAnaDana 9y ago
  What about some kind of automated approach to validation? A major difference between OSM data and something like wikipedia is that, unlike wikipedia, at least some of the OSM data verifiable. A street somewhere, does exist, in a particular spot. There must be some way to croudsource validation.
  - spectre256 9y ago
    Ultimately it comes down to the resources to create such a thing, right? There have been some successes with crowdsourced validations such as Facebook's translation efforts (although I think in that case it was merely a component of the process). And it would rely on a volume of edits that OSM may not have. There may be 1000 people ready to verify the location of the Statue of Liberty, but probably not the location of every building in a remote area being mapped for HOT. Mapbox has built the start of a validation framework, and they have done very cool stuff. Of course then they get criticized for "forking" OSM. There are definitely valid criticisms there, but it seems like there's no way to win.
- IntronExon 9y ago
  I didn’t mean to distract from the main point of your piece (which was a good read btw), it just made me curious in general. Maybe the only way OSM can work is by treating moderation/curation as the problem to solve?
swang 9y ago
that kinda overestimates how long it took wikipedia to get to where it is today. when wikipedia launched, no one knew what it would grow into. why would large amounts of trolls bother harassing wikipedia when in all likelihood it would have just gone nowhere.
rspeer 9y ago
It appears to me that there was a golden age of crowdsourcing that lasted from maybe 1999 to 2010. If you started a project in this time, you had the chance to get lots of quality contributions, for free, from people whose motivation is just to be part of something. Now, you will get: - Modern trolling, as you point out. Well-organized armies of trolls, some working for political purposes, some just for the lulz, can subvert anything. - Disruptive bots (I don't know why this is a change since the golden age, but maybe it's simply that more people are better at programming) - Spam that's a full-time job to moderate - Lack of motivation because there are so many other fascinating things on the Internet - A change in expectations because money has entered the equation -- crowdsourcing is now something that people do as a menial job, and they aren't paid enough to do it well The sabotage by 4chan of the crowdsourcing team in the Pentagon shredder challenge [1; sorry about the stupid title] makes a good indicator of the end of the crowdsourcing era. That happened in 2011. I run a project, ConceptNet [2], that has obtained a lot of its data from crowdsourcing. Years ago, I stopped supporting the ability for the public to edit ConceptNet directly. (This does not mean the project shut down. Far from it. It currently produces state-of-the-art NLP models.) Supporting public contributions became obviously not worth the effort. Some projects that started at the right time and maintained excellent communities are still working, though -- Wiktionary would be an example, and it's one of the places that still provides new data to ConceptNet. [1] https://www.wired.com/2015/02/how-a-lone-hacker-shredded-the-myth-of-crowdsourcing/ https://www.wired.com/2015/02/how-a-lone-hacker-shredded-the... [2] http://conceptnet.io http://conceptnet.io

RosanaAnaDana 9y ago

I've found that in terms of supporting any kind of additional analysis, OSM data to be unreliable to the point of uselessness, which isn't to say there is a better alternative. I think this quote from your article: "One of the most significant technical problems with OSM is the lack of a review model, that is for a change to the map to be staged and then reviewed before being applied. Not having this functionality caused ripples of problems throughout the system, some of which I'll discuss here." is the primary reason I can't regularly use OSM data. Trust me when I say there are millions of instance where having reliable data regarding streets could save me time/ stress. Do you have any thoughts on how to improve the quality of OSMaps? Conceptually, I feel like the product is indispensable. In practice, I feel like the product is utterly unusable.

emacsen 9y ago
This was a hard post to make. OSM was a big part of my life for many years, and I have friends in the project. I like my friends (even the ones I disagree with). But where OSM has failed most in my view (and maybe I need to add this to the article) is that there are now free geographic datasets from local governments and without the ability to integrate them into OSM easily, and then update them (which we basically can't do), I fear OSM is going to ultimately fail at its mission
- toomuchtodo 9y ago
  Is this a technology issue? Or a funding issue?
  - croshan 9y ago
    Seems more like a political issue
  - maxerickson 9y ago
    It's frequently a licensing issue and then also a resources issue. There's relatively generalized tools for diffing data against OSM and outputting a change file, but lots of data is released using ambiguous or slightly incompatible licensing. One example is http://blog.improve-osm.org/en/2018/01/new-features-and-enhancements-in-cygnus/ http://blog.improve-osm.org/en/2018/01/new-features-and-enha...

amygdyl 9y ago

Is there a real hurdle in patents on map graphics held by the major mapping companies and national organisations? Could a goal of being truly open and complete, be incompatible with the interests of privacy, national security even? If community involvement is lacking, I immediately thought about the cimnunies who have real interest in mapping and particularly free open mapping, such as conservationists, residents of small towns concerned about planning and in my home Sussex county, hikers always fighting losing out to farming interests over ancient byways. But these constituencies need tools. I would immediately donate to such efforts if I knew they existed. Or a corporate like ACDSystems, who publish Deneba Canvas which while sadly lost adrift far from the software I remember, incorporated superb GIS and technical drawing capabilities. This would be the kind of marketing that traditionally worked very well, tools and support for local news stories, which use to run a long while anyway.

acabal 9y ago

I remember just a few years ago, I got a new phone and was interested in replacing as many Google apps with OSS maps that I could. (K9 instead of Google's mail app, etc.) Naturally I turned to OSM instead of Google Maps. But I was really surprised by how strange and difficult it was to install and use. As the author mentioned, OSM itself doesn't have any apps, which is very confusing. After some research I figured out that you have to download a 3rd party app. The 3rd party apps that do exist all use different terminologies, require downloading huge amounts of data and picking what maps you want ahead of time, and had different levels of functionality. The UIs were universally terrible (at the time at least). Needless to say I quickly gave up on using OSM and turned back to Google Maps--and I'm their perfect use case, a skilled techie with a deep interest in using free software! I can't imagine trying to explain how to use OSM to my mom, even if I somehow managed to install it for her. I can't comment on the rest of the article but its usability worries are on point.

leokennis 9y ago
Exactly. If the alternative is Google Maps (or even Apple Maps), it’s not good enough to make me pay $5 to download a map of 1 country and forcing me to keep 500mb of data on my device permanently.
striking 9y ago
Maps.me is pretty killer, though. The only issue I see with OSM is lack of traffic data. Generally when I drive with Google Maps, I already know the route but want Google to step in when there's traffic.
- jedberg 9y ago
  > Generally when I drive with Google Maps, I already know the route but want Google to step in when there's traffic. You should try Waze. I too mostly use Google to route me around traffic, but Waze is an order of magnitude better at it. It has more frequently updated traffic data and is willing to take you on side streets. For example the other day I was driving down a major road, and Google said to just stay on it, but Waze took me down a side street for two blocks and then back to the main road, saving me four minutes, which was 25% of the total trip time. Also, when making a long drive, Google chooses a route that is optimal right now for the entire route. Waze accounts for the fact that it will be an hour later when you get to the second half of your trip, and uses that to make better choices.

duozerk 9y ago

OSM gives people the tools to create their own map rather than offering them a simple, out of the box solution On this point and the main part of the first argument, I kinda disagree; the OSM project should specifically aim to do that: providing means for one to create a service. However, providing an actual map service would cost much, much more in bandwidth and I doubt a free software project like this has that kind of funds - actually serving tiles can add up to large costs pretty fast if enough people use the service. So if you want an actual service, you either do the hosting yourself or go see mapbox/whatever.

Avamander 9y ago
Time to move away from awful raster maps and to vector maps?
- duozerk 9y ago
  That would definitely help a lot (and osmdroid at least does support vector map sources); I imagine the traffic would still be significant though.
  - Avamander 9y ago
    Locus (not FOSS) supports vector maps and it's price is way cheaper than an external 1TB drive would cost, just for good map coverage of a few countries.
- sydd 9y ago
  There are some, e.g. OSMAnd on Android, they are awfully slow. I wonder why no one has written a proper, fast Opengl renderer for osm
  - lucb1e 9y ago
    > [like] OSMAnd on Android, they are awfully slow On my Galaxy Note 2 (2012), you're right. I've ordered a new phone yesterday because it's getting quite bad after five years of daily use. My girlfriend bought a phone for €300, and the one I ordered (now, a year later) for €250 actually has the same chipset. It runs great on that. I consider ~250eur to be the bottom of the smartphone market unless you're going for a potato to call and text with (not even browse the web). Given that it runs smooth on that... I'm not sure what you're running OsmAnd on with expectations of fast rendering.

andrewljohnson 9y ago

This post seems half right, half wrong. Wrong that OSM should be a service provider and not a database - I think it should be a database, and corporations are good for OSM. The post calls for OSM to essentially become Mapbox. It's insanely expensive to be Mapbox, and Mapzen just failed trying to be a less corporate version of Mapbox. And then the usage policy part of the article is moot or at least not that consequential if you don't believe OSM should be a service provider. Right: Moderation tools need to be created/fixed. New mappers need to be encouraged/onboarded better. Vandalism is a problem. OSMF culture is toxic. Hidden gatekeepers are toxic. Imports need to be supported. Bots need to be enabled, and we need to consider people who contribute code/tech/imports first class OSM contributors and not just glorify the individual mapper. Probably right: OSM needs layers, or even a more sweeping re-architecture to allow better versioning, moderation, better tooling, and easier understanding of the data. Probably wrong: Expanding the scope of OpenStreetMap to include transient data. Too much other work to do, maybe later for this.

emacsen 9y ago
> The post calls for OSM to essentially become Mapbox The problem is that most companies using OSM are using Mapbox. This leads to a few problems, including attribution, which I decided not to touch on, but is a serious problem that Mapbox doesn't address sufficiently. If they could use OSM directly, they'd be engaged with OSM directly. They'd care about OSM. Right now OSM shoves these users away and makes them into customers of some third party (maybe Mapbox, maybe someone else) instead of caring about OSM itself. > Expanding the scope of OpenStreetMap to include transient data. Too much other work to do, maybe later for this. How about letting users experiment and seeing what cool stuff they can come up with?
- andrewljohnson 9y ago
  > The problem is that most companies using OSM are using Mapbox. Is that even true? And is number of companies even a good metric? Based on revenue/market cap, the bulk of companies use OSM directly. The world's biggest company (Apple) uses OSM directly. Telenav uses it directly. Don't Tesla and Uber use OSM directly too? And we use Mapbox at Gaia GPS, but we also use OSM data directly. > How about letting users experiment and seeing what cool stuff they can come up with? In my experience building a company, if you focus on too many things, you get everything wrong. This smells to me of being not on target for making OSM as strong as possible, even though it has alluring upside.

tobltobs 9y ago

I can understand the frustration of OSM users that they have to create their own maps, but providing free tile servers would never be possible with the restricted funding for OSM.

jimmyrocks 9y ago

Thanks for posting this here! I feel like this is mostly good constructive criticism. I have given a few talks about using OSM data in the US government and it comes down to authoritative data sources and validation. It's a great starting point, but there needs to be another layer (or two) of validation before it can be used at the enterprise level. The National Map Corps has some good examples of how this can be implemented with data stewardship. I am also working on a few projects to take public domain data and put it into OSM. While it's useful and mostly open, it's definitely less open than the original data sources. Another issues is that as soon as someone edits the imported data, it will become less authoritative as well.

emacsen 9y ago
You're welcome this Jimmy. And you rock. You're one of the people who got me into OSM in the first place. And I still love OSM. I want to see it succeed! Go Igles!
- spectre256 9y ago
  As soon as I started reading your article I actually started thinking of Jimmy and how he's been a part of a real government entity (the national park service) sending data directly into OSM.
lucb1e 9y ago
> it will become less authoritative Or better than the authoritative original. Like with Wikipedia, where you often hear that people prefer to cite a single person as source (because "anyone could edit Wikipedia", as if no single person can setup a website or write a paper), at least Wikipedia has had thousands of reads and often dozens or hundreds of reviews. People are scared of the idea "it could be anyone", while the most likely editor is just a peer with good intentions.
- jimmyrocks 9y ago
  In many (most?) cases it can be better than the authoritative sources. But there is also the possibility for vandalism or sloppy work. There are established ways to combat this (peer review, data stewards, reputation scoring) and even OSM has a checkbox for further validation. OSM will never fit every use case, but it's good to understand where it falls short and can improve.

jedberg 9y ago

Open maps used to be the job of the USGS. They stopped doing it because of Google. I was touring the USGS in Palo Alto about 10 years ago, and the librarian there was lamenting the existence of Google. He said that while it was nice that Google was doing all this work, it was causing the USGS to stop doing it, since they could get the data from Google. He predicted that the USGS would get severely cut in their funding for mapping activities because of it, and he was right. He was worried that all the best data would be locked up in Google's servers, and he was right about that too.

jimmyrocks 9y ago
I used to work in the building with the USGS map store. It was very sad to see it close. It wasn't commercial map providers that caused it to close, it was the avilablilty of USGS topo maps online, and a faster data collection to map pipeline. You can still order maps printed on demand directly from the USGS.
maxerickson 9y ago
They weren't ever really in the business of publishing data. (A challenge: download a routable road network for the US from a USGS source)

reaperducer 9y ago

He's not wrong. OSM is too complex for the majority of businesses. I know two that chose to pay Google for their mapping needs rather than jump through OSM's hoops. Imagine if the only way to get Linux on your computer was to download and compile the whole thing from source yourself, instead of buying or downloading a distro. If OSM had a mapping solution like Google's, it would thrive. And just for fun, it should undercut Google's pricing.

spectre256 9y ago
In that sense, perhaps the linux distro model actually makes more sense than centralizing power into one entity? Ubuntu, Redhat, and many others produce their own polished, supported products, sell it, and then give back to the central parts of the community in the process. Mapbox and others certainly seem like they essentially play the part of "OSM Distro" in that regard. And maybe its okay for multiple entities to be able to contribute in different ways.
Doctor_Fegg 9y ago
> If OSM had a mapping solution like Google's, it would thrive. It does, it’s called Mapbox. Plus a number of excellent smaller providers - Thunderforest, Geofabrik, Graphhopper etc.
- beering 9y ago
  And kind of like Linux, you have a patchwork of unsatisfying solutions. Mapbox isn't always cheaper than Google. They have not lowered their tile prices in years, for example. And on geocoding and some other APIs they give out poor results compared to Google. Do they have competition? Cloudmade went in some car-related direction. Mapzen never managed to have a real business plan. And your "excellent small providers"? Yeah, Andy may be able to create some bespoke solution for you if he has time, and Geofabrik can wrangle some host-it-yourself setup onto AWS, but it's still very true that OSM themselves don't have a good story on a lot of mapping needs. You can either browse their web map with unusable search or download the 38GB database, and that's about it. No wonder many businesses just pay Google rather than dive into the rabbit hole of OSM.

aspirin 9y ago

Im my opinion the biggest problem is the data model (or lack of). Tags i.e. key-value pairs are used to add properties to geometry. Mappers use different combinations of keys/values to describe the same thing. The post touches on this, but does not offer solutions other than adding "layers". There should be a strict schema that would be community managed through some process.

maxerickson 9y ago
A strict schema won't help when people still use it incorrectly. Improvement over what exists now would require that the schema be applied strictly and that isn't going to happen with a huge crowdsourced project.
lucb1e 9y ago
Like what, tag proposals where people vote yay or nay? We have those. As an active mapper, I very rarely find two ways of tagging the same thing, so the process seems to work fine. And if you do find a situation where there are multiple options, it takes only a quick check on taginfo to find the more commonly used one. If you go with that, it should converge soon.
- aspirin 9y ago
  Currently there is nothing to stop mappers using tags in wrong ways. It should just be flat out impossible. If a field/tag isn't defined in the schema, it could not be used. The range of values should be limited in the same way, too.
  - maxerickson 9y ago
    How does a schema stop someone from marking a building as a pond?
  - lucb1e 9y ago
    Ah, I see where you're going with this. It would slow innovation down to a trickle, though. Anyone who wants to do something that's not yet standardized would have to stash their edits somewhere and wait for official approval. And that would likely take weeks or months at best. Currently, the best idea wins automatically because good ideas are used by both mapping applications and mappers. Maybe something in the middle, like the current wiki system but with an enforcement rule of whatever is on the wiki. Anyone could freely edit it because it's a wiki (adding allowed options, and removing/changing any that have extremely few uses (zero, or perhaps a few which are then removed)), but at least there is a correct spec. If people really had to adhere to a strict, slow-moving standard with pre-approval, the spec would also become extremely complex and very difficult to implement, because the spec-makers will want to think of every eventuality. See any other large specification ever, like HTML or something. I can't see that ending well. I think a hybrid system might be better than a strict one, given the choice between those two. I like thinking about it though! Always good to have hear ideas.

RcouF1uZ4gsC 9y ago

Even Apple the most valuable company in the world with hundreds of billions of dollars of cash on hand is way behind Google in terms of mapping. https://www.justinobeirne.com/google-maps-moat/ https://www.justinobeirne.com/google-maps-moat/ If Apple has trouble keeping feature parity with Google Maps, there is no way that an endeavor like Open Street Maps has any real chance of making something with the breadth, features, and usability of Google Maps.

reaperducer 9y ago
> there is no way that an endeavor like Open Street Maps has any real chance of making something with the breadth, features, and usability of Google Maps But does it need to? Isn't there a market for basic mapping, without all the needless feature bloat? I'd rather a mapping service concentrate on getting roads right, and not whether there is a Howard Johnson's nearby.
- zbentley 9y ago
  Most people are using those roads to get to somewhere, be it a HoJo or a street address. So the geotagging ability is very, very important. I suspect a map system with vastly superior routing information (just roads and coordinates) and slightly inferior geotagging would still lose to one with the opposite balance.
spectre256 9y ago
True, at least for the moment, although that doesn't mean we shouldn't try. One nice thing about open data is it can outlast. What do you think will survive and be readily accessible on the internet longer, Wikipedia or Quora? My money's on Wikipedia 100%. With OSM things are similar. Obviously geo data has additional ways it can grow stale, but still. Google is big today but there's no guarantee for the future.
detaro 9y ago
Goes both ways: Openstreetmap is better for some things already, and I don't see Google catching up to it at them. A big issue is consistency though: OpenStreetMap varies way more in quality between regions, to the point that detailed conversations almost make no sense without disclaimers about where you are talking. OpenStreetMap is a really good map where a strong community exists (either locally or dedicated remote interest) and a pretty bad map if no large effort has touched the area.

pornel 9y ago

I've tried to build an open check-in app based on OSM data, and I've ran into many of the problems mentioned in the article: • Tags and relations are a total mess. There are many different and overlapping schemes. There is obviously outdated/sub-standard data, but automated fixes are forbidden. There are some neat tools to make sense of the tag data, but ultimately I had to deal with all that mess and complexity on my end. • The lack of moderation or a way to officially act as an intermediary for edits (other than clunky oAuth flow) meant I couldn't expose an Add/Edit/Fix functionality for my users. Similarly, if I wanted to let my users upload a photo of a venue, there was no service to store it, no identifier or tag to associate it. • The API returns only nodes entirely within an area, rather than everything intersecting a given area. If you fetch small lat/long area, you won't get a node for the country/city/district it's in, unless you stumble upon a point in its border or label. That meant I couldn't fetch things from OSM API as needed, and had to have my own copy of the data and the DB and custom search. • The map looks ugly without buildings. There was free building data available for areas I was interested in, but I was told to copy them by hand.

maxerickson 9y ago
Were you doing your own map rendering? In that case you wouldn't have to add the buildings to OSM to show them, just merge the data during your map preparation. Nominatim does have a reverse geocode api which returns the sort of place information you want in your third point (given a coordinate, returns an address).
ris 9y ago
> There is obviously outdated/sub-standard data, but automated fixes are forbidden. Please consider arms race encouraging automated edits would begin. A lot of people have a lot of awfully different views about how map data should be recorded (you thought source code formatting was contentious?) > I couldn't expose an Add/Edit/Fix functionality for my users. Lots of tools have managed this. > if I wanted to let my users upload a photo of a venue, there was no service to store it No. That's not what OpenStreetMap is. They aren't there to give out free image storage. > The API returns only nodes entirely within an area Yes, it's meant to be a low-level API which makes few assumptions about the meaning of any of the data. For the ability to make more complex queries you could have looked at Overpass. And possibly there is a space for a third party service to provide an API with such an ability. Again, I don't see that as OSM's business - OSM here should be considered more like the UN. > There was free building data available for areas I was interested in, but I was told to copy them by hand. And yet NYC have managed somehow to get most city building data into OSM, so perhaps you haven't investigated every possible option for getting what you want here...?

gpsx 9y ago

Previously I really like the flexibility of the tagging system in open street map. I used something like it for a unrelated commercial product and we have found we have to switch away from it. And we had very tight control over the definitions of the tagging we were using. The tools were difficult, as is mentioned in the main post. That is certainly an inconvenience but I figured we could muddle through that. The bigger problem is that the customers just never got it. They wanted a single, authoritative classification for an object. Our plan is to move to a system inspired by the model used in BIM for classification. The princicple difference is that in open street map all data identifying an object has to lie within the tags. In the alternate system, a single identifier for the object type is used, and this references external metadata that gives you additional information which does not have to be embedded in the tags on the object. An example of additional information held external to the map and tags is that a bathroom is a type of room. I don't have to include in the map data that the object is both a bathroom and a room. I should add, the external meta data also gives a strict definition of what types of tags (and maybe values) are allowed on the given type of object. If you need something a little different, you can make a new type. I am not sure how well this type of system could work in open street map. Is it too rigid? Or would there be too much proliferation of types? Classification is always hard and I am not sure people have solved that problem in any large context. And of course, changes like this would mean a new database. If people ever do think about making an OSM 2.0 maybe this, or another tagging alternative, is something to consider. (edit: typos)

pbowyer 9y ago
I like the sound of this, like an object graph of data from OSM. > I am not sure how well this type of system could work in open street map. Is it too rigid? Or would there be too much proliferation of types? Classification is always hard and I am not sure people have solved that problem in any large context. Ontologies are well-researched. Unfortunately they're not my area of expertise as I turned away from doing a PhD in them and went into business. I have been playing with a controlled vocabulary since Christmas for a personal project and the complexity of definition has made me think this is why Google et al have gone for a more flexible, machine-learned typing, and why the types for schema.org are so broad. Having been negative, I think this is exactly what OSM needs to organise the world's objects.

just_testing 9y ago

I'm also interested in the development of OSM (being an editor and a data consumer) and I've witnessed first hand some of your issues. For instance, once I've had to create my own "geocoder" for transit fines in São Paulo using OSM data, because no geocoder commercially available would be able to deal with the data. So, now that we clarified I'm not a troll, nor a bot, I ask you: how should we deal with those issues? Join en masse the OSMF so we can vote for a change? Begin the "New-OSM-But-Better Foundation"? I also think having layers (particularly, layers vouched by governments) would be awesome. I saw several governmental branches in Brazil wanting to be more integrated with OSM - this would allow them to stay "inside" OSM (and being an accredited citizen), instead of creating their own silos, importing data from time to time.

emacsen 9y ago
Honestly, I have no idea.

SteveCoast 9y ago

I'm the DRI for most of these problems. Random thoughts: I think David Ogilvy said something like "Search all the parks in your city, you'll find no statues of committees". Everyone was super keen to make OSM in to a web of committees, and this is one root of the problem from an organizational point of view. It's not super clear to me what the OSMF has done in the last (to pick a number) 5 years. (I don't regard "doing the same thing as last year" as relevant to anything, I just have a personal bias towards new things). From a technical point of view we merged sysadmin (who inherently want stability) with development (who inherently want change) and stability won (it doesn't involve arguments or as much work, I guess). That's why the API looks the same way I designed it X number of years ago. It's still a wonderful project, but more of a work of art than anything else. It's not like wikipedia didn't have all the same problems. Making a project that really evolves and grows over large periods of time is really hard. Like - most of the things on the S&P 500 won't be there in 10 or 20 years or something. I think what really happens is that projects go through a lifecycle and something new comes along to replace them, rather than anything getting fixed. I'm not sure if this is a good example, but, you could try to "fix" Apache, or you could just do this new thing, "nginx". For all kinds of reasons, the new thing is more efficient. One of the few counter examples I think is going to be Amazon. The world has moved on from making vector maps and OSM solved almost all the related problems (just not geocoding, which annoys me no end). It moved on, because of OSM! All the money and interest in maps is now around autonomous navigation, which means 3D maps and other things outside of OSMs area.

lucb1e 9y ago
> I'm the DRI All I can find is "Dietary Reference Intake" on https://www.merriam-webster.com/medical/DRI https://www.merriam-webster.com/medical/DRI -- what do you mean? (I agree with most of your post by the way (except for the bit about it being art, which would mean it has no function except to be art), I'm just unclear on what you meant here.)
- SteveCoast 9y ago
  Directly Responsible Individual https://en.wikipedia.org/wiki/Steve_Coast https://en.wikipedia.org/wiki/Steve_Coast

WhitneyLand 9y ago

contact email below

emacsen 9y ago
You can email me at emacsen at emacsen dot net

btilly 9y ago

Here is my experience. I work for a company that syndicates location data for businesses. We had a couple of businesses that wanted to syndicate to OSM. It. Was. A. Nightmare. We have addresses for every store. We don't know where the building is. Any store we place is going to be easy to place on a street near the building. If you've not placed it inside of the strip mall on the right spot, other people delete it. And what do you do with NYC? Suppose that you have a store that is somewhere on the second floor of a skyscraper. We don't know where it is in that building. Have you tried to map out thousands of department stores? We know hours, phone numbers, and addresses for each department. We have NO idea what the actual store layout is. Without the layout, OSM basically didn't seem to want the rest of what we can give them. If a customer is willing to pay us to deal with OSM, we're going to discourage them and ask for top dollar if they insist. It is that painful.

juliendorra 9y ago
Are you complaining that OSM contributors rejected wrong or imprecise location data about your client’s stores? If you are, I would like you to also consider this from the point of view of (maybe volunteer) contributors who prefer less data but correct data in OSM. Maybe draw a parallel with Wikipedia where a page that is not up to the standards might be deleted. (Not a contributor to OSM myself, but I have helped the French Drupal community and cofounded the Museomix community. Also, I appreciate precise data in OSM)
- crooked-v 9y ago
  The broader point is that there's a structural issue here, as there's no ability to link data to addresses in a form that doesn't require literally knowing the location of a store within a building ahead of time.
- btilly 9y ago
  No. I am stating that OSM has no setup for, or desire to, accept useful but incomplete data. And unless they do, they will not have data available that other services have. I know where Google, Bing, Facebook, Yelp, Foursquare, etc get phone numbers and current business hours for our clients. They are straightforward for us to work with, and we are happy to give it to them. OSM is not easy to work with us, so don't get that data. OSM has a choice. Be easy to work with, or have limited data. OSM has chosen to be hard to work with. That is a valid choice, but then you have to live with the result.

ris 9y ago

FWIW I don't really think any of these problems are "new" and don't think OSM is truly in any more trouble than it's ever been. And as many tensions as there obviously are in OSM, it hasn't stopped us getting pretty damn far. And if anyone thinks that OSM may be stagnating I would suggest getting involved in improving their local area (it's not hard!) and I think they will quickly find out for themselves.

nottorp 9y ago

I attempted to do an offline map with OSM/Mapbox/etc. I found out that it's a major bother even if you're willing to pay someone for the data. Project got canceled so I don't have a final opinion, but i'll avoid OSM next time...

6487698hhhh 9y ago
We use Esri for rendering vector tiles and it works well.

madez 9y ago

I would love to mirror OSM including all the data, the rendering and the website. Is there a way to do that? I can get the data[0], but is there some guide or container image that includes all the rest that is needed in a working condition? [0] https://wiki.openstreetmap.org/wiki/Planet.osm https://wiki.openstreetmap.org/wiki/Planet.osm

maxerickson 9y ago
There's some guides for setting up tile rendering at https://switch2osm.org/ https://switch2osm.org/ It doesn't entirely make sense to mirror the editing api, unless you want to fork OSM. And I guess a lot of the user data isn't really readily available. If your goal is to help the project you might consider running a tile node https://wiki.openstreetmap.org/wiki/Servers/Tile_CDN https://wiki.openstreetmap.org/wiki/Servers/Tile_CDN
- deleted 9y ago
  [deleted]
maxerickson 9y ago
More directly answering your question, most of the configuration of the various servers backing OSM is available here: https://github.com/openstreetmap/chef https://github.com/openstreetmap/chef

sorenjan 9y ago

One thing that made me a bit nervous to start editing is the lack of an easy way to roll back changes, at least I couldn't find one using the iD editor. If I make a mistake and save it I need to do new edits to try and fix it, possibly making things worse, instead of just removing the entire changeset. I had some issues with multipolygons and had to edit the same things maybe four or five times before I got it right. And why isn't the Strava heatmap easier to use? They allow OpenStreetMap to use the heatmap for tracing [1], but doing so in the iD editor requires you to copy a long url from the wiki and use a custom layer. This only shows the heatmap, no aerial photos or other mapping data, and it disappears when zoomed in too much. [0] https://i.imgur.com/V4Tn96F.png https://i.imgur.com/V4Tn96F.png [1] https://wiki.openstreetmap.org/wiki/Strava#Data_Permission_-_Allowed_for_tracing.21 https://wiki.openstreetmap.org/wiki/Strava#Data_Permission_-...

bhousel 9y ago
Hey sorenjan - The OSM Wiki is not correct. We've received no license from Strava that allows tracing outside of their iD fork. Current progress on securing permission to use the heatmap for OSM tracing is happening here: https://github.com/osmlab/editor-layer-index/pull/373 https://github.com/osmlab/editor-layer-index/pull/373
- sorenjan 9y ago
  Ok, good to know. It seems mostly academic though, since using their iD fork with the slide add-on [0] is a better user experience, and someone from Strava mentions in the Github issue that it's approved internally. Of course I'm no lawyer and I understand that proper licensing is important, I'll just use their fork for now when tracing their heatmap. I see that you're the iD maintainer, are there any plans on adding the slide functionality? Or b-splines which could get tessellated client side before saving to the database? I can take this opportunity to thank you and the other contributors, I probably wouldn't edit OSM if it weren't for iD. [0] https://labs.strava.com/slide/ https://labs.strava.com/slide/

aluhut 9y ago

CTRL+F street complete Nothing. Even though you've commented on my post. Guess, it would have got in the way of the flow of negativity. Same goes for maps.me.

rspeer 9y ago
This sounds like it was meant as some kind of direct message to emacsen. Do you want to explain to anyone else what you mean?
- aluhut 9y ago
  He already commented recently under a repost of his former article. I've pointed out the good things about OSM. He proceeded with answering to my comment but completely ignoring the content of it while advertising this piece to come. I'm still unable to find those good things in this article which supports my theory of overdrawn and almost(?) total negativity and ignorance of the good aspects of OSM.

NelsonMinar 9y ago

This is an extremely thoughtful and constructive post. Thanks to Serge for making it. I looked closely at getting deeply involved with OSM a few years back. And bounced off hard because the community is infested with assholes. There's a lot of very rude people who are active on the mailing lists that set the culture. It made me uninterested in trying to work through those cultural problems to improve the technical project. I'm not alone in this feeling.

altitudinous 9y ago

That is tl;dr I got criticised when I was a young coder for writing long rambling emails and articles. Now I am older I see why. Too much info in the world, I don't have time to read that although I am interested in why this person thinks openstreetmap is in trouble, but man a wall of text. This is why scientific articles have extracts at the top, bulletpoints work well too.

tomkinson 9y ago

Thank you for your contributions and writing this. Just spent a week of development time 'fixing' Nomination reverse geo-code issues and creating a pay for RGC backup because we exceeded the 1per sec max and had NO idea when it would turn back on. To boot, no help whatsoever determining why when we used it successfully for over a year. All of a sudden NOPE, we're cut off with foggy reasoning and no explanation of why or when we'll be back up. I've spent several months on maps. All of them. I know them through and through. I could sleep speak API docs and pricing for mapping services. We need OSM. OSM needs many of these 'fixes'. ASAP. No idea who was running things there. You gotta work on your analogies ;), but yeah if a user doesn't know how much the total usage is and can only use a percentage of that unknown total, it's ridiculous. Let's get a premium account going at resonable rates. Same for reverse geocode. No more than a 1ps max. Ok so what if we have a surge. Error 3. Give me a break. And so brashly the docs state this almost warning deterring any user from ever wanting to integrate because there is a big ol' flag saying 'we DO NOT want to scale with you!' At a bare minimum how about you average per second rates over a 10min aggregate. Soooo much room for overhaul and improvement here. This is a big start. Number one on HN! Let's fix/upgrade the antiquated OSM tech, logic and direction. It's a new world and the world needs OSM.

rmc 9y ago
You can install Nominatim om your own hardware and then make as many queries as you want. There are also companies offering hosted Nominatim (e.g. geofabrik), which will come with a limit and a way to see how much you have used.

jokoon 9y ago

I believe that OSM should deliver data, and not a service because it's extremely data-demanding. Although I have to say that since OSM offers data, they could make an effort into providing good exports of that data, with better segmentation and choice of format. You should not have to download planet.osm or use some overpass API. The choices are very limited, and offering data through XML is insufficient.

qwerty456127 9y ago

IMHO internet projects, especially free data-centric projects shold just be made as granular as possible. There should be a project to maintain the actual data base and independent projects to display and integrate this data. And as far as I know this is how it actually is in the world of OSM though the actual projects are not being developed and marketed actively enough. I may be wrong - I have never really used OSM (I prefer the proprietary map project native to the country I live in - it does the local job much better than both OSM and Google), just read about it a number of times.

ultim8k 9y ago

I personally believe that world needs a "wikipedia" for maps, not another hard to understand tool or technology. Yes. Technology matters of course, but not everyone is an engineer or entrepreneur.

donpdonp 9y ago

This was a wonderful post, thanks for the honesty and detail, emacsen. What I'd like to see as far as a technical overhaul, is reimagining the entire thing as a 3D dataset. Needing to model a paper map is no longer the goal, modeling space in 3 dimentions seems to be the more exciting/useful way forward. I'd like to have a model of my town in 3D, either at mostly manual low-poly resolution (aka minecraft) thats done by hand, or (though much more data intensive) with point cloud/lidar data. Web pages can receive the 3d data and render a 2d model, or the server can project 3d onto 2d tiles before delivery.