9 ms·

Overture Maps Foundation releases open map dataset

402 points by chippy 3y ago

dhx 3y ago
I have an interest in this topic as a contributor to AllThePlaces[1], an open source project collating Scrapy spiders (MIT license) that crawl websites of franchises/retail chains that you'd find listed in name-suggestion-index[2] to retrieve location data (CC-0 license). The project is just short of collecting 3 million points of interest from almost 1700 spiders. Overture Maps appears to be quite a closed and proprietary project, with claims of openness limited to being able to download a data set and accompanying schema specification. Some issues that immediately come to mind: 1. There is no published description for how the data was generated. End users thus are given no assurance of how accurate and complete the data is. a. As an example, administrative boundaries are frightfully complex and include disputed boundaries, significant ambiguity in definition of boundaries, and trade-off between precision of boundaries versus performance of algorithms using administrative boundary data. Which definition of a boundary does Overture Maps adhere to, or can it support multiple definitions? b. It's probable that Microsoft have contributed ld+json/microdata geographic data from BingBot crawls of the Internet. This data is notoriously incorrect, including fields mixed up and invalidly repurposed, "CLOSED" in field names to denote closure of a place 5 years ago but the web page remains online, and much ambiguity in opening hours specifications. For AllThePlaces, many of the spiders developed require human consideration, sometimes of considerable complexity, to piece together horribly messy data that is published by shop and restaurant franchises, and other organisations providing location data via their websites. c. For location information where +/- 1-5m accuracy and precision may be required (e.g. individual shops within a shopping centre[3]), source data is typically provided by the authoritative sources with 1mm precision and +/- 10-100m accuracy. AllThePlaces, Overture Maps, Google Maps and similar still need human editors (OpenStreetMap editors) to do on-the-ground surveys to pinpoint precise locations and to standardise the definition of a location (e.g. for a point, should it be the centroid of the largest regular polygon which could be placed in the overall irregular polygon, the center of mass of a planar lamina, the location of the main entrance, or some other definition?). d. If Overture Maps is dependent on BingBot for place data, they'll miss an enormous number of points of interest that BingBot would never be able to find. For example, an undocumented REST/JSON/GraphQL API call or modification to parameters to an observed store locator API call may be necessary to return all locations and relevant fields of data. Website developers routinely do stupid things with robots.txt such as instruct a bot to crawl 10k pages (1GB+) from a sitemap last updated 5 years ago rather than make 10 fast API calls for up-to-date data (5MB). Overture Maps would be free to consume data from AllThePlaces as it is CC-0 licensed, and possibly correlate it with other data sources such as BingBot crawl data, a government database of licensed commercial premises or postal address geocoding data. However the messiness of data in various sources would be approaching impossible to reconcile, even for humans, and Overture Maps would possibly have to decide whether to err on the side of having duplicates, or lack completeness. 2. There is no published tooling for how someone else can reproduce the same data. a. AllThePlaces users fairly frequently experience the wrath of Cloudflare, Imperva and other Internet-breaking third parties, as well as custom geographic blocking schemes and more rarely, overzealous rate limiting mechanisms. If Overture Maps is dependent on BingBot crawls, they'll have a slight advantage over AllThePlaces due to deliberate whitelisting of BingBot from the likes of Cloudflare, Imperva, customer firewalls, etc. However, no matter whether you're AllThePlaces or Overture Maps or anyone else, if you want to capture as many points of interest as possible across the world, use of residential ISP subnets and anti-bot-detection software is increasingly required. They'll need people in dozens of countries each crawling websites targeted to the same country, using residential ISP address space. Otherwise they end up with an American view of the world, or a European view of the world, or something else that isn't the full picture. b. If Overture Maps has locations incorrect for a franchise/brand due to a data cleansing problem or sourcing data from a bad source (perhaps non-authoritative), there are no software repositories for the franchise/brand to raise an issue or submit a patch against. [1] https://www.alltheplaces.xyz/ https://www.alltheplaces.xyz/ [2] http://nsi.guide/ http://nsi.guide/ [3] Example Australian shopping centre as captured by AllThePlaces: https://www.alltheplaces.xyz/map/#18.07/-33.834646/150.989528 https://www.alltheplaces.xyz/map/#18.07/-33.834646/150.98952...
- mistrial9 3y ago
  strong agree -- additionally, there is clear public relations going on, announcing maps like this as "trusted" or "official" backing, presumably for decisions over commerce, safety, insurance claims or other non-security but valuable, data-oriented activity. The public or public oversight, has what recourse here?
- tony_cannistra 3y ago
  > 2. There is no published tooling for how someone else can reproduce the same data. This is true, and is actually the whole point of Overture. Overture was developed to enable private companies to leverage open data (like OpenStreetMap) but also combine it with their proprietary data and processes. The intention is to share the result with a relatively permissive license (a new thing called the Community Database License Agreement) but keep the process and underlying data proprietary.
  - justinclift 3y ago
    Sounds like that's going to be a problem for proving/reproducing results independently then. :(
  - uneekname 3y ago
    Yes, exactly. Overture is, for better or worse, looking to create a dataset that is more difficult to mess with / contribute to. I'm big on OpenStreetMap, but I can't deny its a bit of a liability for Facebook and other companies that display maps to their users. There is the occasional vandalism edit that simply can't be shown to an end-user. Facebook put significant effort into maintaining a moderated version of the OSM database that lags behind the real-time edits. Facebook, Microsoft, TomTom, etc. know this is a ton of work and want to pool their resources. Making it open also helps to openly compete with Google, the other big map data provider. If you want to contribute to Overture as an end-user, AFAICT your best option is to edit OpenStreetMap and see if your changes eventually get pulled in. Overture has promised the OSM community that they'll make much of their data available to be contributed back, we'll see if that pans out. When it comes to AllThePlaces -- as an OSM nerd it seems like there is an opportunity to build a better bridge between this and OpenStreetMap, to make it easier to quickly update businesses in an area. Recently there has been a pretty successful push to link OSM data with WikiData, using tools like the OSM ↔ Wikidata matcher [0]. For POIs, it's a lot of work to add a bunch of local businesses, even with tools like EveryDoor [1]. It would be so cool to see AllThePlaces integration into RapID for example, if there isn't already(?) [0] https://osm.wikidata.link/ https://osm.wikidata.link/ [1] https://every-door.app/ https://every-door.app/
pininja 3y ago
POIs, buildings, transportation network, and admin boundaries layers.. on my!
sdfghswe 3y ago
What's the difference between this and Open Street Maps?
- pininja 3y ago
  To start, the admin boundaries and POIs are given a more permissive license. They’re mission is to make permissive map datasets.
- mikeyouse 3y ago
  Ha I was just wondering the same and Overture has a FAQ that answers that exact question; Overture is a data-centric map project, not a community of individual map editors. Therefore, Overture is intended to be complementary to OSM. We combine OSM with other sources to produce new open map data sets. Overture data will be available for use by the OpenStreetMap community under compatible open data licenses. Overture members are encouraged to contribute to OSM directly.
  - zigzag312 3y ago
    We've focused on designing a data schema that is easy for developers to quickly understand and use in building map products. https://docs.overturemaps.org/ https://docs.overturemaps.org/ OSM focuses on open map editing, but due to it's flexible schema, it can be hard to extract needed information from OSM. Overture seems to focus on providing map data (from multiple sources) that can be used more easily. EDIT: OSM is also trying to improve the OSM data model for easier processing. https://github.com/osmlab/osm-data-model https://github.com/osmlab/osm-data-model
- matkoniecz 3y ago
  OpenStreetMap is community of people creating fully open map database. From roads, shops and rivers to tourism attractions, hiking routes and hospitals, with all kind of detail. Anyone can contribute (and as someone quite involved in OSM: if you want to map something, especially area near you: you are welcome!). Overture is a new group of companies releasing some datasets on open licenses, but methods used to create them remain proprietary. Some of released data is theirs, many datasets are repackaged OpenStreetMap data.
mikeyouse 3y ago
The GitHub repo is here; https://github.com/OvertureMaps/data https://github.com/OvertureMaps/data -- Licenses look super permissive - I'm not that familiar with the state of Open Mapping but the GERS idea looks great too. https://docs.overturemaps.org/gers/ https://docs.overturemaps.org/gers/
- orra 3y ago
  The CDLA license feels like NIH... but at least it looks genuinely permissive, as opposed to fake open source. Curiously, two of the four layers use the ODbL.
  - matkoniecz 3y ago
    > Curiously, two of the four layers use the ODbL. Probably this data is reformatted OpenStreetMap data or derived from it (so is licensed as ODBL and requires attributing OpenStreetMap)
tony_cannistra 3y ago
For context, Overture Maps is a project that intends to enable large players in the geospatial space (TomTom, Amazon, Microsoft, but notably _not_ Google) to leverage open data sets (OpenStreetMap among them) alongside proprietary data and processes that they own. The consortium intends to enable a framework for enhancing geospatial data based in open data sets (e.g. OSM) with their own proprietary processes, and re-release it with a permissive license (the Community Database License Agreement - CDLAv2), but keep the data and processes required to create that dataset proprietary. The project has created a lot of conversation in the OpenStreetMap community, but in general I think it's good to see so many resources put into the OSM-adjacent world.
- gumby 3y ago
  I came to ask for this info so thanks for the succinct summary!
- alexchamberlain 3y ago
  How do they do that? I thought OSM was licensed in such a way that if you combine it with other data, the whole thing needed to be released with the same license?
  - tony_cannistra 3y ago
    You are mostly right, except that in addition to the same license, the Share-Alike provision in the OSM license (the ODbL [0]) also enables derivatives to be re-released with "A compatible license." I am not a lawyer, but I would assume someone at Overture has deemed their license (the CDLAv2 [1]) to be "compatible" with ODbL. (edit: From the link in OP, the OSM-based data is being released with ODbL. I think I'm wrong about license compatibility. Again, not a lawyer.). [0]: https://opendatacommons.org/licenses/odbl/1-0/ https://opendatacommons.org/licenses/odbl/1-0/ [1]: https://cdla.dev/permissive-2-0/ https://cdla.dev/permissive-2-0/
    - Doctor_Fegg 3y ago
      CDLA Permissive is not downstream compatible with ODbL. ODbL is a sharealike licence, CDLA Permissive (as the name suggests) isn’t. Overture can’t relicense OSM-derived content under CDLA Permissive. From a brief skim of the landing page, it looks like the OSM-derived content is not being offered permissively. See https://overturemaps.org/download/ https://overturemaps.org/download/.
deleted 3y ago
[deleted]
joshstrange 3y ago
If you are looking for maps themselves (a la Google Maps) I highly recommend checking out Protomaps [0]. It provides a single file (PMTile format) that contains all the map data and you can request ranges from the single file to get the required vector data you need for a given area/zoom. So you can setup something like the following: S3 (host the myfile.pmtiles) <- Lambda (takes the x/y/z from the path and requests the correct range) <- Cloudfront cache tile response Then you can setup tiles.mydomain.com (or use the cloudfront domain directly) and then use Leaflet or similar on the frontend to fetch/render the tiles. For Leaflet you use the protomaps plugin/lib and give it a url like "https://tiles.yourdomain.com/20230408/{z}/{x}/{y}.mvt https://tiles.yourdomain.com/20230408/{z}/{x}/{y}.mvt" where "20230408" maps to "20230408.pmtiles" in your S3 bucket. Now I can drop new pmtiles files into that bucket and update my clients to use the new source. And since the tiles are in vector format you can theme them however you want in the client which is neat. Lastly you don't have to use the 100+GB whole-earth tileset. You can use a tool [1] (provided by the same guy) to download a dataset for just a given geographical region. The .pmtiles file is a little over 100GB but the whole setup took me only an hour or two max to get running and will cost way less than Google Maps to run. [0] https://protomaps.com/ https://protomaps.com/ [1] https://app.protomaps.com/downloads/small_map https://app.protomaps.com/downloads/small_map
- mdaniel 3y ago
  their demo isn't doing them any favors since it seems to stop at some random zoomlevel, eliding the kind of detail that I'd care about if I were evaluating that solution digging into their GH repos surfaces https://protomaps.github.io/PMTiles/?url=https%3A%2F%2Fr2-public.protomaps.com%2Fprotomaps-sample-datasets%2Fprotomaps-basemap-opensource-20230408.pmtiles https://protomaps.github.io/PMTiles/?url=https%3A%2F%2Fr2-pu... which is at least more detailed if not quite a GMaps killer
  - qwertox 3y ago
    I agree, if it weren't for your comment, I was about to ditch it as a candidate because it stopped zooming at a bad level. But your link shows that it contains a tremendous amount of data. Edit: And I missed parents link to https://app.protomaps.com/downloads/small_map https://app.protomaps.com/downloads/small_map which is actually better.
divbzero 3y ago
Between this, Natural Earth, OpenStreetMap, USGS, and others, the availability of map data today would be stunning to early cartographers.
- antod 3y ago
  I tried my hand at pulling data from opentopology recently to use in blender and thought it was an amazing source of data for my country - I later realised that it seems like only the US and NZ are well represented there though (guess I was just lucky - thank you LINZ).
ChuckMcM 3y ago
Great stuff, at some point I feel like Google is going to have to try to extract even more revenue out of maps than they currently get and it will really empower these sorts of collaborative map systems. I wish there was a way for people to fund satellite imagery that got pushed into these systems after purchase. Sunnyvale, for example, paid for a lot of imagery of the city that they use/used in staff discussions about traffic, zoning, etc. It would be nice if they could then push those images into the open data set.
- Kadin 3y ago
  There's really no technical reason why imagery like that couldn't be contributed, and IIRC Google does use municipally-owned imagery in its products. A single city might not be a big enough source for them to bother, I don't know, but I know particularly in Europe I have seen copyright notices for national and regional governments when looking at imagery. Though there are a lot of sources for satellite imagery right now; Google may not be that hard-up for new stuff. I suspect the commercial vendors they work with probably image the entire CONUS area every 12 months or so. The imagery that would seem to be more in-demand would be the aerial photos used at very high zoom levels. I'm not a geospatial expert but I think these images are combined with some sort of LIDAR or multi-spectrum imagery to have height maps in tandem with the visuals. That strikes me as pretty expensive to obtain.
jijji 3y ago
I like that they provide this data, however when you try to actually retrieve it, It looks like they went out of their way to make it as convoluted as possible to try to retrieve any of the actual data.... so you have to use DuckDB and then do the import using that? why not just support mysql dump files? why require someone to have DuckDB? is DuckDB that popular? also the links they provide don't work so it doesn't look like any of it's available.... how is somebody supposed to use this stuff? they require you to also have Amazon S3 with some query language that's non-standard to be able to talk to it.... I don't get it I'm sure I'm not the only one but it needs to be more generic instead of the way they're doing it
- mdaniel 3y ago
  yeah, I thought about mentioning that it's kind of rude to lock that data behind AWS or Azure accounts, but realistically no one can be surprised that Big Cloud Providers want people to use more big cloud services There's nothing stopping either AWS or Azure from granting `s3:Get` and `s3:List` on those buckets to enable unauthenticated reads, if they were thus interested
  - mikeocool 3y ago
    I think they did? aws --no-sign-request s3 ls s3://overturemaps-us-west-2/release/2023-07-26-alpha.0/ works fine for me.
    - mdaniel 3y ago
      Ah, my mistake for not trying it; I saw the s3:// and assumed incorrectly. Thanks so much for trying that and reporting back!
dfgssdfgsdfg 3y ago
Is there any quicksand on the map? What about Lil Terry, is he on the map?
polemic 3y ago
It's unclear to me how Overture manages to "license wash" OSM data here. > Transportation: The OMF’s Transportation layer represents a worldwide road network derived from data in the OpenStreetMap project. This community-built data has been recast into the Overture data format which provides consistent segmentation of the data and a linear reference system to support additions of data such as speed limits or real-time traffic. The OSM ODbL is crystal clear that OSM contributors have to be credited. I don't believe that CDLA Permissive v2.0 magically allows Overture to bypass it. --EDIT: I missed that they're using different licenses per dataset, the transport theme is OBDL, which I'm sure will trip up users who are not careful.
- uneekname 3y ago
  Overture could be seen as hostile to OpenStreetMap, but I do not think they are washing any licenses here. The article explicitly says that the CDLA license applies to data provided by Meta and Microsoft, not data from OSM. I think it's unfortunate that Meta and Microsoft aren't contributing their data to OSM instead of releasing it under a different license, but c'est la vie.
- mikeyouse 3y ago
  Seems that any new data generated out of Overture Maps could be uploaded to OSM if they chose to do so? The CDLA license is permissive; https://cdla.dev/permissive-2-0/ https://cdla.dev/permissive-2-0/ My read is that they're not going to avoid credited OSM, but rather they're going to credit OSM / maintain the license for the parts they use from OSM and then the rest will be CDLA 2.0 licensed for anyone to use.
kozikow 3y ago
Anyone loaded it to Bigquery yet? I know this is "booo-google", but I just want to write some joins with other tables I happen to have in bigquery. I'm wondering if there is some "community" BQ rather than maintaining import of my own.
mike_d 3y ago
> Administrative Boundaries: A global open dataset of national and regional administrative boundaries, this boundary data includes regional names which have been translated into over 40 different languages to support international use. Oh that is going to be fun. If I recall correctly Google Maps alters the boundaries of places based on the views of the location the map is being requested from to avoid getting in the middle of disputes. Not "correctly" showing boundaries is a crime in many countries. Edit: here is a source https://qz.com/224821/see-how-borders-change-on-google-maps-depending-on-where-you-view-them https://qz.com/224821/see-how-borders-change-on-google-maps-...
- risyachka 3y ago
  Unless you are making a google maps rival, in 99% of cases you won't need those countries with controversial borders. And unless you have millions and millions of users, no one will care even in those countries.
deleted 3y ago
[deleted]
Mithridates1 3y ago
Does this add impedances to the OSM street segments?
zmix 3y ago
Will OSM incorporate this information into their own map?
anukul 3y ago
Has anyone been able to visually assess this dataset's accuracy against OSM / Google Maps etc for any given region? Looks like it will be a while before that can be done, seeing as it uses a custom schema.

402 points by chippy 3y ago

dhx 3y ago

I have an interest in this topic as a contributor to AllThePlaces[1], an open source project collating Scrapy spiders (MIT license) that crawl websites of franchises/retail chains that you'd find listed in name-suggestion-index[2] to retrieve location data (CC-0 license). The project is just short of collecting 3 million points of interest from almost 1700 spiders. Overture Maps appears to be quite a closed and proprietary project, with claims of openness limited to being able to download a data set and accompanying schema specification. Some issues that immediately come to mind: 1. There is no published description for how the data was generated. End users thus are given no assurance of how accurate and complete the data is. a. As an example, administrative boundaries are frightfully complex and include disputed boundaries, significant ambiguity in definition of boundaries, and trade-off between precision of boundaries versus performance of algorithms using administrative boundary data. Which definition of a boundary does Overture Maps adhere to, or can it support multiple definitions? b. It's probable that Microsoft have contributed ld+json/microdata geographic data from BingBot crawls of the Internet. This data is notoriously incorrect, including fields mixed up and invalidly repurposed, "CLOSED" in field names to denote closure of a place 5 years ago but the web page remains online, and much ambiguity in opening hours specifications. For AllThePlaces, many of the spiders developed require human consideration, sometimes of considerable complexity, to piece together horribly messy data that is published by shop and restaurant franchises, and other organisations providing location data via their websites. c. For location information where +/- 1-5m accuracy and precision may be required (e.g. individual shops within a shopping centre[3]), source data is typically provided by the authoritative sources with 1mm precision and +/- 10-100m accuracy. AllThePlaces, Overture Maps, Google Maps and similar still need human editors (OpenStreetMap editors) to do on-the-ground surveys to pinpoint precise locations and to standardise the definition of a location (e.g. for a point, should it be the centroid of the largest regular polygon which could be placed in the overall irregular polygon, the center of mass of a planar lamina, the location of the main entrance, or some other definition?). d. If Overture Maps is dependent on BingBot for place data, they'll miss an enormous number of points of interest that BingBot would never be able to find. For example, an undocumented REST/JSON/GraphQL API call or modification to parameters to an observed store locator API call may be necessary to return all locations and relevant fields of data. Website developers routinely do stupid things with robots.txt such as instruct a bot to crawl 10k pages (1GB+) from a sitemap last updated 5 years ago rather than make 10 fast API calls for up-to-date data (5MB). Overture Maps would be free to consume data from AllThePlaces as it is CC-0 licensed, and possibly correlate it with other data sources such as BingBot crawl data, a government database of licensed commercial premises or postal address geocoding data. However the messiness of data in various sources would be approaching impossible to reconcile, even for humans, and Overture Maps would possibly have to decide whether to err on the side of having duplicates, or lack completeness. 2. There is no published tooling for how someone else can reproduce the same data. a. AllThePlaces users fairly frequently experience the wrath of Cloudflare, Imperva and other Internet-breaking third parties, as well as custom geographic blocking schemes and more rarely, overzealous rate limiting mechanisms. If Overture Maps is dependent on BingBot crawls, they'll have a slight advantage over AllThePlaces due to deliberate whitelisting of BingBot from the likes of Cloudflare, Imperva, customer firewalls, etc. However, no matter whether you're AllThePlaces or Overture Maps or anyone else, if you want to capture as many points of interest as possible across the world, use of residential ISP subnets and anti-bot-detection software is increasingly required. They'll need people in dozens of countries each crawling websites targeted to the same country, using residential ISP address space. Otherwise they end up with an American view of the world, or a European view of the world, or something else that isn't the full picture. b. If Overture Maps has locations incorrect for a franchise/brand due to a data cleansing problem or sourcing data from a bad source (perhaps non-authoritative), there are no software repositories for the franchise/brand to raise an issue or submit a patch against. [1] https://www.alltheplaces.xyz/ https://www.alltheplaces.xyz/ [2] http://nsi.guide/ http://nsi.guide/ [3] Example Australian shopping centre as captured by AllThePlaces: https://www.alltheplaces.xyz/map/#18.07/-33.834646/150.989528 https://www.alltheplaces.xyz/map/#18.07/-33.834646/150.98952...

mistrial9 3y ago
strong agree -- additionally, there is clear public relations going on, announcing maps like this as "trusted" or "official" backing, presumably for decisions over commerce, safety, insurance claims or other non-security but valuable, data-oriented activity. The public or public oversight, has what recourse here?
tony_cannistra 3y ago
> 2. There is no published tooling for how someone else can reproduce the same data. This is true, and is actually the whole point of Overture. Overture was developed to enable private companies to leverage open data (like OpenStreetMap) but also combine it with their proprietary data and processes. The intention is to share the result with a relatively permissive license (a new thing called the Community Database License Agreement) but keep the process and underlying data proprietary.
- justinclift 3y ago
  Sounds like that's going to be a problem for proving/reproducing results independently then. :(
- uneekname 3y ago
  Yes, exactly. Overture is, for better or worse, looking to create a dataset that is more difficult to mess with / contribute to. I'm big on OpenStreetMap, but I can't deny its a bit of a liability for Facebook and other companies that display maps to their users. There is the occasional vandalism edit that simply can't be shown to an end-user. Facebook put significant effort into maintaining a moderated version of the OSM database that lags behind the real-time edits. Facebook, Microsoft, TomTom, etc. know this is a ton of work and want to pool their resources. Making it open also helps to openly compete with Google, the other big map data provider. If you want to contribute to Overture as an end-user, AFAICT your best option is to edit OpenStreetMap and see if your changes eventually get pulled in. Overture has promised the OSM community that they'll make much of their data available to be contributed back, we'll see if that pans out. When it comes to AllThePlaces -- as an OSM nerd it seems like there is an opportunity to build a better bridge between this and OpenStreetMap, to make it easier to quickly update businesses in an area. Recently there has been a pretty successful push to link OSM data with WikiData, using tools like the OSM ↔ Wikidata matcher [0]. For POIs, it's a lot of work to add a bunch of local businesses, even with tools like EveryDoor [1]. It would be so cool to see AllThePlaces integration into RapID for example, if there isn't already(?) [0] https://osm.wikidata.link/ https://osm.wikidata.link/ [1] https://every-door.app/ https://every-door.app/

pininja 3y ago

POIs, buildings, transportation network, and admin boundaries layers.. on my!

sdfghswe 3y ago

What's the difference between this and Open Street Maps?

pininja 3y ago
To start, the admin boundaries and POIs are given a more permissive license. They’re mission is to make permissive map datasets.
mikeyouse 3y ago
Ha I was just wondering the same and Overture has a FAQ that answers that exact question; Overture is a data-centric map project, not a community of individual map editors. Therefore, Overture is intended to be complementary to OSM. We combine OSM with other sources to produce new open map data sets. Overture data will be available for use by the OpenStreetMap community under compatible open data licenses. Overture members are encouraged to contribute to OSM directly.
- zigzag312 3y ago
  We've focused on designing a data schema that is easy for developers to quickly understand and use in building map products. https://docs.overturemaps.org/ https://docs.overturemaps.org/ OSM focuses on open map editing, but due to it's flexible schema, it can be hard to extract needed information from OSM. Overture seems to focus on providing map data (from multiple sources) that can be used more easily. EDIT: OSM is also trying to improve the OSM data model for easier processing. https://github.com/osmlab/osm-data-model https://github.com/osmlab/osm-data-model
matkoniecz 3y ago
OpenStreetMap is community of people creating fully open map database. From roads, shops and rivers to tourism attractions, hiking routes and hospitals, with all kind of detail. Anyone can contribute (and as someone quite involved in OSM: if you want to map something, especially area near you: you are welcome!). Overture is a new group of companies releasing some datasets on open licenses, but methods used to create them remain proprietary. Some of released data is theirs, many datasets are repackaged OpenStreetMap data.

mikeyouse 3y ago

The GitHub repo is here; https://github.com/OvertureMaps/data https://github.com/OvertureMaps/data -- Licenses look super permissive - I'm not that familiar with the state of Open Mapping but the GERS idea looks great too. https://docs.overturemaps.org/gers/ https://docs.overturemaps.org/gers/

orra 3y ago
The CDLA license feels like NIH... but at least it looks genuinely permissive, as opposed to fake open source. Curiously, two of the four layers use the ODbL.
- matkoniecz 3y ago
  > Curiously, two of the four layers use the ODbL. Probably this data is reformatted OpenStreetMap data or derived from it (so is licensed as ODBL and requires attributing OpenStreetMap)

tony_cannistra 3y ago

For context, Overture Maps is a project that intends to enable large players in the geospatial space (TomTom, Amazon, Microsoft, but notably _not_ Google) to leverage open data sets (OpenStreetMap among them) alongside proprietary data and processes that they own. The consortium intends to enable a framework for enhancing geospatial data based in open data sets (e.g. OSM) with their own proprietary processes, and re-release it with a permissive license (the Community Database License Agreement - CDLAv2), but keep the data and processes required to create that dataset proprietary. The project has created a lot of conversation in the OpenStreetMap community, but in general I think it's good to see so many resources put into the OSM-adjacent world.

gumby 3y ago
I came to ask for this info so thanks for the succinct summary!
alexchamberlain 3y ago
How do they do that? I thought OSM was licensed in such a way that if you combine it with other data, the whole thing needed to be released with the same license?
- tony_cannistra 3y ago
  You are mostly right, except that in addition to the same license, the Share-Alike provision in the OSM license (the ODbL [0]) also enables derivatives to be re-released with "A compatible license." I am not a lawyer, but I would assume someone at Overture has deemed their license (the CDLAv2 [1]) to be "compatible" with ODbL. (edit: From the link in OP, the OSM-based data is being released with ODbL. I think I'm wrong about license compatibility. Again, not a lawyer.). [0]: https://opendatacommons.org/licenses/odbl/1-0/ https://opendatacommons.org/licenses/odbl/1-0/ [1]: https://cdla.dev/permissive-2-0/ https://cdla.dev/permissive-2-0/
  - Doctor_Fegg 3y ago
    CDLA Permissive is not downstream compatible with ODbL. ODbL is a sharealike licence, CDLA Permissive (as the name suggests) isn’t. Overture can’t relicense OSM-derived content under CDLA Permissive. From a brief skim of the landing page, it looks like the OSM-derived content is not being offered permissively. See https://overturemaps.org/download/ https://overturemaps.org/download/.

deleted 3y ago

[deleted]

joshstrange 3y ago

If you are looking for maps themselves (a la Google Maps) I highly recommend checking out Protomaps [0]. It provides a single file (PMTile format) that contains all the map data and you can request ranges from the single file to get the required vector data you need for a given area/zoom. So you can setup something like the following: S3 (host the myfile.pmtiles) <- Lambda (takes the x/y/z from the path and requests the correct range) <- Cloudfront cache tile response Then you can setup tiles.mydomain.com (or use the cloudfront domain directly) and then use Leaflet or similar on the frontend to fetch/render the tiles. For Leaflet you use the protomaps plugin/lib and give it a url like "https://tiles.yourdomain.com/20230408/{z}/{x}/{y}.mvt https://tiles.yourdomain.com/20230408/{z}/{x}/{y}.mvt" where "20230408" maps to "20230408.pmtiles" in your S3 bucket. Now I can drop new pmtiles files into that bucket and update my clients to use the new source. And since the tiles are in vector format you can theme them however you want in the client which is neat. Lastly you don't have to use the 100+GB whole-earth tileset. You can use a tool [1] (provided by the same guy) to download a dataset for just a given geographical region. The .pmtiles file is a little over 100GB but the whole setup took me only an hour or two max to get running and will cost way less than Google Maps to run. [0] https://protomaps.com/ https://protomaps.com/ [1] https://app.protomaps.com/downloads/small_map https://app.protomaps.com/downloads/small_map

mdaniel 3y ago
their demo isn't doing them any favors since it seems to stop at some random zoomlevel, eliding the kind of detail that I'd care about if I were evaluating that solution digging into their GH repos surfaces https://protomaps.github.io/PMTiles/?url=https%3A%2F%2Fr2-public.protomaps.com%2Fprotomaps-sample-datasets%2Fprotomaps-basemap-opensource-20230408.pmtiles https://protomaps.github.io/PMTiles/?url=https%3A%2F%2Fr2-pu... which is at least more detailed if not quite a GMaps killer
- qwertox 3y ago
  I agree, if it weren't for your comment, I was about to ditch it as a candidate because it stopped zooming at a bad level. But your link shows that it contains a tremendous amount of data. Edit: And I missed parents link to https://app.protomaps.com/downloads/small_map https://app.protomaps.com/downloads/small_map which is actually better.

divbzero 3y ago

Between this, Natural Earth, OpenStreetMap, USGS, and others, the availability of map data today would be stunning to early cartographers.

antod 3y ago
I tried my hand at pulling data from opentopology recently to use in blender and thought it was an amazing source of data for my country - I later realised that it seems like only the US and NZ are well represented there though (guess I was just lucky - thank you LINZ).

ChuckMcM 3y ago

Great stuff, at some point I feel like Google is going to have to try to extract even more revenue out of maps than they currently get and it will really empower these sorts of collaborative map systems. I wish there was a way for people to fund satellite imagery that got pushed into these systems after purchase. Sunnyvale, for example, paid for a lot of imagery of the city that they use/used in staff discussions about traffic, zoning, etc. It would be nice if they could then push those images into the open data set.

Kadin 3y ago
There's really no technical reason why imagery like that couldn't be contributed, and IIRC Google does use municipally-owned imagery in its products. A single city might not be a big enough source for them to bother, I don't know, but I know particularly in Europe I have seen copyright notices for national and regional governments when looking at imagery. Though there are a lot of sources for satellite imagery right now; Google may not be that hard-up for new stuff. I suspect the commercial vendors they work with probably image the entire CONUS area every 12 months or so. The imagery that would seem to be more in-demand would be the aerial photos used at very high zoom levels. I'm not a geospatial expert but I think these images are combined with some sort of LIDAR or multi-spectrum imagery to have height maps in tandem with the visuals. That strikes me as pretty expensive to obtain.

jijji 3y ago

I like that they provide this data, however when you try to actually retrieve it, It looks like they went out of their way to make it as convoluted as possible to try to retrieve any of the actual data.... so you have to use DuckDB and then do the import using that? why not just support mysql dump files? why require someone to have DuckDB? is DuckDB that popular? also the links they provide don't work so it doesn't look like any of it's available.... how is somebody supposed to use this stuff? they require you to also have Amazon S3 with some query language that's non-standard to be able to talk to it.... I don't get it I'm sure I'm not the only one but it needs to be more generic instead of the way they're doing it

mdaniel 3y ago
yeah, I thought about mentioning that it's kind of rude to lock that data behind AWS or Azure accounts, but realistically no one can be surprised that Big Cloud Providers want people to use more big cloud services There's nothing stopping either AWS or Azure from granting `s3:Get` and `s3:List` on those buckets to enable unauthenticated reads, if they were thus interested
- mikeocool 3y ago
  I think they did? aws --no-sign-request s3 ls s3://overturemaps-us-west-2/release/2023-07-26-alpha.0/ works fine for me.
  - mdaniel 3y ago
    Ah, my mistake for not trying it; I saw the s3:// and assumed incorrectly. Thanks so much for trying that and reporting back!

dfgssdfgsdfg 3y ago

Is there any quicksand on the map? What about Lil Terry, is he on the map?

polemic 3y ago

It's unclear to me how Overture manages to "license wash" OSM data here. > Transportation: The OMF’s Transportation layer represents a worldwide road network derived from data in the OpenStreetMap project. This community-built data has been recast into the Overture data format which provides consistent segmentation of the data and a linear reference system to support additions of data such as speed limits or real-time traffic. The OSM ODbL is crystal clear that OSM contributors have to be credited. I don't believe that CDLA Permissive v2.0 magically allows Overture to bypass it. --EDIT: I missed that they're using different licenses per dataset, the transport theme is OBDL, which I'm sure will trip up users who are not careful.

uneekname 3y ago
Overture could be seen as hostile to OpenStreetMap, but I do not think they are washing any licenses here. The article explicitly says that the CDLA license applies to data provided by Meta and Microsoft, not data from OSM. I think it's unfortunate that Meta and Microsoft aren't contributing their data to OSM instead of releasing it under a different license, but c'est la vie.
mikeyouse 3y ago
Seems that any new data generated out of Overture Maps could be uploaded to OSM if they chose to do so? The CDLA license is permissive; https://cdla.dev/permissive-2-0/ https://cdla.dev/permissive-2-0/ My read is that they're not going to avoid credited OSM, but rather they're going to credit OSM / maintain the license for the parts they use from OSM and then the rest will be CDLA 2.0 licensed for anyone to use.

kozikow 3y ago

Anyone loaded it to Bigquery yet? I know this is "booo-google", but I just want to write some joins with other tables I happen to have in bigquery. I'm wondering if there is some "community" BQ rather than maintaining import of my own.

mike_d 3y ago

> Administrative Boundaries: A global open dataset of national and regional administrative boundaries, this boundary data includes regional names which have been translated into over 40 different languages to support international use. Oh that is going to be fun. If I recall correctly Google Maps alters the boundaries of places based on the views of the location the map is being requested from to avoid getting in the middle of disputes. Not "correctly" showing boundaries is a crime in many countries. Edit: here is a source https://qz.com/224821/see-how-borders-change-on-google-maps-depending-on-where-you-view-them https://qz.com/224821/see-how-borders-change-on-google-maps-...

risyachka 3y ago
Unless you are making a google maps rival, in 99% of cases you won't need those countries with controversial borders. And unless you have millions and millions of users, no one will care even in those countries.

deleted 3y ago

[deleted]

Mithridates1 3y ago

Does this add impedances to the OSM street segments?

zmix 3y ago

Will OSM incorporate this information into their own map?

anukul 3y ago

Has anyone been able to visually assess this dataset's accuracy against OSM / Google Maps etc for any given region? Looks like it will be a while before that can be done, seeing as it uses a custom schema.