6 ms·

I really wish Google would offer better lower level API's, and stop trying to do high level API's like "solar potential" and "pollen prediction". I basically j

I really wish Google would offer better lower level API's, and stop trying to do high level API's like "solar potential" and "pollen prediction". I basically just want a data dump of their whole map, like OSM provides, but I understand why they don't want to do that... So instead give me access to the whole maps database to run bigquery queries over it, and make me pay per record I touch or record in the result set. I'd love to be able to answer questions like "How far, on average, are my customers home addresses from the nearest footpath?", or "What percentage of residential addresses in the UK have an ATM within 1 mile", or "Give me a route from A to B, but not via any unpaved streets", or "Give me a list of the 100 biggest cities that have no ice rinks".

pseg134 3y ago
Yeah and then someone would just export all of the data if they offered low level access. I can’t think of any reason google would take that risk for minimal API fees.
- londons_explore 3y ago
  For most of the data, it's already possible to siphon it off. Just browse with a instrumented browser and you can download all the 3D map data for a city in a few minutes from a single session, simply emulating a user zooming around the city. Business data can likewise be methodologically downloaded, thousands of businesses per user account per day doesn't hit their rate limits. The polygons of roads and stuff are also easily extracted from a browser. And the Android/iOS mobile apps allow you to 'download for offline use', which is obfusticated but not properly encrypted. Charging $1 per 1000 records for API access would cost massive amounts for a competitor to download all the data - and besides, maps data is fairly easy to hide a few erroneous entries in and catch anyone using the data to start a clone of Google Maps.
  - mtmail 3y ago
    > Business data can likewise be methodologically downloaded Google doesn't allow to store the data permanently. Just because it's technically possible doesn't make it legal. From the Google Places API terms "you must not pre-fetch, index, store, or cache any Content except under the limited conditions stated in the terms" "You can display Places API results on a Google Map, or without a map. If you want to display Places API results on a map, then these results must be displayed on a Google Map. It is prohibited to use Places API data on a map that is not a Google map."
    - londons_explore 3y ago
      I was thinking of users who just wanted to take all the data and ignore any T&C's. Since the Linkedin web-scraping case, T&C's on data on a public website effectively doesn't apply anyway.
      - dontupvoteme 3y ago
        They actually ended up losing that case, but they went to extreme lengths such as using proxies after getting banned and such. IIRC the important thing was that horrible CFAA law can't be brought into consideration because the supreme court shot that down.
cheschire 3y ago
I wonder if it's because there isn't that much map data to be had, not like search data, so no matter what price they set their competitors could siphon off all their data for relatively cheap. And sure, a user agreement is a way to litigate after the fact (assuming you catch them in the act) but once they have the data it doesn't matter.
- londons_explore 3y ago
  I suspect the reason they don't do this is because they want more pricing control. For their solar API for example, they know how much time solar surveyors spend getting roof profiles, and they can therefore price their product to maximize revenue. However if they just let you access all the data for any usecase, there will be plenty of people who pay Google a few bucks for data that might cost millions of dollars to obtain any other way.
cornedor 3y ago
I don't think mapping data from Google Maps is accurate enough to do this. Perhaps you're better off using a service that uses OSM data. But it depends on the region and use case.
- KeplerBoy 3y ago
  Does OSM really have more information than google maps? Surely google has a lot more stuff than what the webapp shows.
  - bombolo 3y ago
    Depends on the area… for example in my area OSM has all the mountain paths that completely do not exist on google maps.
  - matkoniecz 3y ago
    Depends on area and information type.
danpalmer 3y ago
I get why this feels like a good idea, and I’ve wanted similar myself, but having built APIs, ensuring that users use them correctly is hard. Designing APIs that are hard to use wrongly is tricky, but a much better approach than documentation. There are lots of other reasons and it’s easy to jump to Google wanting to protect their data, but creating correctly usable APIs is a generally applicable reason that is perhaps more charitable.
- londons_explore 3y ago
  Whats wrong with "all maps stuff is in this huge bigquery table. Here are a bunch of example queries to get you going." SELECT business_name, business_phone FROM pois WHERE business_type = 'bank' and place_review_count > 10 ORDER BY distance(Location('New York'), location) ASC LIMIT 10; Doesn't seem particularly hard for users to use 'right' - most people using an API will have some technical background and be able to use that.
  - danpalmer 3y ago
    Example, purely hypothetical for this scenario, but based on a real experience I had recently... What if there was a data source that went into this from a third party? What if that third party had requirements about the timeline of removing data? This could be because it contained user-generated content and something was found to be offensive or even illegal, or it could be because it contained something proprietary that needed to be correctly licenced, all sorts of possibilities. If you provide a data dump, you then need any integrators to add a data refresh process. That's an additional step that many won't do. Now you can't honour contracts or potentially legal requirements. Sure you can document this, sure you can put it in contracts, but that may not be enough.
  - amichal 3y ago
    "map stuff" is much harder to understand than your query implies. As an example Have you tried understanding all the possible things that you can get in 'address_components' depending on the input params, the region you are querying from, the region you results are coming from, the geopolitical situation on that data, the entity that was matched etc etc https://developers.google.com/maps/documentation/geocoding/requests-geocoding https://developers.google.com/maps/documentation/geocoding/r... https://developers.google.com/maps/documentation/geocoding/requests-geocoding#Types https://developers.google.com/maps/documentation/geocoding/r... Don't forget entities in side of other entities like businesses within a mall. Don't forget that the user might want a specific service from the bank. The "bank" label doesn't tell you if they accept street traffic, have live tellers, are just an ATM or a corporate office etc etc.
nycdatasci 3y ago
This is pretty close: https://overturemaps.org/ https://overturemaps.org/
xinayder 3y ago
The answer is in your post. Use OpenStreetMap. It's free as long as you respect the dataset license and you can do a lot of things with it. If you use OSM, please consider contributing back by either spreading word or adding more data to the map itself.
- yellow_lead 3y ago
  OSM is great but it cannot answer those questions accurately.
  - Ajedi32 3y ago
    True, but that's kind of a chicken-egg problem. If OSM were more popular the resulting influx of contributors and other resources would solve the data quality problem pretty quickly.
  - dvdkon 3y ago
    Neither can Google maps. Many countries have official maps, some are even open data, but even those don't have perfect coverage of "unofficial" features like footpaths. Here in the Czech Republic, a combination of OSM and official data (ZABAGED) is the best bet, somewhere else it could be Google's data, if they made it available.
    - yellow_lead 3y ago
      It depends where you're looking at. In US, I think google would do pretty well - better than OSM, for I.e this ATM question. Obviously, no maps are always 100% accurate.
- dheera 3y ago
  > adding more data to the map itself I'd love to but it would eat into my work time and I'd get a bad performance review.
blamazon 3y ago
Check out overpass turbo if you haven't already: https://overpass-turbo.eu/ https://overpass-turbo.eu/
punnerud 3y ago
You can convert OSM into Sqlite3 and query on it. I build an example with SQLite to convert from coordinates to address. Then I had to find closest addresses up to a limit. https://github.com/punnerud/rgcosm https://github.com/punnerud/rgcosm Some some also build on it with parameters: https://github.com/BlackCatDevel0per/rgcosm https://github.com/BlackCatDevel0per/rgcosm
karussell 3y ago
> Give me a route from A to B, but not via any unpaved streets With GraphHopper you can customize routes in many different ways and e.g. prefer paved roads: https://graphhopper.com/maps/?point=50.963703%2C13.777981&point=50.949749%2C13.795166&profile=foot&layer=TF+Outdoors&custom_model=%7B%22priority%22%3A%5B%7B%22if%22%3A%22surface%21%3DPAVED%26%26surface%21%3DASPHALT%26%26surface%21%3DMISSING%22%2C%22multiply_by%22%3A%220.5%22%7D%5D%7D https://graphhopper.com/maps/?point=50.963703%2C13.777981&po... Click on the gear button in the top left to see, edit and disable the customization. Also the Isochrone or Shortest Path Tree (spt) API can be used to answer "How far, on average, are my customers home addresses from the nearest footpath?" or "What percentage of residential addresses in the UK have an ATM within 1 mile" (fetch the ATM locations e.g. via overpass or a reverse Geocoding API) Note, that I'm one of the founders of GraphHopper.
- petre 3y ago
  Graphhopper is great. Thank you.
deleted 3y ago
[deleted]

pseg134 3y ago

Yeah and then someone would just export all of the data if they offered low level access. I can’t think of any reason google would take that risk for minimal API fees.

londons_explore 3y ago
For most of the data, it's already possible to siphon it off. Just browse with a instrumented browser and you can download all the 3D map data for a city in a few minutes from a single session, simply emulating a user zooming around the city. Business data can likewise be methodologically downloaded, thousands of businesses per user account per day doesn't hit their rate limits. The polygons of roads and stuff are also easily extracted from a browser. And the Android/iOS mobile apps allow you to 'download for offline use', which is obfusticated but not properly encrypted. Charging $1 per 1000 records for API access would cost massive amounts for a competitor to download all the data - and besides, maps data is fairly easy to hide a few erroneous entries in and catch anyone using the data to start a clone of Google Maps.
- mtmail 3y ago
  > Business data can likewise be methodologically downloaded Google doesn't allow to store the data permanently. Just because it's technically possible doesn't make it legal. From the Google Places API terms "you must not pre-fetch, index, store, or cache any Content except under the limited conditions stated in the terms" "You can display Places API results on a Google Map, or without a map. If you want to display Places API results on a map, then these results must be displayed on a Google Map. It is prohibited to use Places API data on a map that is not a Google map."
  - londons_explore 3y ago
    I was thinking of users who just wanted to take all the data and ignore any T&C's. Since the Linkedin web-scraping case, T&C's on data on a public website effectively doesn't apply anyway.
    - dontupvoteme 3y ago
      They actually ended up losing that case, but they went to extreme lengths such as using proxies after getting banned and such. IIRC the important thing was that horrible CFAA law can't be brought into consideration because the supreme court shot that down.

cheschire 3y ago

I wonder if it's because there isn't that much map data to be had, not like search data, so no matter what price they set their competitors could siphon off all their data for relatively cheap. And sure, a user agreement is a way to litigate after the fact (assuming you catch them in the act) but once they have the data it doesn't matter.

londons_explore 3y ago
I suspect the reason they don't do this is because they want more pricing control. For their solar API for example, they know how much time solar surveyors spend getting roof profiles, and they can therefore price their product to maximize revenue. However if they just let you access all the data for any usecase, there will be plenty of people who pay Google a few bucks for data that might cost millions of dollars to obtain any other way.

cornedor 3y ago

I don't think mapping data from Google Maps is accurate enough to do this. Perhaps you're better off using a service that uses OSM data. But it depends on the region and use case.

KeplerBoy 3y ago
Does OSM really have more information than google maps? Surely google has a lot more stuff than what the webapp shows.
- bombolo 3y ago
  Depends on the area… for example in my area OSM has all the mountain paths that completely do not exist on google maps.
- matkoniecz 3y ago
  Depends on area and information type.

danpalmer 3y ago

I get why this feels like a good idea, and I’ve wanted similar myself, but having built APIs, ensuring that users use them correctly is hard. Designing APIs that are hard to use wrongly is tricky, but a much better approach than documentation. There are lots of other reasons and it’s easy to jump to Google wanting to protect their data, but creating correctly usable APIs is a generally applicable reason that is perhaps more charitable.

londons_explore 3y ago
Whats wrong with "all maps stuff is in this huge bigquery table. Here are a bunch of example queries to get you going." SELECT business_name, business_phone FROM pois WHERE business_type = 'bank' and place_review_count > 10 ORDER BY distance(Location('New York'), location) ASC LIMIT 10; Doesn't seem particularly hard for users to use 'right' - most people using an API will have some technical background and be able to use that.
- danpalmer 3y ago
  Example, purely hypothetical for this scenario, but based on a real experience I had recently... What if there was a data source that went into this from a third party? What if that third party had requirements about the timeline of removing data? This could be because it contained user-generated content and something was found to be offensive or even illegal, or it could be because it contained something proprietary that needed to be correctly licenced, all sorts of possibilities. If you provide a data dump, you then need any integrators to add a data refresh process. That's an additional step that many won't do. Now you can't honour contracts or potentially legal requirements. Sure you can document this, sure you can put it in contracts, but that may not be enough.
- amichal 3y ago
  "map stuff" is much harder to understand than your query implies. As an example Have you tried understanding all the possible things that you can get in 'address_components' depending on the input params, the region you are querying from, the region you results are coming from, the geopolitical situation on that data, the entity that was matched etc etc https://developers.google.com/maps/documentation/geocoding/requests-geocoding https://developers.google.com/maps/documentation/geocoding/r... https://developers.google.com/maps/documentation/geocoding/requests-geocoding#Types https://developers.google.com/maps/documentation/geocoding/r... Don't forget entities in side of other entities like businesses within a mall. Don't forget that the user might want a specific service from the bank. The "bank" label doesn't tell you if they accept street traffic, have live tellers, are just an ATM or a corporate office etc etc.

nycdatasci 3y ago

This is pretty close: https://overturemaps.org/ https://overturemaps.org/

xinayder 3y ago

The answer is in your post. Use OpenStreetMap. It's free as long as you respect the dataset license and you can do a lot of things with it. If you use OSM, please consider contributing back by either spreading word or adding more data to the map itself.

yellow_lead 3y ago
OSM is great but it cannot answer those questions accurately.
- Ajedi32 3y ago
  True, but that's kind of a chicken-egg problem. If OSM were more popular the resulting influx of contributors and other resources would solve the data quality problem pretty quickly.
- dvdkon 3y ago
  Neither can Google maps. Many countries have official maps, some are even open data, but even those don't have perfect coverage of "unofficial" features like footpaths. Here in the Czech Republic, a combination of OSM and official data (ZABAGED) is the best bet, somewhere else it could be Google's data, if they made it available.
  - yellow_lead 3y ago
    It depends where you're looking at. In US, I think google would do pretty well - better than OSM, for I.e this ATM question. Obviously, no maps are always 100% accurate.
dheera 3y ago
> adding more data to the map itself I'd love to but it would eat into my work time and I'd get a bad performance review.

blamazon 3y ago

Check out overpass turbo if you haven't already: https://overpass-turbo.eu/ https://overpass-turbo.eu/

punnerud 3y ago

You can convert OSM into Sqlite3 and query on it. I build an example with SQLite to convert from coordinates to address. Then I had to find closest addresses up to a limit. https://github.com/punnerud/rgcosm https://github.com/punnerud/rgcosm Some some also build on it with parameters: https://github.com/BlackCatDevel0per/rgcosm https://github.com/BlackCatDevel0per/rgcosm

karussell 3y ago

> Give me a route from A to B, but not via any unpaved streets With GraphHopper you can customize routes in many different ways and e.g. prefer paved roads: https://graphhopper.com/maps/?point=50.963703%2C13.777981&point=50.949749%2C13.795166&profile=foot&layer=TF+Outdoors&custom_model=%7B%22priority%22%3A%5B%7B%22if%22%3A%22surface%21%3DPAVED%26%26surface%21%3DASPHALT%26%26surface%21%3DMISSING%22%2C%22multiply_by%22%3A%220.5%22%7D%5D%7D https://graphhopper.com/maps/?point=50.963703%2C13.777981&po... Click on the gear button in the top left to see, edit and disable the customization. Also the Isochrone or Shortest Path Tree (spt) API can be used to answer "How far, on average, are my customers home addresses from the nearest footpath?" or "What percentage of residential addresses in the UK have an ATM within 1 mile" (fetch the ATM locations e.g. via overpass or a reverse Geocoding API) Note, that I'm one of the founders of GraphHopper.

petre 3y ago
Graphhopper is great. Thank you.

deleted 3y ago

[deleted]