4 ms·

I scraped google API exactly like this in grad school, but sometimes when you're building, you take a calculated risk to build something better. For my thesis,

I scraped google API exactly like this in grad school, but sometimes when you're building, you take a calculated risk to build something better. For my thesis, I was trying to load two Street View photos side by side in a browser to compare people's perceptions side-by-side. Google maps at the time required you to load a javascript viewer for each image you requested. Think Hot-or-Not for cities. The experience needed to instantly load a new image after the user voted because I knew they were only going to be on the site for maybe 30 seconds before they got bored and went back to reddit. I needed to collect as many votes as possible within that time period. So I knowingly broke Google's ToS and prefetched the images on my server so I could provide the user experience I wanted. "I'm a small operation. Surely they won't know." I wrote a server side screen scraper to load the Street View images and exposed the scraped images with an API. Now my site was faaaast. I could load Street View images instantly and in the end got over a million data points doing this. But then one day soon after it stopped working. Then I got a cease and desist email from someone at google legal. They didn't respond to any requests for turning it back on or even to discuss. Radio silence. That was terrifying. Since this was my thesis, I needed help getting my keys turned back on. Google in the end was very accommodating, but only after I used my nuclear option: asking lab director Joi Ito to bug Megan Smith while she was still at Google to help. I was connected to some engineer and told them what I was doing and why. They said stop. But then a week or so later, they sent me a beta invite to their new Street View images api, where you can feed in a lat, lng, header to a query string and they'll just serve the image now. Pretty cool. Offending site for the interested: http://pulse.media.mit.edu/ http://pulse.media.mit.edu/

kbenson 8y ago
> "I'm a small operation. Surely they won't know." I wrote a server side screen scraper to load the Street View images and exposed the scraped images with an API. I find the trick, when you're worried about this, is to use the regular API normally, but save the data that comes in from the normal usage of the API. i.e. Use the Google maps viewer and after the image is loaded, grab it however you have to, and post it to another API you've created that allows you to save the image. You're scraping their site, but you're not doing it in an automated way, and it should be undetectable. After a while, you've got a good library of images from normal use. So you code up a switch that you can toggle that changes it from loading from the Google maps viewer to using your API to get images. If you want to grow your image library, randomly assign some percentage of people to using the Google viewer (and save them they download), and the rest to your library of images you've accumulated. Or use a cookie or JS localStorage variable to track whether they are a returning person, and the first time always give them the quick library version, and if they return give them the Google maps viewer version (or just switch the percentages from 90/10 to 10/90, etc). If they're willing to give you the data free within their ToS, there's very little technologically they can do to stop you from easily (or moderately easily, in the harder cases) storing the data. Worst case for someone looking to save it would be if they generate an image for the content and just serve the image, and that's not that hard to work around either, if the data is structured.
- Retric 8y ago
  Somone handing you data does not remove any copyrights on this data. Which is where you can get in real trouble with your approach. That said, this kind of thing is detectable. If users of an application are far less likely to download common data it quickly looks odd. And while that’s not conclusive, they can just look at how your application functions to see what’s going on.
  - kbenson 8y ago
    > Somone handing you data does not remove any copyrights on this data. Oh, I'm not making any claim that it's legal. I'm just noting that if you've decided you want to scrape and are disregarding the ToS, there are ways to make it less likely to get you blocked. > That said, this kind of thing is detectable. If users of an application are far less likely to download common data it quickly looks odd. In the approach I outlined, you either load the Google JS payload and use it entirely as normal (and just do something extra with the data it provides), or you don't load it at all and run entirely locally. There are things they can do, such as embed analytical code in their payload to test for certain things, but it's just a cat and mouse game at that point. > And while that’s not conclusive, they can just look at how your application functions to see what’s going on. Assuming it's a public application (in this case it is), and that they have reason to look at it. If it's just spiky load, where sometimes there is load and other times there isn't or it's less, that's not really indicative of something odd going on, especially if you're relatively small.
tactwork 8y ago
You're story is absolutely amazing...and relateable... Also love your app. Will be sharing that.
- deleted 8y ago
  [deleted]

kbenson 8y ago

> "I'm a small operation. Surely they won't know." I wrote a server side screen scraper to load the Street View images and exposed the scraped images with an API. I find the trick, when you're worried about this, is to use the regular API normally, but save the data that comes in from the normal usage of the API. i.e. Use the Google maps viewer and after the image is loaded, grab it however you have to, and post it to another API you've created that allows you to save the image. You're scraping their site, but you're not doing it in an automated way, and it should be undetectable. After a while, you've got a good library of images from normal use. So you code up a switch that you can toggle that changes it from loading from the Google maps viewer to using your API to get images. If you want to grow your image library, randomly assign some percentage of people to using the Google viewer (and save them they download), and the rest to your library of images you've accumulated. Or use a cookie or JS localStorage variable to track whether they are a returning person, and the first time always give them the quick library version, and if they return give them the Google maps viewer version (or just switch the percentages from 90/10 to 10/90, etc). If they're willing to give you the data free within their ToS, there's very little technologically they can do to stop you from easily (or moderately easily, in the harder cases) storing the data. Worst case for someone looking to save it would be if they generate an image for the content and just serve the image, and that's not that hard to work around either, if the data is structured.

Retric 8y ago
Somone handing you data does not remove any copyrights on this data. Which is where you can get in real trouble with your approach. That said, this kind of thing is detectable. If users of an application are far less likely to download common data it quickly looks odd. And while that’s not conclusive, they can just look at how your application functions to see what’s going on.
- kbenson 8y ago
  > Somone handing you data does not remove any copyrights on this data. Oh, I'm not making any claim that it's legal. I'm just noting that if you've decided you want to scrape and are disregarding the ToS, there are ways to make it less likely to get you blocked. > That said, this kind of thing is detectable. If users of an application are far less likely to download common data it quickly looks odd. In the approach I outlined, you either load the Google JS payload and use it entirely as normal (and just do something extra with the data it provides), or you don't load it at all and run entirely locally. There are things they can do, such as embed analytical code in their payload to test for certain things, but it's just a cat and mouse game at that point. > And while that’s not conclusive, they can just look at how your application functions to see what’s going on. Assuming it's a public application (in this case it is), and that they have reason to look at it. If it's just spiky load, where sometimes there is load and other times there isn't or it's less, that's not really indicative of something odd going on, especially if you're relatively small.

tactwork 8y ago

You're story is absolutely amazing...and relateable... Also love your app. Will be sharing that.

deleted 8y ago
[deleted]