3 ms·
> when Yelp and all the rest of the local businesses directories realized they were being scraped and the data was being used to google's advantage, http://yel
by SeanLuke 16y ago
> when Yelp and all the rest of the local businesses directories realized they were being scraped and the data was being used to google's advantage,
http://yelp.com/robots.txt http://yelp.com/robots.txt
Not a mention of GoogleBot.
- ddemchuk 16y agowell, they're presented with either 1) killing their organic search traffic by blocking Google or 2) allowing the crawlers but not being able to stop their data from being aggregated and included in local results. Yelp's API specifically says their data cannot be aggregated with other sources. When presented with that, Google simply said they weren't using the API, they were scraping. Apparently scraping is a free pass to do whatever the hell you want if you're Google.