3 ms·

google at this point is "phonebook" of the internet. it is understandable that you want to be in phonebook but also that you dont want somebody random to enter

google at this point is "phonebook" of the internet. it is understandable that you want to be in phonebook but also that you dont want somebody random to enter, use your resources and copy everything because of some other interests. if there would be some official list of all "phonebooks" and you would allow those to scan your site but block others i would do it. scrapers from my side, spam my logs/analytics, fake real visits, try to spend my adword budget, some of them sign up with spam data and screw your conversions and testing and so on. if it was only search engine i would allow them but its not. at this point i have the whole aws ip range blocked.

lazyjones 9y ago
> it is understandable that you want to be in phonebook but also that you dont want somebody random to enter, use your resources and copy everything because of some other interests. It's not that simple: Google has used scraped content in the past to build competing products to the sites it was from. Examples: reviews from shopping sites for Google Shopping, Yelp reviews for Google Places. Google can do it because it basically owns your front door. Everyone else can't. > at this point i have the whole aws ip range blocked. From past experience: this will likely block some apps, possibly services like Alexa from accessing your site.