4 ms·

* most sites prohibit scraping but beg for Google to scrape them * many good products/websites are based on scraped content * many good products/websites are

* most sites prohibit scraping but beg for Google to scrape them * many good products/websites are based on scraped content * many good products/websites are not feasible because of scraping limitations and limited access to data, even publicly funded data (e.g. no real estate ads with noise overlay despite the EU mandating noise maps in all member states; member states have prohibitive access rules for these maps) * "rogue" scraping causes problems for many websites; blocking it creates problems for legitimate users of various proxy/anonymization services like Tor. Captchas are not a long-term solution, good programming defeats them. I'd welcome a simple technical solution for scraping that takes into account the interests of site owners, other publishers, the public. The sooner publishers get together and build one, the better for them.

manigandham 9y ago
Indexing content is different from scraping content.
- nyrina 9y ago
  True, indexing content is not the same. But you can't index anything you haven't already scraped. Indexing is simply the second step in the the process, scraping is the first step, and (users) searching the index is the third step.
viach 9y ago
Could you please elaborate on how good programming defeats captchas?
disiplus 9y ago
google at this point is "phonebook" of the internet. it is understandable that you want to be in phonebook but also that you dont want somebody random to enter, use your resources and copy everything because of some other interests. if there would be some official list of all "phonebooks" and you would allow those to scan your site but block others i would do it. scrapers from my side, spam my logs/analytics, fake real visits, try to spend my adword budget, some of them sign up with spam data and screw your conversions and testing and so on. if it was only search engine i would allow them but its not. at this point i have the whole aws ip range blocked.
- lazyjones 9y ago
  > it is understandable that you want to be in phonebook but also that you dont want somebody random to enter, use your resources and copy everything because of some other interests. It's not that simple: Google has used scraped content in the past to build competing products to the sites it was from. Examples: reviews from shopping sites for Google Shopping, Yelp reviews for Google Places. Google can do it because it basically owns your front door. Everyone else can't. > at this point i have the whole aws ip range blocked. From past experience: this will likely block some apps, possibly services like Alexa from accessing your site.

manigandham 9y ago

Indexing content is different from scraping content.

nyrina 9y ago
True, indexing content is not the same. But you can't index anything you haven't already scraped. Indexing is simply the second step in the the process, scraping is the first step, and (users) searching the index is the third step.

viach 9y ago

Could you please elaborate on how good programming defeats captchas?

disiplus 9y ago

google at this point is "phonebook" of the internet. it is understandable that you want to be in phonebook but also that you dont want somebody random to enter, use your resources and copy everything because of some other interests. if there would be some official list of all "phonebooks" and you would allow those to scan your site but block others i would do it. scrapers from my side, spam my logs/analytics, fake real visits, try to spend my adword budget, some of them sign up with spam data and screw your conversions and testing and so on. if it was only search engine i would allow them but its not. at this point i have the whole aws ip range blocked.

lazyjones 9y ago
> it is understandable that you want to be in phonebook but also that you dont want somebody random to enter, use your resources and copy everything because of some other interests. It's not that simple: Google has used scraped content in the past to build competing products to the sites it was from. Examples: reviews from shopping sites for Google Shopping, Yelp reviews for Google Places. Google can do it because it basically owns your front door. Everyone else can't. > at this point i have the whole aws ip range blocked. From past experience: this will likely block some apps, possibly services like Alexa from accessing your site.