5 ms·
People wouldn't block Google because allowing Google scraping offers a return in the form of more traffic that offsets the cost. $BRAND_NEW_ENGINE wouldn't have
by Analemma_ 10y ago
People wouldn't block Google because allowing Google scraping offers a return in the form of more traffic that offsets the cost. $BRAND_NEW_ENGINE wouldn't have that advantage.
- thesmallestcat 10y agoGoogle's crawlers don't exactly perform 2FA before they crawl a website. If impersonating Google's crawlers doesn't suit your fancy, there are all manner of ways to anonymize a crawler. So I think blocking is one of the least interesting challenges. The other side of the coin is robots.txt is not inherently adversarial, and a crawler could waste quite some time and energy crawling truly meaningless content. That, in my mind, is the interesting challenge: Obsoleting robots.txt with an intelligent (and gentle) crawler.
- desdiv 10y agoGoogle doesn't feel like being impersonated so they offer an easy way to authenticate Googlebot: https://support.google.com/webmasters/answer/80553?hl=en https://support.google.com/webmasters/answer/80553?hl=en
- deleted 10y ago[deleted]