4 ms·
If Google today started ignoring robots.txt, not many people would start blocking Google's crawlers, assuming they continued to do their job efficiently. robots
by thesmallestcat 10y ago
If Google today started ignoring robots.txt, not many people would start blocking Google's crawlers, assuming they continued to do their job efficiently. robots.txt is security by obscurity at best.
- Analemma_ 10y agoPeople wouldn't block Google because allowing Google scraping offers a return in the form of more traffic that offsets the cost. $BRAND_NEW_ENGINE wouldn't have that advantage.
- thesmallestcat 10y agoGoogle's crawlers don't exactly perform 2FA before they crawl a website. If impersonating Google's crawlers doesn't suit your fancy, there are all manner of ways to anonymize a crawler. So I think blocking is one of the least interesting challenges. The other side of the coin is robots.txt is not inherently adversarial, and a crawler could waste quite some time and energy crawling truly meaningless content. That, in my mind, is the interesting challenge: Obsoleting robots.txt with an intelligent (and gentle) crawler.
- desdiv 10y agoGoogle doesn't feel like being impersonated so they offer an easy way to authenticate Googlebot: https://support.google.com/webmasters/answer/80553?hl=en https://support.google.com/webmasters/answer/80553?hl=en
- deleted 10y ago[deleted]
- angry-hacker 10y agoThey do ignore robots.txt sometimes, the pages just font end up in serps.