5 ms·
There's a technical reason - the blocked pages might open infinite URL spaces or bring the site down (crawler hitting /cgi-bin).
by cmarschner 13y ago
There's a technical reason - the blocked pages might open infinite URL spaces or bring the site down (crawler hitting /cgi-bin).
- mburns 13y agoThose cornercases can exist on sites that don't have a robots.txt and still have to be crawled correctly.
- memracom 13y agoThat is NOT a technical reason. Technically speaking a robots.txt that says User-agent: * Disallow: / means that you should not crawl the site today. It should have no effect whatsoever on displaying pages that WERE crawled before the timestamp on the robots.txt file.
- mmagin 13y agoActually, if you want to interpret robots.txt that way, it raises the problem of "how long can I consider a robots.txt valid for?"