3 ms·
That is NOT a technical reason. Technically speaking a robots.txt that says User-agent: * Disallow: / means that you should not crawl the site today. It shou
by memracom 13y ago
That is NOT a technical reason.
Technically speaking a robots.txt that says
User-agent: *
Disallow: /
means that you should not crawl the site today. It should have no effect whatsoever on displaying pages that WERE crawled before the timestamp on the robots.txt file.
- mmagin 13y agoActually, if you want to interpret robots.txt that way, it raises the problem of "how long can I consider a robots.txt valid for?"