3 ms·
Robots.txt is a mechanism for providing instructions to automated crawlers but I don't think they've ever been promised to be used when a URL is manually or aut
by jeroenhd 4y ago
Robots.txt is a mechanism for providing instructions to automated crawlers but I don't think they've ever been promised to be used when a URL is manually or automatically submitted through other means (i.e. another site linking to yours). In those cases, a single page will probably be crawled, but the rest of the domain probably won't.