4 ms·
The Google crawlers, at least, will only consider a robots.txt on a subdomain valid for the subdomain it was served from.[1] [1]: https://developers.google.com
by erikano 14y ago
The Google crawlers, at least, will only consider a robots.txt on a subdomain valid for the subdomain it was served from.[1]
[1]: https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt https://developers.google.com/webmasters/control-crawl-index...