2 ms·
One Question: What robots.txt do you have to obey? http://app.imdb.com/robots.txt http://app.imdb.com/robots.txt or http://www.imdb.com/robots.txt http://www.i
by berendhh 14y ago
One Question:
What robots.txt do you have to obey?
http://app.imdb.com/robots.txt http://app.imdb.com/robots.txt
or
http://www.imdb.com/robots.txt http://www.imdb.com/robots.txt?
The second one is rather restrictive (and the first one does not exist...)
- erikano 14y agoThe Google crawlers, at least, will only consider a robots.txt on a subdomain valid for the subdomain it was served from.[1] [1]: https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt https://developers.google.com/webmasters/control-crawl-index...