3 ms·

maybe it has been discussed before (i didn't follow the craigslist discussions) but it's funny that craigslist does not have any robots.txt on it's city subdoma

by franze 14y ago

maybe it has been discussed before (i didn't follow the craigslist discussions) but it's funny that craigslist does not have any robots.txt on it's city subdomains i.e.: http://newyork.craigslist.org/mnh/sub/3195940666.html http://newyork.craigslist.org/mnh/sub/3195940666.html is hosted on http://newyork.craigslist.org/ http://newyork.craigslist.org/ but http://newyork.craigslist.org/robots.txt http://newyork.craigslist.org/robots.txt makes an HTTP 302 redirect to http://www.craigslist.org/robots.txt http://www.craigslist.org/robots.txt, but http://www.craigslist.org/robots.txt http://www.craigslist.org/robots.txt is only a valid robots.txt for http://www.craigslist.org http://www.craigslist.org, and not for http://newyork.craigslist.org/ http://newyork.craigslist.org/, as the host, protocol and port of the robots.txt file is an essential part of its validity range. so basically newyork.craigslist.org does not have any robots.txt and can be crawled as you like. if you want more about it, see https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt https://developers.google.com/webmasters/control-crawl-index... >The directives listed in the robots.txt file apply only to >the host, protocol and port number where the file is hosted.

killyourheros 14y ago
Isn't the host still craigslist?
sigmadelta 14y ago
Saying that "newyork.craigslist.org does not have any robots.txt and can be crawled as you like" is false. Search engines follow redirects until valid robots.txt files are found. From that same document you linked: 3xx (redirection) Redirects will generally be followed until a valid result can be found (or a loop is recognized). We will follow a limited number of redirect hops (RFC 1945 for HTTP/1.0 allows up to 5 hops) and then stop and treat it as a 404. Handling of robots.txt redirects to disallowed URLs is undefined and discouraged. Handling of logical redirects for the robots.txt file based on HTML content that returns 2xx (frames, JavaScript, or meta refresh-type redirects) is undefined and discouraged.

by franze 14y ago

killyourheros 14y ago

Isn't the host still craigslist?

sigmadelta 14y ago

Saying that "newyork.craigslist.org does not have any robots.txt and can be crawled as you like" is false. Search engines follow redirects until valid robots.txt files are found. From that same document you linked: 3xx (redirection) Redirects will generally be followed until a valid result can be found (or a loop is recognized). We will follow a limited number of redirect hops (RFC 1945 for HTTP/1.0 allows up to 5 hops) and then stop and treat it as a 404. Handling of robots.txt redirects to disallowed URLs is undefined and discouraged. Handling of logical redirects for the robots.txt file based on HTML content that returns 2xx (frames, JavaScript, or meta refresh-type redirects) is undefined and discouraged.