4 ms·

Hmm ... $ curl --dump-header - 'http://shop.nordstrom.com/robots.txt' HTTP/1.1 403 Forbidden Server: AkamaiGHost Mime-Version: 1.0 Content-

by troels 12y ago

Hmm ... $ curl --dump-header - 'http://shop.nordstrom.com/robots.txt' HTTP/1.1 403 Forbidden Server: AkamaiGHost Mime-Version: 1.0 Content-Type: text/html Content-Length: 281 Expires: Fri, 27 Feb 2015 23:42:13 GMT Date: Fri, 27 Feb 2015 23:42:13 GMT Connection: close ... Probably a bad idea to deny serving robots.txt to bots (even if they ought to interpret that as a total ban)

felixvolny 12y ago
Funny, when I do the same thing I get a completely different response: $ curl --dump-header - 'http://shop.nordstrom.com/robots.txt' http://shop.nordstrom.com/robots.txt' HTTP/1.1 200 OK Cache-Control: private Content-Type: text/html; charset=utf-8 Server: Microsoft-IIS/7.5 X-Powered-By: ASP.NET Content-Length: 865 Date: Sat, 28 Feb 2015 07:04:36 GMT Connection: keep-alive
- troels 12y ago
  Yeah, the clue is in "Server: AkamaiGHost". The site is clearly behind some sort of anti-crawl protection which decides it doesn't like me. I think it would make sense to make an exception for exactly that file though.