3 ms·
Funny, when I do the same thing I get a completely different response: $ curl --dump-header - 'http://shop.nordstrom.com/robots.txt' http://shop.nordstrom.com/
by felixvolny 12y ago
Funny, when I do the same thing I get a completely different response:
$ curl --dump-header - 'http://shop.nordstrom.com/robots.txt' http://shop.nordstrom.com/robots.txt'
HTTP/1.1 200 OK
Cache-Control: private
Content-Type: text/html; charset=utf-8
Server: Microsoft-IIS/7.5
X-Powered-By: ASP.NET
Content-Length: 865
Date: Sat, 28 Feb 2015 07:04:36 GMT
Connection: keep-alive
- troels 12y agoYeah, the clue is in "Server: AkamaiGHost". The site is clearly behind some sort of anti-crawl protection which decides it doesn't like me. I think it would make sense to make an exception for exactly that file though.