4 ms·
Nike.com's robots.txt file
- JoshTriplett 10y agoOdd to prohibit crawling PDFs; modern search engines handle those just fine.
- John23832 10y agoThey probably just want to avoid having to serve them.
- ryanbertrand 10y agojust crawl it...expect SWFs and PDFs.
- antongribok 10y agoIn Firefox I got access denied... In case anyone else can't get to it, here's a pastebin: http://pastebin.com/raw/amTGwEEa http://pastebin.com/raw/amTGwEEa
- atombath 10y agoNot even disallowing crappy crawlers like majestic 12? They're going to scuff your Jordans!
- kombucha2 10y agothats pretty great
- Ffaine 10y agoThat's a nice work.