15 ms·
Nike.com's robots.txt
- cdot2 5y agoSomeone put some effort into that
- causi 5y agoWebmasters also put highly specific URLs they don't want the world to see in robots.txt I'd be interested in browsing these pages.
- amelius 5y agoEh, it's for robots to read, not humans.
- mfollert 5y agoNow I feel attacked. :(
- bmurray7jhu 5y agoSee also: https://www.bloomberg.com/robots.txt https://www.bloomberg.com/robots.txt
- dazc 5y ago> If you can read this then you should apply here. If you also know why we're making some basic mistakes here, especially apply here.
- ewidar 5y agoI don't know much about the content of robots.txt, can you share some mistakes they are making here?
- dazc 5y agoIt's not really a good idea to restrict crawlers at directory level since it doesn't prevent any of those pages being indexed. You can get weird behaviour such as googlebot trying to crawl a page but not being allowed to. If this happens a lot then they are going to penalise you for it because they don't want a bunch of pages in their index that they don't know the content of. If this number of pages is a significant percentage of your site then you are in real trouble. Much better to use x-robots noindex, nofollow for pages you don't want to be in the public domain. https://developers.google.com/search/docs/advanced/crawling/block-indexing https://developers.google.com/search/docs/advanced/crawling/...
- zinekeller 5y ago... if you're specifically targeting Google. If you are a neutral site (in this case, caters to a lot of people - especially in Asia, where Google as a search engine is a hit-or-miss proposition), then this might backfire sine other crawlers won't understand X-Robots. You can mitigate this by knowing which spiders knows X-Robots (Google, Bing and Yandex AFAIK) and whitelist them while still using a disallow directive for the rest of them.
- deleted 5y ago[deleted]
- agilob 5y ago>Disallow: /polska Why is only poland blocked for all robots? Why is there no /deutchland?
- zinekeller 5y agoWhat's this: https://www.bloomberg.com/polska https://www.bloomberg.com/polska Seems that Bloomberg hosts something for the Polish, but I'm not sure what.
- agilob 5y agoFinancial reports about companies, not for their average readers
- undebuggable 5y agoLooks like aggregation of press releases coming from public companies listed on Warsaw's stock exchange.
- cookiengineer 5y agoThat I, Robot (Asimov) reference though. [1] [1] https://en.wikipedia.org/wiki/I,_Robot https://en.wikipedia.org/wiki/I,_Robot
- MikeDelta 5y agoI can really appreciate the ASCII art
- codetrotter 5y agoI can’t, because on iPhone it wraps.
- yololol 5y agoTry landscape mode :)
- codetrotter 5y agoStill wraps unfortunately. I’m using Safari on iOS 15.2.1, running on an iPhone X.
- jdkjs 5y agoTap the AA button next to the url field and zoom off using the bottom left button.
- soheilpro 5y agoAlso last.fm's (the last few lines): https://www.last.fm/robots.txt https://www.last.fm/robots.txt
- bouncing 5y agoDisallow: /harming/humans Honeypot or snarky?
- echelon 5y agoIt's a clever encoding of Asimov's Laws of Robotics Disallow: /harming/humans Disallow: /ignoring/human/orders Disallow: /harm/to/self > First Law. A robot may not injure a human being or, through inaction, allow a human being to come to harm. > Second Law. A robot must obey the orders given it by human beings except where such orders would conflict with the First Law. > Third Law. A robot must protect its own existence as long as such protection does not conflict with the First or Second Law. https://en.wikipedia.org/wiki/Three_Laws_of_Robotics https://en.wikipedia.org/wiki/Three_Laws_of_Robotics If you ever get the chance to read a bit of Asimov, I highly recommend it. I'm particularly fond of his Foundation series.
- moehm 5y agoObligatory XKCD: https://xkcd.com/1613/ https://xkcd.com/1613/
- cduzz 5y agoThe modern version of the 3 laws is: 1) don't give a robot a gun 2) don't teach it how to put more bullets into gun after breaking #1 3) don't teach it how to make bullets after breaking #2 3a) don't give it your amazon API keys or other credit card info (edited for formatting)
- robin_reala 5y ago1st rule of robotics.
- drcongo 5y ago
- zinekeller 5y agoI'm not sure if I should focus on the detailed logo at the end or the allowed parts for Chinese spiders (especially that it's not blocked for other spiders - is this how you signal priorities for Chinese spiders?) Edit: I've now noticed the disallow directive for other countries specifically targeted for Chinese spiders.
- danuker 5y agoIndeed. I am also somewhat puzzled about what there is to see here. I see the Chinese spiders are only allowed on the /cn/ directory, while disallowed from the root (/). Another thing I find mildly strange is that the help for certain countries is restricted. This is not what robots.txt should be used for, and it creates incentive for bots to disregard robots.txt.
- zinekeller 5y ago> Another thing I find mildly strange is that the help for certain countries is restricted. This is not what robots.txt should be used for, and it creates incentive for bots to disregard robots.txt. It seems that they did exist, but moved to specific TLDs (like for Brazil, https://www.nike.com.br https://www.nike.com.br). Lazy that they didn't redirect all help files to the correct location though.
- deleted 5y ago[deleted]
- thrdbndndn 5y agoThis is to make sure they only crawl the Chinese version of their site. It makes sense considering they have this: https://purpose.nike.com/statement-on-xinjiang https://purpose.nike.com/statement-on-xinjiang
- rambojazz 5y agoWhat's interesting about this other than the ASCII art?
- giorgioz 5y agoThe *just crawl it*. Also technically it's quite a big sitemap. It uses sub-sitemap files. It also specify to ignore everything but the chinese translation of the website for 3 chinese search engines.
- choward 5y agoI don't get it. What is this other than a Nike ad?
- anotheraccount9 5y agoReally nothing.
- vesinisa 5y ago# www.nike.com robots.txt -- just crawl it. I think it's play on the Nike's famous slogan "Just do it".
- only4here 5y agoPlus the big ASCII art at the bottom of the file.
- deleted 5y ago[deleted]
- paulcole 5y agoAre you upset by all ads? If so, you might want to find a place to spend time online that’s not an ad/content marketing project for a VC firm.
- mandiantBob 5y agoSee also youtube's https://www.youtube.com/robots.txt https://www.youtube.com/robots.txt
- culi 5y agoWow is this really from 2000? Some of those url paths certainly look it
- Svip 5y agoNo, it is a reference to Flight of the Conchords' The Humans are Dead.[1] [1] https://www.youtube.com/watch?v=mpe1R6veuBw https://www.youtube.com/watch?v=mpe1R6veuBw
- hayleox 5y agoMight I recommend this live version over the studio release? https://www.youtube.com/watch?v=0BcFHvEpP7A https://www.youtube.com/watch?v=0BcFHvEpP7A
- koollman 5y agoit's from a song of flight of the conchords, called robots. https://www.youtube.com/watch?v=hUnibKe_o18 https://www.youtube.com/watch?v=hUnibKe_o18
- d--b 5y ago“Just crawl it”
- zinekeller 5y agoA different kind of robots.txt: this is annotated on why they're blocked: https://meta.wikimedia.org/robots.txt https://meta.wikimedia.org/robots.txt
- npteljes 5y agoThis is fantastic. The wget one even has instruction in how to play nice.
- renewiltord 5y ago> /wiki/Requests_for_comment/Rodrigo_Tetsuo_Argenton@brwikimedia Everyone Loves Rodrigo
- BbzzbB 5y ago># Crawlers that are kind enough to obey, but which we'd rather not have ># unless they're feeding search engines. >[disallows UbiCrawler, DOC and zao] Anyone understand this bit? They are obedient spiders yet are disallowed because of purpose? Are these obedient but still overly intensive spiders?
- zinekeller 5y agoIt seems that all of them were built for recursive archiving in the early '00s for personal storage (so they were programs literally installed on computers rather than a specific group). Probably a legacy thing (considering at that time Wikipedia literally runs on a single server). I found a robots.txt file that even list a greater variety of said software: https://www.iatse.com/robots.txt https://www.iatse.com/robots.txt
- ketanip 5y agojust crawl it, a perfect ad.
- undebuggable 5y agoThe bad actors don't follow the robots.txt, or rather are bad actors because they don't follow robots.txt neither the informal courtesy practices. Then the main role of robots.txt is public shaming?
- chrisweekly 5y agoThere are robots which follow robots.txt but which site operators prefer to block or content-restrict, for various reasons. See eg the annotated wikipedia robots.txt me tioned upthread.
- Beltalowda 5y agoThere are really three types of crawlers: good actors, bad actors, and "grey actors". The grey actors can be just by making a mistake (or outright incompetence), or they're testing where the limits are, etc. robots.txt does nothing for the bad actors, but for grey actors it's a bit more complicated as they can move from "grey" to "good". Adding something for that in robots.txt in addition to some other limits (if needed) can be useful.
- flavius29663 5y agoIf you disallow something and it still get crawled, doesn't that reveal them as bad actors?
- Beltalowda 5y agoNot necessarily; could be a bug or other mistake such as incomplete support, or just plain ignorance/incompetence.
- deleted 5y ago[deleted]
- deleted 5y ago[deleted]
- atulvi 5y ago* https://www.etsy.com/robots.txt https://www.etsy.com/robots.txt * https://www.tripadvisor.com/robots.txt https://www.tripadvisor.com/robots.txt
- marginalia_nu 5y agoAs a search engine operator, I can confirm webmasters put a lot of weird crap in robots.txt. ASCII art, poems, emoji, love letters, sudoku puzzles, philosophical questions, whatever. It's cute, but odds are at least some parser will choke on them and depending on how the crawler is coded, either ignore your robots.txt; or skip crawling your site. It's not great either way as the format is more of a set of conventions than a proper well defined language all specified up in BNF. Webmasters also put highly specific URLs they don't want the world to see in robots.txt. Also not a great idea.
- hughrr 5y agoI made my robots.txt 2Gb of garbage from dev random to see how many spiders I could break. At least three Chinese ones downloaded the whole file.
- 5e92cb50239222b 5y agoNext time drop a symlink from robots.txt to /dev/urandom and see how it goes.
- dspillett 5y agoYou might want to rate limit that if you don't magically have free unlimited bandwidth quota.
- marginalia_nu 5y agoI hear it's decades of bad luck to do unless you check that the crawler's user-agent isn't "search.marginalia.nu"; but you could just tarpit the connection.
- dijit 5y agoWhile I'm in general not a fan of wildcard certificates, yours doesn't work for the bare domain: https://i.imgur.com/uKVK5xd.png https://i.imgur.com/uKVK5xd.png I was hitting it plaintext first, so a simple redirect to some subdomain instead of a bare redirect to https would probably work fine.
- throwawayninja 5y agoI love the organization! Not sure if the comments are exactly to-spec, but sounds reasonable enough for parsers to ignore broken lines just like how css is parsed.
- Yeri 5y agoCloudflare: https://www.cloudflare.com/robots.txt https://www.cloudflare.com/robots.txt Google, facebook and bbc's are surprisingly boring :)
- M4tze 5y agoI bet that was done by JGC
- rendall 5y agoI saw this on mobile, first and thought "What the heck is that `# hMMMMMMMMMMMMmhysooosyhdmNMMMMMMMMMMMMMMMMMMmds/-` stuff??"
- deleted 5y ago[deleted]
- seanwilson 5y agoFrom one of my projects: https://www.checkbot.io/robots.txt https://www.checkbot.io/robots.txt Also, it's a really common misunderstanding that robots.txt can be used to keep pages out of search results. You need to use noindex meta tags or noindex headers for that. Robots.txt tells crawlers which pages they can't fetch, but they're still allowed to include those pages in search results if they want.
- Cthulhu_ 5y agoAnd even those are advisory, if you really need to block them, you have to add rules to your webserver of known user agents and IPs (because plenty of indexers will spoof their user agent as well. Google probably does that so they can penalize those that offer different content to search engines than regular users. Doesn't really work though).
- tamaharbor 5y agoYeah, Nike is so wonderful.
- impetus21 5y agoThey could have put Nike check ascii art...
- ChicagoBoy11 5y agoIt's at the bottom.
- bodge5000 5y agoI've got great news for you
- sunjester 5y agojust crawl it... i love the humor on these guys.
- remram 5y agoA pet peeve of mine is sites that block the W3C Link Checker. GitHub does this, preventing me from using the Link Checker on my documentation sites that link to GitHub projects and files... I'm not even trying to read their files at all, just to check whether my link is 404. Good on W3C for respecting standards, but this means I just have to copy every one of those links from the report into my address bar, like a robot, to check them...