5 ms·
Nordstrom's robots.txt
- nottestuser 12y agoYou're rolling along thinking this is just some bad web developer somewhere until you get to the last Disallow...
- lfowles 12y agoWell, I'm on a list now..
- nly 12y agoYou have something against petite young ladies of a modest stature? Sizeist!
- CoreSet 12y agoI don't get it: is that disallow targeting some sort of porn spambot crawler?
- prokizzle 12y agoMaybe that was their intention, but that's not how it's done.
- jarin 12y agoThey probably made a product with that slug by accident and are doing that to try to get it removed from search engine results.
- meej 12y agoIt makes me wonder if Nordstrom ever stocked the now-defunct Pornstar clothing brand.
- philip1209 12y agoMaybe some mal-intended inbound links that they don't want to be associated with?
- cmdrfred 12y agoIs this some sort of new terrible negative seo? Associate a target url for a bunch of negative things?
- Springtime 12y agoDisallow is used to exclude certain URLs from being crawled, in this case it looks like a search query as the store's search uses this pattern: http://shop.nordstrom.com/sr/query http://shop.nordstrom.com/sr/query .
- IgorPartola 12y agoCould that be a honeypot to trap bots that don't respect robots.txt?
- prokizzle 12y agomaybe, but why would THAT be the honey pot? it could be any phrase, really.
- IgorPartola 12y agoWell, you'd want them to try to crawl it. Perhaps there was a bot crawling their site for such links and they wanted to identify specifically that one. It seems most likely that such URL's never existed, but this would be a decent way to identify a bot that's otherwise not distinguishable.
- huhtenberg 12y agoLooks more like a mop-up after a website infection incident.
- ashmud 12y agoI have had to do that before. The URLs in that case seemed to have a more random distribution than most of these.
- shouldbeworking 12y agoThis other retailer's page is about the same except for that one entry: http://www.neimanmarcus.com/robots.txt http://www.neimanmarcus.com/robots.txt
- jjcm 12y agoAnyone have any theories on why they're blocking specific products?
- ossreality 12y agoThere are keywords in a number of those that would cause the Nordstrom products to show up when searching for more nefarious things. I'm guessing this was their way of keeping Nordstrom's out of the results for some of those searches. Some of the other URLs though, I can't tell.
- jj666 12y agoDesigners often pull stuff from stores or negotiate exlusives with another store. This could have been some attempts of making sure some search results don't show up anymore.
- wmil 12y agoThey might have signed an agreement restricting "lowest advertised price". They're pretty common.
- ben010783 12y agoI can't think of a single reason. Some of the URLs still work too.
- thephyber 12y agoThe point of robots.txt "Disallow"s is to tell search engine crawlers not to crawl a URL (which could exist). It's not to tell search engines that URLs matching a pattern don't exist.
- inyourtenement 12y agoI've dealt with retailers who had to remove clothing products for legal reasons, usually copyright complaints. It could be a knockoff of another design, or copyrighted characters on the clothes. If there is any backdoor way to get to the product page, they could be in trouble.
- abakker 12y agoDo a google search for some of those products - ends up being even weirder when you see some of the sites linking to them. edit: especially considering that google still links to them, it just doesn't show their description, just gives the standard error for "couldn't show this description because it was blocked by the site's robots.txt"
- gerbal 12y agoGoogle indexes pages that link to those pages, but can't index the pages themselves. The PageRank algorithm still sees inbound links to unindexable pages.
- WestCoastJustin 12y agoReading, reading, reading... wtf! I hope that is an abbreviation for something, but to be honest, I don't want to Google and find out. Disallow: /sr/extra-small-teen-porn*
- pvsnp 12y agoThat's just way too specific to be an accident, wonder what the story is..
- olivierkaisin 12y agoI wouldn't be surprised to hear that they did put that url into their robots.txt because it is a search result page generated by a bot and that has been visited and indexed by Google…
- hurin 12y agoI'm not up to date on how google's current search indexing algorithm works. Supposing someone has a lot of bots that post links to `extra-small-teen-pony` or some other /sr query on nordstrom.com from other sites - would google index that?
- cmdrfred 12y agoIf they threw a custom 404 that reports 200. That's how I assumed those phone number lookup sites work.
- untog 12y agoI'm not sure what letters would go on the end of "porn" to make a clothing item, but I'll bet it exists. Or someone misspelt "pony", or... well, you get the idea. No matter what, you really don't want your site to appear in search results for those terms.
- deleted 12y ago[deleted]
- woonord 12y agoHi! The robots file is not a text file. I know! Products blocked due to possible copyright infringement. A lot are disputes over product name or t-shirt slogan. We were ranking for some terms we did not want to be associated with so we added the offending search result page to the file. Not sure how it ever got indexed in the first place. Peace!
- simonebrunozzi 12y agoI like you already :)
- filmgirlcw 12y agoawesome response!
- woonord 12y agothanks! I have some inside knowledge!
- troels 12y agoI don't think you actually want to do that. Blocking in robots.txt will prevent Google from crawling the URL - not from indexing it. You actually want them to crawl the URL and then respond with a 404 or 410. If there are inbound links pointing to that URL, you should disavow them in GWT.
- eli 12y agoa "noindex" robots meta tag would work too
- prokizzle 12y agoAlso, in Disallow: /sr/mattress* Disallow: /sr/mattresses* Doesn't A imply B?
- deleted 12y ago[deleted]
- api 12y ago"Disallow: /sr/extra-small-teen-porn" Yeah someone got pwned, or had a serious HR incident.
- woonord 12y agoUpdate! offending term we rank on "extra small teen porn" - offending page /sr/petite-extra-small. Robots updated (live tomorrow)! Asked Google to remove the URL.
- temuze 12y agoSo uh, how did you find out that you match for this phrase?
- thephyber 12y agoGoogle Webmaster Tools shows you the search terms used to click thru to your website from Google Searches. I'm not sure if it still works, but referer URLs used to log search terms in the web access logs.
- Buge 12y agoProbably looked at their logs and found a lot of visits to that URL.
- troels 12y agoHmm ... $ curl --dump-header - 'http://shop.nordstrom.com/robots.txt' HTTP/1.1 403 Forbidden Server: AkamaiGHost Mime-Version: 1.0 Content-Type: text/html Content-Length: 281 Expires: Fri, 27 Feb 2015 23:42:13 GMT Date: Fri, 27 Feb 2015 23:42:13 GMT Connection: close ... Probably a bad idea to deny serving robots.txt to bots (even if they ought to interpret that as a total ban)
- felixvolny 12y agoFunny, when I do the same thing I get a completely different response: $ curl --dump-header - 'http://shop.nordstrom.com/robots.txt' http://shop.nordstrom.com/robots.txt' HTTP/1.1 200 OK Cache-Control: private Content-Type: text/html; charset=utf-8 Server: Microsoft-IIS/7.5 X-Powered-By: ASP.NET Content-Length: 865 Date: Sat, 28 Feb 2015 07:04:36 GMT Connection: keep-alive
- troels 12y agoYeah, the clue is in "Server: AkamaiGHost". The site is clearly behind some sort of anti-crawl protection which decides it doesn't like me. I think it would make sense to make an exception for exactly that file though.
- rockdiesel 12y agoWhy they are even bothering to allow their search results to be indexed is beyond me. Any decent SEO would tell you this is a terrible idea. If their in-house SEO told them this was a good idea, then they seriously need to re-evaluate their program. Not only has Matt Cutts said you shouldn't be doing this [1][2], but it's also listed in Google's Webmaster Guidelines as things NOT to do [3] A quick query of "site:nordstrom.com/sr/" shows they have 260,000 search results pages in Google's index. Just a few search results pages with 0 results their system is creating and allowing Google to index: http://shop.nordstrom.com/sr/sorrelli http://shop.nordstrom.com/sr/sorrelli http://shop.nordstrom.com/sr/query http://shop.nordstrom.com/sr/query http://shop.nordstrom.com/sr/flogg http://shop.nordstrom.com/sr/flogg There can be benefit in creating pages at scale, but this is a textbook example on how not to go about it. 1. https://www.mattcutts.com/blog/search-results-in-search-results/ https://www.mattcutts.com/blog/search-results-in-search-resu... 2. http://searchengineland.com/googles-cutts-auto-generated-content-search-results-in-our-index-violate-our-guidelines-171553 http://searchengineland.com/googles-cutts-auto-generated-con... 3. https://support.google.com/webmasters/answer/35769 https://support.google.com/webmasters/answer/35769
- United857 12y agoI guess they removed that porn line. I clicked through on it and was perplexed what the big deal was, and I had to read the comments here to know.
- Buge 12y agoIt's still there for me. Maybe they use a CDN that serves different things to different locations or does caching?
- prokizzle 12y agoScreenshot of original robots.txt for when the changes get pushed to production: http://imgur.com/8zkK8w2 http://imgur.com/8zkK8w2
- deleted 12y ago[deleted]