5 ms·
robots.txt applies to the source of the links, not the target. So if, say, http://www.paulgraham.com/ http://www.paulgraham.com/ links to http://m.bing.com/sea
by aristidb 16y ago
robots.txt applies to the source of the links, not the target.
So if, say, http://www.paulgraham.com/ http://www.paulgraham.com/ links to http://m.bing.com/search http://m.bing.com/search, then http://m.bing.com/robots.txt http://m.bing.com/robots.txt does not apply to that.
EDIT: If you think this is wrong, please explain it instead of just downvoting me, because I think it is pretty unfair that I lose karma for explaining my interpretation.
- benologist 16y agoWhy would someone with no authority over your site linking to your site override your robots.txt? Edit: I didn't downvote you, but I think you're wrong because it makes no sense - if I link to your site that shouldn't give search engines a free pass to ignore your wishes and do whatever they want with your content.
- aristidb 16y agoI understand a robots.txt "Disallow: /foo" to mean that it must not crawl that page, i.e. look at the links _inside_ that page.
- benologist 16y agoI've always interpreted it to mean they're explicitly not allowed to touch it - no exceptions (unless you actually specified exceptions for them which you can see on the 3rd last example at http://www.robotstxt.org/robotstxt.html http://www.robotstxt.org/robotstxt.html).
- moultano 16y agoOne counter intuitive thing. They are allowed to link to it in search results (using links that point to it to rank it for insance) but can't use the content of the page.
- aristidb 16y agoInteresting, and I've always interpreted it the other way. Looks like you are right. But the language on www.robotstxt.org is pretty vague, too. (What does it mean to "visit" an URL?) The "RFC" on http://www.robotstxt.org/norobots-rfc.txt http://www.robotstxt.org/norobots-rfc.txt (Section 3.2.2) states: "These lines indicate whether accessing a URL that matches the corresponding path is allowed or disallowed. Note that these instructions apply to any HTTP method on a URL." And I think both interpretations are in theory (but probably not in practice) valid, at least if a search engine is willing to add a site to its index without accessing it (which is unlikely, but not impossible).
- Zakuzaa 16y agoThat's not true.
- pcowans 16y agoI think you're confusing robots.txt with the nofollow attribute value: http://en.wikipedia.org/wiki/Nofollow http://en.wikipedia.org/wiki/Nofollow
- gorset 16y agoyup. Anchor text is fair game. Keywords taken from anchor texts are usually a good indication on what the linked page is about. A page can have a high page rank without ever being crawled - given that other high ranked pages have linked to it with good keywords in the anchor text. Such pages can often be identified by the lack of snippet texts. Edit: I may have misunderstood you, you can only use the data found in links, not follow them.
- ajays 16y agoIf a page is not crawlable, but listed in the search results based on anchor text (or the URL), then it will not have any snippets. You can show snippets only if you crawl the page itself.
- ajays 16y agoIt just means a crawler cannot retrieve the page; in other words, a crawler cannot GET a URL for whom there's an exclusion pattern in robots.txt .