4 ms·
I understand a robots.txt "Disallow: /foo" to mean that it must not crawl that page, i.e. look at the links _inside_ that page.
by aristidb 16y ago
I understand a robots.txt "Disallow: /foo" to mean that it must not crawl that page, i.e. look at the links _inside_ that page.
- benologist 16y agoI've always interpreted it to mean they're explicitly not allowed to touch it - no exceptions (unless you actually specified exceptions for them which you can see on the 3rd last example at http://www.robotstxt.org/robotstxt.html http://www.robotstxt.org/robotstxt.html).
- moultano 16y agoOne counter intuitive thing. They are allowed to link to it in search results (using links that point to it to rank it for insance) but can't use the content of the page.
- aristidb 16y agoInteresting, and I've always interpreted it the other way. Looks like you are right. But the language on www.robotstxt.org is pretty vague, too. (What does it mean to "visit" an URL?) The "RFC" on http://www.robotstxt.org/norobots-rfc.txt http://www.robotstxt.org/norobots-rfc.txt (Section 3.2.2) states: "These lines indicate whether accessing a URL that matches the corresponding path is allowed or disallowed. Note that these instructions apply to any HTTP method on a URL." And I think both interpretations are in theory (but probably not in practice) valid, at least if a search engine is willing to add a site to its index without accessing it (which is unlikely, but not impossible).