4 ms·

Google seems to see robots.txt as "more what you call guidelines, than actual rules". Sites that block googlebot or all bots with robots.txt still turn up in go

Google seems to see robots.txt as "more what you call guidelines, than actual rules". Sites that block googlebot or all bots with robots.txt still turn up in google searches, just without a description, and are obviously still indexed.

dorgo 7y ago
>"more what you call guidelines, than actual rules" they can index without scraping. It is enough that other websites have links to you site. So the google bot follows the rules in robots.txt to the letter. "no-index" is the way to stay away from google.
- SahAssar 7y ago
  They can't read my no-index if they obey my robots.txt. Do they break the robots.txt to be able to read my no-index or do they assume my "Disallow: /" means I'm fine with them indexing/linking? Without the noindex part of robots.txt (which google decided to ignore not so long ago) this is not solvable.
jefftk 7y ago
robots.txt is a tool to control crawling, not to specify how you would like your site to be displayed (or not) in search results. If you don't want search engines to include your site, set: <meta name="robots" content="noindex"> while to block just Google do: <meta name="googlebot" content="noindex"> See https://support.google.com/webmasters/answer/93710 https://support.google.com/webmasters/answer/93710 If Googlebot is not respecting robots.txt, and is crawling something it's been instructed not to crawl, let me know and I can file a bug? (Disclosure: I work for Google but not on Search, speaking only for myself)
- areyousure 7y ago
  But that requires that Googlebot be allowed to crawl the page in robots.txt in the first place. How do you tell Googlebot to not crawl your site and to not index it either? Previously, one could use the undocumented "Noindex" directive in robots.txt, but this will be disabled soon: https://webmasters.googleblog.com/2019/07/a-note-on-unsupported-rules-in-robotstxt.html https://webmasters.googleblog.com/2019/07/a-note-on-unsuppor...
  - jefftk 7y ago
    The blog post you link has a bunch of alternatives, but I agree they're not great. If there are a lot of webmasters who want to be able to noindex through robots.txt then making the case for adding noindex to the standard would be a good next step. (Still speaking only for myself)
    - ademarre 7y ago
      Googlebot actually used to support a noindex rule in robots.txt, but they are removing it. https://webmasters.googleblog.com/2019/07/a-note-on-unsupported-rules-in-robotstxt.html https://webmasters.googleblog.com/2019/07/a-note-on-unsuppor...
      - jefftk 7y ago
        Yes, that was linked above. It looks like this is part of reducing support to what's in the spec?
        ademarre 7y ago
        Oops, yep. I didn't see that context.
  - tylerl 7y ago
    The bot doesn't need to crawl your site for it to be indexed; it crawls other sites that link to yours. You can specify your index preferences in Webmaster Tools. Don't know if there's a domain-wide off switch in there, but there probably is.
    - deleted 7y ago
      [deleted]
    - SahAssar 7y ago
      Using Webmaster Tools is not a good option since it requires you register with the exact company you are probably trying to not interact with.
- SahAssar 7y ago
  I sent you an email, and I'm posting it here but without identifying info: --- Hi Jeff, Thank you for your comment, I'm replying via email to send some info I'd rather not share on HN, but will post the same redacted in HN. I used to (back when starting my web-dev career) run a one man show development team of a web agency and all our development/pre-prod sites (that had to be unauthed) had robots.txt to disallow all bots, but they still popped up in Google. Searching some of the old domains in google I found an example here: http://***.***/*** http://***.***/***, and attached is an example of it showing up in a SERP and a what the robots.txt looks like (and I'm pretty sure that the robots.txt has looked like that since that page was created). In this case it is just one page that nobody will care about, and since I'm not working on projects that are open but "robots.txt hidden" anymore I don't know if it is as bad as it used to be, but I regularly see pages with the "No information is available for this page" whose domains have robots.txt's that disallow all bots but still show up in Google. Please let me know if I missed anything :)
  - jefftk 7y ago
    Thanks for sending the screenshot! That site shows up with "no information is available for this page", which means that while robots.txt has disallowed bots from crawling it the page is still linked from other pages that do allow crawling. The robots.txt protocol gives instructions to crawlers about how they should interact with the site. If you instead want to give instructions to indexes, use the noindex meta tag.
    - SahAssar 7y ago
      You're right, I was wrong about how to expect a "Disallow: /" to work. But isn't it sorta odd to have a protocol to control crawling (which is usually done to index) but (almost) require a compliant indexer to crawl all pages to comply with the indexing rules? In this example the robots.txt has clearly told all bots to not crawl this site, but the only way to read the meta tag (or equivalent header) is to crawl the site. So I assume that in this case google either assumes that it is fine to crawl URL's that it has found elsewhere while ignoring the robots.txt or it assumes that pages disallowed by robots.txt are "open for indexing/linking", which would mean that any page both disallowed by robots.txt and which has a noindex meta tag would still show up, right? What is the intended behavior if a page is disallowed by robots.txt and still linked by another indexed page? Will it get crawled or just assumed to be okay for indexing/linking? Is there any way to tell Google not to index/link and not to crawl?

dorgo 7y ago

>"more what you call guidelines, than actual rules" they can index without scraping. It is enough that other websites have links to you site. So the google bot follows the rules in robots.txt to the letter. "no-index" is the way to stay away from google.

SahAssar 7y ago
They can't read my no-index if they obey my robots.txt. Do they break the robots.txt to be able to read my no-index or do they assume my "Disallow: /" means I'm fine with them indexing/linking? Without the noindex part of robots.txt (which google decided to ignore not so long ago) this is not solvable.

jefftk 7y ago

robots.txt is a tool to control crawling, not to specify how you would like your site to be displayed (or not) in search results. If you don't want search engines to include your site, set: <meta name="robots" content="noindex"> while to block just Google do: <meta name="googlebot" content="noindex"> See https://support.google.com/webmasters/answer/93710 https://support.google.com/webmasters/answer/93710 If Googlebot is not respecting robots.txt, and is crawling something it's been instructed not to crawl, let me know and I can file a bug? (Disclosure: I work for Google but not on Search, speaking only for myself)

areyousure 7y ago
But that requires that Googlebot be allowed to crawl the page in robots.txt in the first place. How do you tell Googlebot to not crawl your site and to not index it either? Previously, one could use the undocumented "Noindex" directive in robots.txt, but this will be disabled soon: https://webmasters.googleblog.com/2019/07/a-note-on-unsupported-rules-in-robotstxt.html https://webmasters.googleblog.com/2019/07/a-note-on-unsuppor...
- jefftk 7y ago
  The blog post you link has a bunch of alternatives, but I agree they're not great. If there are a lot of webmasters who want to be able to noindex through robots.txt then making the case for adding noindex to the standard would be a good next step. (Still speaking only for myself)
  - ademarre 7y ago
    Googlebot actually used to support a noindex rule in robots.txt, but they are removing it. https://webmasters.googleblog.com/2019/07/a-note-on-unsupported-rules-in-robotstxt.html https://webmasters.googleblog.com/2019/07/a-note-on-unsuppor...
    - jefftk 7y ago
      Yes, that was linked above. It looks like this is part of reducing support to what's in the spec?
      - ademarre 7y ago
        Oops, yep. I didn't see that context.
- tylerl 7y ago
  The bot doesn't need to crawl your site for it to be indexed; it crawls other sites that link to yours. You can specify your index preferences in Webmaster Tools. Don't know if there's a domain-wide off switch in there, but there probably is.
  - deleted 7y ago
    [deleted]
  - SahAssar 7y ago
    Using Webmaster Tools is not a good option since it requires you register with the exact company you are probably trying to not interact with.
SahAssar 7y ago
I sent you an email, and I'm posting it here but without identifying info: --- Hi Jeff, Thank you for your comment, I'm replying via email to send some info I'd rather not share on HN, but will post the same redacted in HN. I used to (back when starting my web-dev career) run a one man show development team of a web agency and all our development/pre-prod sites (that had to be unauthed) had robots.txt to disallow all bots, but they still popped up in Google. Searching some of the old domains in google I found an example here: http://***.***/*** http://***.***/***, and attached is an example of it showing up in a SERP and a what the robots.txt looks like (and I'm pretty sure that the robots.txt has looked like that since that page was created). In this case it is just one page that nobody will care about, and since I'm not working on projects that are open but "robots.txt hidden" anymore I don't know if it is as bad as it used to be, but I regularly see pages with the "No information is available for this page" whose domains have robots.txt's that disallow all bots but still show up in Google. Please let me know if I missed anything :)
- jefftk 7y ago
  Thanks for sending the screenshot! That site shows up with "no information is available for this page", which means that while robots.txt has disallowed bots from crawling it the page is still linked from other pages that do allow crawling. The robots.txt protocol gives instructions to crawlers about how they should interact with the site. If you instead want to give instructions to indexes, use the noindex meta tag.
  - SahAssar 7y ago
    You're right, I was wrong about how to expect a "Disallow: /" to work. But isn't it sorta odd to have a protocol to control crawling (which is usually done to index) but (almost) require a compliant indexer to crawl all pages to comply with the indexing rules? In this example the robots.txt has clearly told all bots to not crawl this site, but the only way to read the meta tag (or equivalent header) is to crawl the site. So I assume that in this case google either assumes that it is fine to crawl URL's that it has found elsewhere while ignoring the robots.txt or it assumes that pages disallowed by robots.txt are "open for indexing/linking", which would mean that any page both disallowed by robots.txt and which has a noindex meta tag would still show up, right? What is the intended behavior if a page is disallowed by robots.txt and still linked by another indexed page? Will it get crawled or just assumed to be okay for indexing/linking? Is there any way to tell Google not to index/link and not to crawl?