3 ms·
But that requires that Googlebot be allowed to crawl the page in robots.txt in the first place. How do you tell Googlebot to not crawl your site and to not ind
by areyousure 7y ago
But that requires that Googlebot be allowed to crawl the page in robots.txt in the first place.
How do you tell Googlebot to not crawl your site and to not index it either?
Previously, one could use the undocumented "Noindex" directive in robots.txt, but this will be disabled soon: https://webmasters.googleblog.com/2019/07/a-note-on-unsupported-rules-in-robotstxt.html https://webmasters.googleblog.com/2019/07/a-note-on-unsuppor...
- jefftk 7y agoThe blog post you link has a bunch of alternatives, but I agree they're not great. If there are a lot of webmasters who want to be able to noindex through robots.txt then making the case for adding noindex to the standard would be a good next step. (Still speaking only for myself)
- ademarre 7y agoGooglebot actually used to support a noindex rule in robots.txt, but they are removing it. https://webmasters.googleblog.com/2019/07/a-note-on-unsupported-rules-in-robotstxt.html https://webmasters.googleblog.com/2019/07/a-note-on-unsuppor...
- tylerl 7y agoThe bot doesn't need to crawl your site for it to be indexed; it crawls other sites that link to yours. You can specify your index preferences in Webmaster Tools. Don't know if there's a domain-wide off switch in there, but there probably is.
- deleted 7y ago[deleted]
- SahAssar 7y agoUsing Webmaster Tools is not a good option since it requires you register with the exact company you are probably trying to not interact with.