7 ms·

Hi, author here. Google stopped supporting robots.txt [edit: as a way to fully remove your site] a few years ago, so these meta tags are now the recommended way

• new_guy 5y ago
  Did you actually read your link? That's not at all what it says.
  • tao_oat 5y ago
    To be clear, stopped supporting robots.txt noindex a few years ago. Combined with the fact that Google might list your site [based only on third-party links][1], robots.txt isn't an effective way to remove your site from Google's results. Sorry, could have been clearer. [1]: https://developers.google.com/search/docs/advanced/robots/intro https://developers.google.com/search/docs/advanced/robots/in...
  • dd82 5y ago
    >noindex in robots meta tags: Supported both in the HTTP response headers and in HTML, the noindex directive is the most effective way to remove URLs from the index when crawling is allowed. Seems clear enough to me
  • jen20 5y ago
    Quote from the linked article: “ For those of you who relied on the noindex indexing directive in the robots.txt file, which controls crawling, there are a number of alternative options:” The first option is the meta tag. It does mention an alternative directive for robots.txt, however.
    • ghassanmas 5y ago
      What about the blocking google bot by their IPs, also combined with user-agent wouldn't that stop the crawlers Google crawlers IPs https://www.lifewire.com/what-is-the-ip-address-of-google-818153 https://www.lifewire.com/what-is-the-ip-address-of-google-81...
      • benatkin 5y ago
        That will stop the crawlers but you could still show up in the search results, because of other web pages. From GP: > If other pages point to your page with descriptive text, Google could still index the URL without visiting the page
    • intricatedetail 5y ago
      Meta tag implies robot is going to fetch the page, effectively at very least using up you bandwidth. It should be illegal.
  • gnabgib 5y ago
    This page has a little more detail: https://developers.google.com/search/docs/advanced/crawling/remove-information#i-control-the-web-page https://developers.google.com/search/docs/advanced/crawling/... "If other pages point to your page with descriptive text, Google could still index the URL without visiting the page. If you want to block your page from search results, use another method such as password protection or noindex. "
• Animats 5y ago
  Did you think that mighty Google would pay attention to your puny "noindex" tag? Ha!
  • deleted 5y ago
    [deleted]
  • thih9 5y ago
    According to google's own docs, this should work. > You can prevent a page from appearing in Google Search by including a noindex meta tag in the page's HTML code, or by returning a noindex header in the HTTP response. Source: https://developers.google.com/search/docs/advanced/crawling/block-indexing https://developers.google.com/search/docs/advanced/crawling/...
    • bryanrasmussen 5y ago
      I mean technically that says that your site won't appear in search results, not that your site won't be used to profile people, determine other site ratings based on your site's content etc. they won't show your site's content, but that doesn't mean they won't use your site's content.
      • thih9 5y ago
        I thought that (i.e. removing the site from google search) was the goal. I'd review the other usage on a case by case basis; e.g. determining ratings of other sites seems fair use to me. I'd guess you're allowing others to use your site's content when you're making your site public (TINLA).
        • bryanrasmussen 5y ago
          maybe, but I guess I would be cantankerous enough to see the goal as preventing google from profiting off your site.
          • deleted 5y ago