5 ms·

Google has been very clear lately (via John Mueller) regarding getting pages indexed or removed from the index. If you want to make sure a URL is not in their

Google has been very clear lately (via John Mueller) regarding getting pages indexed or removed from the index. If you want to make sure a URL is not in their index then you have to 'allow' them to crawl the page in robots.txt and use a noindex meta tag on the page to stop indexing. Simply disallowing the page from being crawled in robots.txt will not keep it out of the index. In fact, I've seen plenty of pages still rank well despite the page being disallowed in robots.txt. A great example of this is the keyword "backpack" in Google. You'll see the site doesn't want it indexed (it's disallowed in robots.txt) but the site still ranks well for a popular keyword).

LunaSea 7y ago
Doesn't that indicate that Google doesn't respect robots.txt then?
- C1sc0cat 7y ago
  Robots stops a page from being crawled the noindex tag stops it getting into the index. Google is also slow to honour 404 and drop pages which can hang around for ages, Bing is much faster to remove 404 pages.
  - rafaelm 7y ago
    Usually, creating "410 Gone"[0] response for the URL and running the URL through the URL Inspection Tool [1] can help make things a bit faster. But yeah, it does take a while to get these 404s removed. [0] https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/410 https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/410 [1] https://support.google.com/webmasters/answer/6065812?hl=en https://support.google.com/webmasters/answer/6065812?hl=en
    - inlined 7y ago
      That distinction exists in many systems. E.g. for cloud events, 404 is considered with skepticism because it could be a race condition in provisioning or transient issue whereas 410 requires data streams to be cut off.
      - C1sc0cat 7y ago
        Then you should serve 503
        inlined 7y ago
        5xx means that the server made a mistake. 4xx means that the caller made a mistake. Sending a request to a GONE url is canonically classified as a “user” or sender error.
- bryanrasmussen 7y ago
  No, disallow means that you are not allowed to crawl the page. You have to crawl the page to no you cannot index it. But how do you index it if you do not crawl the page, well if another page that you can crawl and index points to the page you cannot index as authoritative on a keyword then it be in the index with that keyword, even if you do not have the actual crawled content of the page.
  - henryfjordan 7y ago
    It really feels like they need to allow a list of `noindex` pages in the robots.txt then...
    - wayoutthere 7y ago
      The whole point is they don’t want you to easily opt out of it.
      - dredmorbius 7y ago
        Keep walking down that path to find v. heavy regulation.
        wayoutthere 7y ago
        Not if you buy the government first. I think they’re already a victim of their own greed though. It’s bad enough I started using DDG for search because the results are now more relevant. Google’s advertising algorithms are designed to subtly nudge sites into paying for placement — which means there’s a “non-content” element to the search results that makes it into the user experience. I feel like there was a tipping point a year or two ago where the results just stopped being useful — The best analogy I can find is how search engines used to be in the days before AltaVista. Then AltaVista came out and the results were far more relevant (if not perfect). Google -> DDG feels like that in 2019. That “non-content” element will only grow over time as Google seeks revenue growth — growth across all of Google’s non-advertising revenue streams combined are not enough to move the needle compared to the scale their ad business has — of which search ads are by far the most profitable. So they will further try to monetize search; it’s their cash cow but I think a small player like DDG could easily overtake them as the quality of Google’s search results (to the end user) continue to decline.
rafaelm 7y ago
That's correct. If a URL is blocked using robots.txt, Google will never be able to see the "noindex" tag on the page. URLs blocked in robots.txt can get discovered through other links and they will get displayed in the search results. However, you will not see any information like the meta description on these blocked URLs. There's a good explanation about this here, including a video from former Googler, Matt Cutts: https://yoast.com/prevent-site-being-indexed/ https://yoast.com/prevent-site-being-indexed/
- luckylion 7y ago
  > However, you will not see any information like the meta description on these blocked URLs. True, but that's not the only thing. If it ever was in the index, it takes forever to be removed, if it gets removed at all. Send 404 or 410, Disallow it or set it to noindex - you may get lucky or you may not. You can of course "hide it from search results", but that only works for 90 days (iirc, may be 120, something in that range). Those leftovers will typically lose rankings, but they often stay indexed, easy to spot with a site: query.
  - inlined 7y ago
    Reindexing a page is dynamic based on noteworthiness and volatility iirc, but individual links can be reindexed on the fly since the Percolator index. The 90d number was from an old system when indexes were broken into shards that had to be swapped out wholesale. Percolator white paper: https://ai.google/research/pubs/pub36726 https://ai.google/research/pubs/pub36726
    - luckylion 7y ago
      I don't mean reindexing, I mean "hiding from the index" ("Remove URLs" in GSC). It works instantly, but only for a limited time, after which it will re-appear in the index if you haven't gotten it out of the index (via 410, noindex or disallow). Since these other ways don't always work, if you're unlucky and want it to stay gone, you need to hide it again (and again and again). I've had clients that were hacked and had spammy content injected into their site and it took (literally!) years for that to get removed (we tried combinations of 404, 410, noindex and disallow).
areyousure 7y ago
> You'll see the site doesn't want it indexed (it's disallowed in robots.txt) but the site still ranks well for a popular keyword). Which site? [Edit: I have now found https://www.gcsbackpack.com/ https://www.gcsbackpack.com/ on page 6 of the results, and this was presumably the intended site.]

LunaSea 7y ago

Doesn't that indicate that Google doesn't respect robots.txt then?

C1sc0cat 7y ago
Robots stops a page from being crawled the noindex tag stops it getting into the index. Google is also slow to honour 404 and drop pages which can hang around for ages, Bing is much faster to remove 404 pages.
- rafaelm 7y ago
  Usually, creating "410 Gone"[0] response for the URL and running the URL through the URL Inspection Tool [1] can help make things a bit faster. But yeah, it does take a while to get these 404s removed. [0] https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/410 https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/410 [1] https://support.google.com/webmasters/answer/6065812?hl=en https://support.google.com/webmasters/answer/6065812?hl=en
  - inlined 7y ago
    That distinction exists in many systems. E.g. for cloud events, 404 is considered with skepticism because it could be a race condition in provisioning or transient issue whereas 410 requires data streams to be cut off.
    - C1sc0cat 7y ago
      Then you should serve 503
      - inlined 7y ago
        5xx means that the server made a mistake. 4xx means that the caller made a mistake. Sending a request to a GONE url is canonically classified as a “user” or sender error.
bryanrasmussen 7y ago
No, disallow means that you are not allowed to crawl the page. You have to crawl the page to no you cannot index it. But how do you index it if you do not crawl the page, well if another page that you can crawl and index points to the page you cannot index as authoritative on a keyword then it be in the index with that keyword, even if you do not have the actual crawled content of the page.
- henryfjordan 7y ago
  It really feels like they need to allow a list of `noindex` pages in the robots.txt then...
  - wayoutthere 7y ago
    The whole point is they don’t want you to easily opt out of it.
    - dredmorbius 7y ago
      Keep walking down that path to find v. heavy regulation.
      - wayoutthere 7y ago
        Not if you buy the government first. I think they’re already a victim of their own greed though. It’s bad enough I started using DDG for search because the results are now more relevant. Google’s advertising algorithms are designed to subtly nudge sites into paying for placement — which means there’s a “non-content” element to the search results that makes it into the user experience. I feel like there was a tipping point a year or two ago where the results just stopped being useful — The best analogy I can find is how search engines used to be in the days before AltaVista. Then AltaVista came out and the results were far more relevant (if not perfect). Google -> DDG feels like that in 2019. That “non-content” element will only grow over time as Google seeks revenue growth — growth across all of Google’s non-advertising revenue streams combined are not enough to move the needle compared to the scale their ad business has — of which search ads are by far the most profitable. So they will further try to monetize search; it’s their cash cow but I think a small player like DDG could easily overtake them as the quality of Google’s search results (to the end user) continue to decline.

rafaelm 7y ago

That's correct. If a URL is blocked using robots.txt, Google will never be able to see the "noindex" tag on the page. URLs blocked in robots.txt can get discovered through other links and they will get displayed in the search results. However, you will not see any information like the meta description on these blocked URLs. There's a good explanation about this here, including a video from former Googler, Matt Cutts: https://yoast.com/prevent-site-being-indexed/ https://yoast.com/prevent-site-being-indexed/

luckylion 7y ago
> However, you will not see any information like the meta description on these blocked URLs. True, but that's not the only thing. If it ever was in the index, it takes forever to be removed, if it gets removed at all. Send 404 or 410, Disallow it or set it to noindex - you may get lucky or you may not. You can of course "hide it from search results", but that only works for 90 days (iirc, may be 120, something in that range). Those leftovers will typically lose rankings, but they often stay indexed, easy to spot with a site: query.
- inlined 7y ago
  Reindexing a page is dynamic based on noteworthiness and volatility iirc, but individual links can be reindexed on the fly since the Percolator index. The 90d number was from an old system when indexes were broken into shards that had to be swapped out wholesale. Percolator white paper: https://ai.google/research/pubs/pub36726 https://ai.google/research/pubs/pub36726
  - luckylion 7y ago
    I don't mean reindexing, I mean "hiding from the index" ("Remove URLs" in GSC). It works instantly, but only for a limited time, after which it will re-appear in the index if you haven't gotten it out of the index (via 410, noindex or disallow). Since these other ways don't always work, if you're unlucky and want it to stay gone, you need to hide it again (and again and again). I've had clients that were hacked and had spammy content injected into their site and it took (literally!) years for that to get removed (we tried combinations of 404, 410, noindex and disallow).

areyousure 7y ago

> You'll see the site doesn't want it indexed (it's disallowed in robots.txt) but the site still ranks well for a popular keyword). Which site? [Edit: I have now found https://www.gcsbackpack.com/ https://www.gcsbackpack.com/ on page 6 of the results, and this was presumably the intended site.]