3 ms·

Robots stops a page from being crawled the noindex tag stops it getting into the index. Google is also slow to honour 404 and drop pages which can hang around

Robots stops a page from being crawled the noindex tag stops it getting into the index. Google is also slow to honour 404 and drop pages which can hang around for ages, Bing is much faster to remove 404 pages.

rafaelm 7y ago
Usually, creating "410 Gone"[0] response for the URL and running the URL through the URL Inspection Tool [1] can help make things a bit faster. But yeah, it does take a while to get these 404s removed. [0] https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/410 https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/410 [1] https://support.google.com/webmasters/answer/6065812?hl=en https://support.google.com/webmasters/answer/6065812?hl=en
- inlined 7y ago
  That distinction exists in many systems. E.g. for cloud events, 404 is considered with skepticism because it could be a race condition in provisioning or transient issue whereas 410 requires data streams to be cut off.
  - C1sc0cat 7y ago
    Then you should serve 503
    - inlined 7y ago
      5xx means that the server made a mistake. 4xx means that the caller made a mistake. Sending a request to a GONE url is canonically classified as a “user” or sender error.