3 ms·

I wasn't aware. Can you please update Wikipedia then: https://en.wikipedia.org/wiki/Robots.txt https://en.wikipedia.org/wiki/Robots.txt Maybe also get Google t

• nottorp 2y ago
  It must be nice to believe everything people say by default... ;)
• phit_ 2y ago
  their own docs also specify that the robots.txt does not stop indexing or showing up in search, they even bolded it "it is not a mechanism for keeping a web page out of Google" https://developers.google.com/search/docs/crawling-indexing/robots/intro https://developers.google.com/search/docs/crawling-indexing/...
  • alphan0n 2y ago
    The only way for links to appear in a Google search would be to host a public resource, that is linked from another public resource. If you have specified in your robots.txt that you do not want the page(s) or directories ingested then only the url is indexed (if it is linked from another page). It does prevent the public display of the content of a page and creation description/summary. https://support.google.com/webmasters/answer/7489871?hl=en https://support.google.com/webmasters/answer/7489871?hl=en
  • threeseed 2y ago
    From the docs: “While Google won't crawl or index the content blocked by a robots.txt file” They will show the URL if someone else has linked to it. But the content itself is not indexed.