4 ms·

Crawling and indexing truly are fundamentally different things, and search engines have mechanisms to block both. https://support.google.com/webmasters/answer/

Crawling and indexing truly are fundamentally different things, and search engines have mechanisms to block both. https://support.google.com/webmasters/answer/93710?hl=en https://support.google.com/webmasters/answer/93710?hl=en > You can prevent a page from appearing in Google Search by including a noindex meta tag in the page's HTML code. When Googlebot next crawls that page, Googlebot will see the noindex meta tag and will drop that page entirely from Google Search results, regardless of whether other sites link to it.

benologist 12y ago
For all intents and purposes "indexing" and "crawling" is "appearing in search results", and they conflate these terms themselves in their robots.txt page: > "You only need a robots.txt file if your site includes content that you don't want Google or other search engines to index." https://support.google.com/webmasters/answer/6062608?hl=en https://support.google.com/webmasters/answer/6062608?hl=en They do say noindex overrides 3rd party links so that's something, but it's a moot point either way because this was the approach for the last decade. Having to contact each individual person in charge of each individual URL isn't even effective when the MPAA throws money+DMCA at the task, individuals asking nicely never had a chance.