3 ms·

> search engines that explicitly dishonor the spirit and purpose of instructions not to crawl pages If you believe that to be true, then you should be arguing

> search engines that explicitly dishonor the spirit and purpose of instructions not to crawl pages If you believe that to be true, then you should be arguing for a solution to that problem. A robots noindex meta would work perfectly well to suppress an online bankruptcy notice about an individual.

benologist 12y ago
In that video you'll see Matt Cutts from Google's Web Spam team explaining how they've declared indexing a url on google, and crawling a url's content, are apparently separate things so your express wishes can be overridden by any third party linking to your content, resulting in the URL being on google search and in results with inferred content instead of spidered content. I think this is a problem, but this is not the entirety of the right to be forgotten problem, which just makes what you proposed more efficient because people only have to contact one company with an official procedure.
- ademarre 12y ago
  Crawling and indexing truly are fundamentally different things, and search engines have mechanisms to block both. https://support.google.com/webmasters/answer/93710?hl=en https://support.google.com/webmasters/answer/93710?hl=en > You can prevent a page from appearing in Google Search by including a noindex meta tag in the page's HTML code. When Googlebot next crawls that page, Googlebot will see the noindex meta tag and will drop that page entirely from Google Search results, regardless of whether other sites link to it.
  - benologist 12y ago
    For all intents and purposes "indexing" and "crawling" is "appearing in search results", and they conflate these terms themselves in their robots.txt page: > "You only need a robots.txt file if your site includes content that you don't want Google or other search engines to index." https://support.google.com/webmasters/answer/6062608?hl=en https://support.google.com/webmasters/answer/6062608?hl=en They do say noindex overrides 3rd party links so that's something, but it's a moot point either way because this was the approach for the last decade. Having to contact each individual person in charge of each individual URL isn't even effective when the MPAA throws money+DMCA at the task, individuals asking nicely never had a chance.

benologist 12y ago

In that video you'll see Matt Cutts from Google's Web Spam team explaining how they've declared indexing a url on google, and crawling a url's content, are apparently separate things so your express wishes can be overridden by any third party linking to your content, resulting in the URL being on google search and in results with inferred content instead of spidered content. I think this is a problem, but this is not the entirety of the right to be forgotten problem, which just makes what you proposed more efficient because people only have to contact one company with an official procedure.

ademarre 12y ago
Crawling and indexing truly are fundamentally different things, and search engines have mechanisms to block both. https://support.google.com/webmasters/answer/93710?hl=en https://support.google.com/webmasters/answer/93710?hl=en > You can prevent a page from appearing in Google Search by including a noindex meta tag in the page's HTML code. When Googlebot next crawls that page, Googlebot will see the noindex meta tag and will drop that page entirely from Google Search results, regardless of whether other sites link to it.
- benologist 12y ago
  For all intents and purposes "indexing" and "crawling" is "appearing in search results", and they conflate these terms themselves in their robots.txt page: > "You only need a robots.txt file if your site includes content that you don't want Google or other search engines to index." https://support.google.com/webmasters/answer/6062608?hl=en https://support.google.com/webmasters/answer/6062608?hl=en They do say noindex overrides 3rd party links so that's something, but it's a moot point either way because this was the approach for the last decade. Having to contact each individual person in charge of each individual URL isn't even effective when the MPAA throws money+DMCA at the task, individuals asking nicely never had a chance.