Those are some interesting exclusions, yeah: "" Disallow: /2013/03/21/174840895/ Disallow: /sections/ombudsman/2008/01/frequently_asked_questions_1.html Disa

Those are some interesting exclusions, yeah: "" Disallow: /2013/03/21/174840895/ Disallow: /sections/ombudsman/2008/01/frequently_asked_questions_1.html Disallow: /sections/health-shots/2013/03/11/173816690/new-voices-for-the-voiceless-synthetic-speech-gets-an-upgrade "" Removing something from search engine result pages but not from archives seems like a middle ground that media organizations might consider when people say "I regret having talked on the record about this, can you remove it?" or when they say "hey, this is hurting my reputation and it's not a big deal, can you please make it harder to find?". I used to periodically poll The Harvard Crimson's robots.txt which until a 2015 site redesign used to exclude a lot of articles from search engine results. One example of an article excluded from search engine results for several years was http://www.thecrimson.com/article/2006/10/31/news-in-brief-student-charged-with/ http://www.thecrimson.com/article/2006/10/31/news-in-brief-s... , which also carries a lengthy editor's note with more info about the final disposition of the case and which I think I can see why the article subject might not want to affect their reputation. Among the other things that the Crimson's web site excluded from search results included -- drug arrest stories where the final disposition was "not guilty" and the final article was not hidden (but the original arrest and intermediate reporting were); some deeply personal sex columns; maybe an unruly party story; a story about a grad student being rude at a football game. I wasn't always able to find the articles -- one older piece of content, article id 523203 in their old-style format, was probably published in the print edition on 22 April 2008 based on the publication dates of article ids 523202 and 523204. The Crimson seems to have stopped doing this and most media organizations don't do it; while excluding an article from robots.txt usually makes users' search engine results calm down, sometimes it can have the opposite effect by broadcasting exactly what in the archives people do not want to be seen.