3 ms·

Am I too late to point out that they have an article about sexual violence in the military[1] disallowed in their robots text[2]. I noticed this last summer and

by 3steve 8y ago

Am I too late to point out that they have an article about sexual violence in the military[1] disallowed in their robots text[2]. I noticed this last summer and can't find a reason why, using Wayback Machine you can see it was added a while after publishing. [1]: www.npr.org/2013/03/21/174840895/sexual-violence-victims-say-military-justice-system-is-broken [2]: www.npr.org/robots.txt

mherdeg 8y ago
Those are some interesting exclusions, yeah: "" Disallow: /2013/03/21/174840895/ Disallow: /sections/ombudsman/2008/01/frequently_asked_questions_1.html Disallow: /sections/health-shots/2013/03/11/173816690/new-voices-for-the-voiceless-synthetic-speech-gets-an-upgrade "" Removing something from search engine result pages but not from archives seems like a middle ground that media organizations might consider when people say "I regret having talked on the record about this, can you remove it?" or when they say "hey, this is hurting my reputation and it's not a big deal, can you please make it harder to find?". I used to periodically poll The Harvard Crimson's robots.txt which until a 2015 site redesign used to exclude a lot of articles from search engine results. One example of an article excluded from search engine results for several years was http://www.thecrimson.com/article/2006/10/31/news-in-brief-student-charged-with/ http://www.thecrimson.com/article/2006/10/31/news-in-brief-s... , which also carries a lengthy editor's note with more info about the final disposition of the case and which I think I can see why the article subject might not want to affect their reputation. Among the other things that the Crimson's web site excluded from search results included -- drug arrest stories where the final disposition was "not guilty" and the final article was not hidden (but the original arrest and intermediate reporting were); some deeply personal sex columns; maybe an unruly party story; a story about a grad student being rude at a football game. I wasn't always able to find the articles -- one older piece of content, article id 523203 in their old-style format, was probably published in the print edition on 22 April 2008 based on the publication dates of article ids 523202 and 523204. The Crimson seems to have stopped doing this and most media organizations don't do it; while excluding an article from robots.txt usually makes users' search engine results calm down, sometimes it can have the opposite effect by broadcasting exactly what in the archives people do not want to be seen.
SiempreViernes 8y ago
Yes, others have already pointed out this entirely unrelated and boring configuration mystery about a 5 year old story.
- slim 8y ago
  It's not boring

by 3steve 8y ago

mherdeg 8y ago

Those are some interesting exclusions, yeah: "" Disallow: /2013/03/21/174840895/ Disallow: /sections/ombudsman/2008/01/frequently_asked_questions_1.html Disallow: /sections/health-shots/2013/03/11/173816690/new-voices-for-the-voiceless-synthetic-speech-gets-an-upgrade "" Removing something from search engine result pages but not from archives seems like a middle ground that media organizations might consider when people say "I regret having talked on the record about this, can you remove it?" or when they say "hey, this is hurting my reputation and it's not a big deal, can you please make it harder to find?". I used to periodically poll The Harvard Crimson's robots.txt which until a 2015 site redesign used to exclude a lot of articles from search engine results. One example of an article excluded from search engine results for several years was http://www.thecrimson.com/article/2006/10/31/news-in-brief-student-charged-with/ http://www.thecrimson.com/article/2006/10/31/news-in-brief-s... , which also carries a lengthy editor's note with more info about the final disposition of the case and which I think I can see why the article subject might not want to affect their reputation. Among the other things that the Crimson's web site excluded from search results included -- drug arrest stories where the final disposition was "not guilty" and the final article was not hidden (but the original arrest and intermediate reporting were); some deeply personal sex columns; maybe an unruly party story; a story about a grad student being rude at a football game. I wasn't always able to find the articles -- one older piece of content, article id 523203 in their old-style format, was probably published in the print edition on 22 April 2008 based on the publication dates of article ids 523202 and 523204. The Crimson seems to have stopped doing this and most media organizations don't do it; while excluding an article from robots.txt usually makes users' search engine results calm down, sometimes it can have the opposite effect by broadcasting exactly what in the archives people do not want to be seen.

SiempreViernes 8y ago

Yes, others have already pointed out this entirely unrelated and boring configuration mystery about a 5 year old story.

slim 8y ago
It's not boring