3 ms·
Is removing content the same as disallowing it in a robots.txt? Because they have an interesting article [1] disallowed in their robots.txt that was added a whi
by 3steve 8y ago
Is removing content the same as disallowing it in a robots.txt? Because they have an interesting article [1] disallowed in their robots.txt that was added a while after it was published.
[1]:www.npr.org/2013/03/21/174840895/sexual-violence-victims-say-military-justice-system-is-broken
- throwawayjava 8y ago> Is removing content the same as disallowing it in a robots.txt? No. > Because they have an interesting article [1] disallowed in their robots.txt that was added a while after it was published. Oddly, it's a different seemingly benign story for me: Disallow: /sections/health-shots/2013/03/11/173816690/new-voices-for-the-voiceless-synthetic-speech-gets-an-upgrade IDK why they include a seemingly random story in their robots.txt. I don't think the intention is to hide the story, though.
- oh_sigh 8y agoI get the same link as you in there. I wonder if this is some honeypot technique for detecting misbehaving robots?