3 ms·
That's exactly what they do. They are treating "robots.txt" as an "active legal tool". You could argue that they should make it harder, but I doubt they want to
by dTal 7y ago
That's exactly what they do. They are treating "robots.txt" as an "active legal tool". You could argue that they should make it harder, but I doubt they want to find themselves in court over every little thing. Better to make it easy to block to avoid liability, and also easy to allow that block to lapse - if they forced you to get a court order to hide information, then the information would be gone forever.
- mjevans 7y agoAn active legal tool might be as little as requiring an official claim of copyright and a related Cease and Desist to be mailed physically to them. That would prevent the issue of domain squatters with robots.txt files in most cases, and offer the public a means of challenging such ownership requests (and freeing up the content on archive.org) if they desire.