2 ms·
They could just timestamp different versions of robots.txt (which they probably do already), and respect it depending on date (which is more of a hassle, becaus
by toyg 12y ago
They could just timestamp different versions of robots.txt (which they probably do already), and respect it depending on date (which is more of a hassle, because you have to build it in your UI logic).
- nathanb 12y agoThat would not solve the problem they're trying to solve. Let's say I post something that I shouldn't have posted -- insider stock information, nude photos, whatever. Perhaps something illegal for me to post. I need to make it go away. I need to be able to create a robots.txt today which affects stuff I posted yesterday. This is why archive.org respects the current robots.txt for access to past content.