4 ms·
The Internet Archive should implement some sort of digital signature system to allow website owners with foresight to prevent this.
by TazeTSchnitzel 12y ago
The Internet Archive should implement some sort of digital signature system to allow website owners with foresight to prevent this.
- toyg 12y agoThey could just timestamp different versions of robots.txt (which they probably do already), and respect it depending on date (which is more of a hassle, because you have to build it in your UI logic).
- nathanb 12y agoThat would not solve the problem they're trying to solve. Let's say I post something that I shouldn't have posted -- insider stock information, nude photos, whatever. Perhaps something illegal for me to post. I need to make it go away. I need to be able to create a robots.txt today which affects stuff I posted yesterday. This is why archive.org respects the current robots.txt for access to past content.