2 ms·
There's a forum here: https://archive.org/post/406632/why-does-the-wayback-machine-pay-attention-to-robotstxt https://archive.org/post/406632/why-does-the-wayb
by cogburnd02 12y ago
There's a forum here:
https://archive.org/post/406632/why-does-the-wayback-machine-pay-attention-to-robotstxt https://archive.org/post/406632/why-does-the-wayback-machine...
Maybe if it gets enough attention, and/or the Internet Archive people get enough e-mails about this problem (Wayback Machine obeying robots.txt) then they'll change their mind.
https://archive.org/about/bios.php https://archive.org/about/bios.php
Here's a link with the director's email:
http://brewster.kahle.org/about/ http://brewster.kahle.org/about/