3 ms·
the Archive Team are bloody-minded about preserving information They won't do a good job of that if they encourage too many people to make IP blocks of their m
by chalst 15y ago
the Archive Team are bloody-minded about preserving information
They won't do a good job of that if they encourage too many people to make IP blocks of their machines.
- sbierwagen 15y agoAT, the organization, doesn't own any machines. Operations are all done by members, the majority of whom use consumer ISPs.
- chalst 15y agoDo all of them ignore robots.txt?
- sbierwagen 15y agoWell, when I iterated through the everything2.com namespace and downloaded the majority of their content[1], I respected their robots.txt. After I did that, they changed it to: User-agent: * Disallow: / Does that count? 1: http://bbot.org/blog/archives/2011/01/17/more_fun_with_wget/ http://bbot.org/blog/archives/2011/01/17/more_fun_with_wget/