3 ms·
AT, the organization, doesn't own any machines. Operations are all done by members, the majority of whom use consumer ISPs.
by sbierwagen 16y ago
AT, the organization, doesn't own any machines. Operations are all done by members, the majority of whom use consumer ISPs.
- chalst 16y agoDo all of them ignore robots.txt?
- sbierwagen 16y agoWell, when I iterated through the everything2.com namespace and downloaded the majority of their content[1], I respected their robots.txt. After I did that, they changed it to: User-agent: * Disallow: / Does that count? 1: http://bbot.org/blog/archives/2011/01/17/more_fun_with_wget/ http://bbot.org/blog/archives/2011/01/17/more_fun_with_wget/