3 ms·
Do all of them ignore robots.txt?
by chalst 15y ago
Do all of them ignore robots.txt?
- sbierwagen 15y agoWell, when I iterated through the everything2.com namespace and downloaded the majority of their content[1], I respected their robots.txt. After I did that, they changed it to: User-agent: * Disallow: / Does that count? 1: http://bbot.org/blog/archives/2011/01/17/more_fun_with_wget/ http://bbot.org/blog/archives/2011/01/17/more_fun_with_wget/