Related anecdote: last web a web server of mine was under very heavy load. The site was flapping and I could barely even SSH to the host. I did some basic analy

Related anecdote: last web a web server of mine was under very heavy load. The site was flapping and I could barely even SSH to the host. I did some basic analysis of the httpd logs and found that there were 13,000 unique IPs within the past week and 12,000 belonged to one subnet on Amazon EC2. The user agent for all those requests was ByteSpider. Millions of requests over a few days. I'm into freedom of information and what not but come on!!!