>>>You should always wait for several seconds (better yet, a minute) between requests to the same host. This a thousand times. I learned this lesson AFTER get

>>>You should always wait for several seconds (better yet, a minute) between requests to the same host. This a thousand times. I learned this lesson AFTER getting several angry emails from Admins and getting outright banned from one site for not having any delay between the requests in the first crawler I built.