Crawling too fast from the same IP addy can get that IP blocked. Per 10 mins prob ok, per 1 minute may be too much depending on the volume being downloaded and

Crawling too fast from the same IP addy can get that IP blocked. Per 10 mins prob ok, per 1 minute may be too much depending on the volume being downloaded and the sensitivity (you mention saleable stuff in another comment so this might well make a difference). FYI. I did a reasonable amount of scraping myself. (err, crawling = scraping? or maybe they different things) Edit: tidy + clarify