Rather than emoticon-ing away your responsibility in a potential site DoS, perhaps read up on responsible crawling - https://webarchive.jira.com/wiki/display/He

Rather than emoticon-ing away your responsibility in a potential site DoS, perhaps read up on responsible crawling - https://webarchive.jira.com/wiki/display/Heritrix/Responsible+Crawling https://webarchive.jira.com/wiki/display/Heritrix/Responsibl... + http://blog.mischel.com/2011/12/20/writing-a-web-crawler-politeness/ http://blog.mischel.com/2011/12/20/writing-a-web-crawler-pol... . You had no need for either unlimited or even 50 parallel crawl tasks. And, if your intent is anything but personal use, you shld prbly (re)read https://www.brewtoad.com/legal https://www.brewtoad.com/legal since you just got their attention in a pretty big way.