3 ms·I would add Scrapy (http://scrapy.org/ http://scrapy.org/) for web crawling.by lunchbox 17y agoI would add Scrapy (http://scrapy.org/ http://scrapy.org/) for web crawling.