6 ms·Scrapy can do the job, for sure. We use it to crawl more than 2 billion pages a month. :)by stummjr 11y agoScrapy can do the job, for sure. We use it to crawl more than 2 billion pages a month. :)