> would you be better off using a search engine crawler like Nutch for this? Fwiw we recently met a client who tried Scrapy + Frontera vs Nutch, and their asse

> would you be better off using a search engine crawler like Nutch for this? Fwiw we recently met a client who tried Scrapy + Frontera vs Nutch, and their assessment was that Scrapy + Frontera is twice as fast. Here's a deck on Frontera FYI: http://www.slideshare.net/scrapinghub/frontera-open-source-large-scale-web-crawling-framework http://www.slideshare.net/scrapinghub/frontera-open-source-l... Aside: don't hesitate to get in touch with our sales team if you need help. We're experienced in this type of project.