I have also found node+jQuery an effective web crawling combination. In particular the cheerio library https://github.com/MatthewMueller/cheerio https://github.

I have also found node+jQuery an effective web crawling combination. In particular the cheerio library https://github.com/MatthewMueller/cheerio https://github.com/MatthewMueller/cheerio greatly simplifies data extraction. And as others have mentioned, the asynchronous nature of node is perfectly suited to crawling (as long as you take care not to accidentally DDOS the target site).