You could just use Scrapy [1]. Easy to setup, and plenty of options you can activate if needed. Likely more robust than shell scripts as well. No Hadoop involve

You could just use Scrapy [1]. Easy to setup, and plenty of options you can activate if needed. Likely more robust than shell scripts as well. No Hadoop involved. 1: http://doc.scrapy.org/en/0.24/intro/tutorial.html http://doc.scrapy.org/en/0.24/intro/tutorial.html