Scrapy (http://scrapy.org/ http://scrapy.org/) matches your description pretty well. You can specify which urls to crawl with regular expressions and then provi

Scrapy (http://scrapy.org/ http://scrapy.org/) matches your description pretty well. You can specify which urls to crawl with regular expressions and then provide a bit of code to do some data extraction.