I use Scrapy: http://scrapy.org/ http://scrapy.org/ another Python option: http://twill.idyll.org/ http://twill.idyll.org/ for Rails: http://nokogiri.org/ http:

I use Scrapy: http://scrapy.org/ http://scrapy.org/ another Python option: http://twill.idyll.org/ http://twill.idyll.org/ for Rails: http://nokogiri.org/ http://nokogiri.org/ and perl: http://wwwsearch.sourceforge.net/mechanize/ http://wwwsearch.sourceforge.net/mechanize/