I tried 80legs a couple of weeks ago when it was still in beta. It works well if you're willing to write Java code to process HTTP responses (see http://80legs.

I tried 80legs a couple of weeks ago when it was still in beta. It works well if you're willing to write Java code to process HTTP responses (see http://80legs.pbworks.com/80Apps http://80legs.pbworks.com/80Apps). You can issue new requests in your processing code and write processed data to files which you can download once the crawl finishes. Even without Java, you can perform basic keyword/regex matching on crawled data literally by filling out a web form. And they have an API of course. Very impressive. However, if you want to run crawling on your own infrastructure, I recommend Scrapy (http://scrapy.org/ http://scrapy.org/), a python crawling framework introduced on HN last year. Scrapy solves some of the more time-consuming problems involved in writing a crawler from scratch (multiple simultaneous requests, pipelined processing, raw caching, duplicate URL filtering) and comes with nifty development and administration tools. More importantly, it has an active and helpful set of core developers and good documentation. I am comfortable with both Python and Java but I chose Scrapy over 80legs because I can crawl for free on the machines I already have and I can afford to spend more time crawling from a single IP compared to 80legs which will let me crawl much faster but isn't free. Also, with Scrapy my bot can be 'naughty' - 80legs jobs obey robots.txt and limit the crawling rate per domain. When my crawling needs outgrow my infrastructure, I am going to look at 80legs again.