You'd be surprised how difficult it is to create an extremely fast, scalable architecture for crawling web sites. Took me a few months, anyway (and would probab

You'd be surprised how difficult it is to create an extremely fast, scalable architecture for crawling web sites. Took me a few months, anyway (and would probably take a few more to make it "fill in a web form and go" usable). After that, the processing is pretty straight forward. You've got a URL, headers, and body content. Just extract what you want and crunch.