You may be interested in Nutch (http://lucene.apache.org/nutch/ http://lucene.apache.org/nutch/), an Open Source crawler, which handles indexing etc. for you.

You may be interested in Nutch (http://lucene.apache.org/nutch/ http://lucene.apache.org/nutch/), an Open Source crawler, which handles indexing etc. for you. Its also based on Hadoop, so it should scale nicely just by throwing more machines at the job.