Nutch is good and I would second it, but I would suggest to NOT build a crawler - its not trivial and inadvised in a startup, that is unless your startup is jus

Nutch is good and I would second it, but I would suggest to NOT build a crawler - its not trivial and inadvised in a startup, that is unless your startup is just about building a crawler.