There a quite a few open source web crawlers, in Java, PHP, etc.. Based on what you mentioned, Nutch is alright... But not sure how are you going to get all tho

There a quite a few open source web crawlers, in Java, PHP, etc.. Based on what you mentioned, Nutch is alright... But not sure how are you going to get all those unstructured information out and make it structured.