+10 for Python +1 for Nutch +1 for Hadoop +1 for Amazon EC2 I think you have bundled two things (crawler and parser (or may be scraper)) into one term : crawle

+10 for Python +1 for Nutch +1 for Hadoop +1 for Amazon EC2 I think you have bundled two things (crawler and parser (or may be scraper)) into one term : crawler. Beautiful soup is ok. Give html5lib a try (on google code) - but at some point you are going to have to hack the parser, but that depends on what kind of parsing you want to do.