If you're looking at building your own crawler in Python from scratch, here's a benchmark of SGML parsers: http://72.14.205.104/search?q=cache:LYoRD1GTP2UJ:www

If you're looking at building your own crawler in Python from scratch, here's a benchmark of SGML parsers: http://72.14.205.104/search?q=cache:LYoRD1GTP2UJ:www.oluyede.org/blog/2007/08/25/sgml-python-parsers-benchmark/+benchmark+python+parsers&hl=en&ct=clnk&cd=1&gl=ca http://72.14.205.104/search?q=cache:LYoRD1GTP2UJ:www.oluyede... We've been playing with sgmlop (http://effbot.org/zone/sgmlop-index.htm http://effbot.org/zone/sgmlop-index.htm) for parsing and urllib2 (http://docs.python.org/lib/module-urllib2.html http://docs.python.org/lib/module-urllib2.html) for fetching.