use beautiful soup (best python scraping system I know of). Maybe combine it with mechanize for navigating between the pages. Don't try to create your own patte

use beautiful soup (best python scraping system I know of). Maybe combine it with mechanize for navigating between the pages. Don't try to create your own pattern-matching file. Write a generic crawler class, subclass it for each site. In the end you should just need to write a couple of short site-specific functions for each site. It doesn't take long once you get going - that is, until you run into sites that are unnavigable piles of javascript and unstructured html.