Aside: The last time I had to scrape a lot of data from the web, I additionally used SQLite, which is a breeze to use with Python (basically one import statemen

Aside: The last time I had to scrape a lot of data from the web, I additionally used SQLite, which is a breeze to use with Python (basically one import statement and you're set). It might be overkill for some cases, but I found it a huge boon for keeping track of which pages were scraped, which failed, and doing subsequent data processing and parsing "offline". It made it so much easier to recover from the inevitable random error or different markup somewhere deep in your list of pages to scrape etc.