I haven't run the crawler so I'm not sure what else it does, but if it only parses the home page and fetches the external links, why not read http://news.ycombi

I haven't run the crawler so I'm not sure what else it does, but if it only parses the home page and fetches the external links, why not read http://news.ycombinator.com/rss http://news.ycombinator.com/rss (you can use the feedparser module) and download the pages with urllib? No scraping involved.