lxml.HTML (+ html5lib if needed) is a FAR superior choice indeed. It's an order of magnitude faster, and you can use not only xpath selectors, but also CSS sele

lxml.HTML (+ html5lib if needed) is a FAR superior choice indeed. It's an order of magnitude faster, and you can use not only xpath selectors, but also CSS selectors too, from a lxml dom. or indeed you can use Scrapy, which fixes all these encoding BS, handles per domain crawl rate and concurrent requests etc...