3 ms·
The very first suggestion complains that BeautifulSoup is too slow, but as of version 4, it's actually just a navigation layer on top of your preferred parser.
by takluyver 14y ago
The very first suggestion complains that BeautifulSoup is too slow, but as of version 4, it's actually just a navigation layer on top of your preferred parser. So it's as fast as lxml, and as easy to use as, well, BeautifulSoup.
- kami8845 14y agoHey before BS 4 got released I looked forward to the very same thing you're talking about, using BS on top of lxml. But it seems the performance hasn't really improved: http://www.crummy.com/2012/1/22/0 http://www.crummy.com/2012/1/22/0
- vivek-infosec 14y agoA DOM parser will always be slower than a stream parser ...
- pmorici 14y agoI used BeautifulSoup for a project once because of all the accolades it gets here but found it to be less than robust. It might be sufficient in a scenario where you have a single site you need to scrap but I found it was totally unreliable when used across a wide range of sites and esp. sites with foreign language content.
- ludwigvan 14y agoI started using PyQuery yesterday, after using BeautifulSoup for a long time. It seems much easier to use. pq_page = pquery(url=PAGE_URL) Note that PyQuery has some encoding issues too (or rather the sites I were scraping were too bad, showing two different encodings in meta tag!), here are two different things I have done to workaround: page = requests.get(PAGE_URL) pq_page = pquery(page.text) If that doesn't cut it (because requests detects it wrong too), try forcing the encoding in requests: page = requests.get(PAGE_URL) page.encoding = 'utf-8' pq_page = pquery(page.text)