4 ms·
BeautifulSoup, has received a lot of positive press on HN over the years so when I needed to do some heavy scraping I gave it a spin. It was a total disappoint
by pmorici 14y ago
BeautifulSoup, has received a lot of positive press on HN over the years so when I needed to do some heavy scraping I gave it a spin. It was a total disappointment. It's fine if you are scraping a small set of similar pages from a single site but if you are scraping a large number of pages across many sites and esp pages with text encoding other than ascii / utf-8 it choaks so frequently as to be useless. BeautifulSoup is fine for small jobs but if you were making a web-crawler for example look elsewhere it is totally inadequate.
- wslh 14y agoIn my experience lxml.html is much better.
- d0mine 14y agoThe article shows that BeautifulSoup can use lxml internally. It can also use html5lib.
- thibauts 14y agoI'll second, lxml.html is in my experience very robust and fast. I've been writing a lot of scrapers along the years and the best combination I found so far is requests / lxml.html / gevent. It doesn't get any simpler than this IMO http://pastebin.com/hacxmAjV http://pastebin.com/hacxmAjV
- danneu 14y agoFor fun here's the Ruby+Nokogiri version and my attempt at the Clojure+Enlive version (3rd day learning Clojure). https://gist.github.com/danneu/5131596 https://gist.github.com/danneu/5131596
- thibauts 14y agoPretty nice. I'd like to see how it goes in a real world scenario with concurrency, manipulation of extracted nodes and general HTTP post/auth/etc. I'm not sure about Ruby+Nokogiri but Clojure+Enlive may well be a great choice.
- morganwatch 14y agowhat would you suggest?
- d0mine 14y ago> with text encoding other than ascii / utf-8 it choaks so frequently as to be useless I had the opposite experience though I used it only "for small jobs". If there are no Content-Type (either in http headers or meta http-equiv) that specifies character encoding, no meta charset, and no xml declaration for xhtml, etc that is if the only way to find out character encoding is to guess then even in this case BeautifulSoup includes UnicodeDammit that uses chardet to guess the encoding.
- Camillo 14y agoI don't see a use for BeautifulSoup nowadays. I use lxml.etree for everything (with the HTML parser when needed), and do 99% of queries using XPath. It's the best way to do scraping with Python, in my experience.
- steeve 14y agolxml.HTML (+ html5lib if needed) is a FAR superior choice indeed. It's an order of magnitude faster, and you can use not only xpath selectors, but also CSS selectors too, from a lxml dom. or indeed you can use Scrapy, which fixes all these encoding BS, handles per domain crawl rate and concurrent requests etc...