3 ms·
In my experience lxml.html is much better.
by wslh 14y ago
In my experience lxml.html is much better.
- d0mine 14y agoThe article shows that BeautifulSoup can use lxml internally. It can also use html5lib.
- thibauts 14y agoI'll second, lxml.html is in my experience very robust and fast. I've been writing a lot of scrapers along the years and the best combination I found so far is requests / lxml.html / gevent. It doesn't get any simpler than this IMO http://pastebin.com/hacxmAjV http://pastebin.com/hacxmAjV
- danneu 14y agoFor fun here's the Ruby+Nokogiri version and my attempt at the Clojure+Enlive version (3rd day learning Clojure). https://gist.github.com/danneu/5131596 https://gist.github.com/danneu/5131596
- thibauts 14y agoPretty nice. I'd like to see how it goes in a real world scenario with concurrency, manipulation of extracted nodes and general HTTP post/auth/etc. I'm not sure about Ruby+Nokogiri but Clojure+Enlive may well be a great choice.