2 ms·
I'll second, lxml.html is in my experience very robust and fast. I've been writing a lot of scrapers along the years and the best combination I found so far is
by thibauts 14y ago
I'll second, lxml.html is in my experience very robust and fast. I've been writing a lot of scrapers along the years and the best combination I found so far is requests / lxml.html / gevent.
It doesn't get any simpler than this IMO http://pastebin.com/hacxmAjV http://pastebin.com/hacxmAjV
- danneu 14y agoFor fun here's the Ruby+Nokogiri version and my attempt at the Clojure+Enlive version (3rd day learning Clojure). https://gist.github.com/danneu/5131596 https://gist.github.com/danneu/5131596
- thibauts 14y agoPretty nice. I'd like to see how it goes in a real world scenario with concurrency, manipulation of extracted nodes and general HTTP post/auth/etc. I'm not sure about Ruby+Nokogiri but Clojure+Enlive may well be a great choice.