6 ms·
This is only tangentally related to the article, but on scraping HTML in general: If you're a Python user, use lxml for it. I know most content on the web will
by sho_hn 15y ago
This is only tangentally related to the article, but on scraping HTML in general: If you're a Python user, use lxml for it. I know most content on the web will tell you to use BeautifulSoup, and lxml is something you've only heard of in connection with reading and writing XML, but lxml actually has a lovely HTML sub-package, it's faster than BeautifulSoup, and it's compatible with Python 3. I've gotten lots of good mileage out of it (and no, I'm not a developer on it :)).
- waitwhat 15y agoFYI, BeautifulSoup is actually dropping its own parser entirely for the next version, in favour of being a wrapper around lxml/html5lib/html.parser http://www.crummy.com/2012/02/02/0 http://www.crummy.com/2012/02/02/0
- sho_hn 15y agoAh, interesting, thanks. I will confess that I haven't followed the latest round of developments in BeautifulSoup - I abandoned it last year when I made the switch to Python 3, and the previous effort to port BS to Python 3 had stalled/failed at that point (looks like they're back on track on that one now, too). Then I found I didn't really miss BS while using lxml.html - not sure I see a point in putting it on top of it.
- wladimir 15y agoI think the idea behind BS is that it saves some typing and thinking for common scraping tasks. In that light, it fits perfectly on top of some other (faster) parsing library.
- sho_hn 15y agoYup, but all of the stuff in the Quick Start section of http://www.crummy.com/software/BeautifulSoup/bs4/doc/ http://www.crummy.com/software/BeautifulSoup/bs4/doc/ has more or less close equivalents in lxml.html. That being said, I think it's great that he's still maintaining BS, and porting it to Python 3 in particular - it keeps existing code working and will allow more people to make the switch. And a responsible and committed maintainer is good advertising for a package in itself, of course.
- sanxiyn 15y agoAnother tangentally related recommendation: Use Scrapy if you need something more than one-shot script using BeautifulSoup or lxml. http://scrapy.org/ http://scrapy.org/
- kanzure 15y agoI admit to using BeautifulSoup, lxml, scrapy, nokogiri and mechanize. But these days I just stick with PhantomJS. Just load the DOM with WebKit. http://phantomjs.org/ http://phantomjs.org/
- luser001 15y agoI'm even lazier. :) I used the pjscrape wrapper around phantomjs. https://github.com/nrabinowitz/pjscrape https://github.com/nrabinowitz/pjscrape
- chernevik 15y agoI need to build / find a tool for parsing EDGAR filings for their financial statements tables (not so bad) and parsing those financial tables into usable information (pretty bad, the tables have very bad and often inconsistent HTML layouts). Any suggestions to where I should be looking? Python? XSLT?
- jessedhillon 15y agoContact me, email is in my HN profile.
- anand_nalya 15y agoUse HTML Tidy + XSLT.
- wslh 15y agoI use HTMLUnit for scraping javascript based sites. Since Python is a lovely language and HTMLUnit is written in Java I use Jython. You can take a look at one of my articles on the subject: http://blog.databigbang.com/web-scraping-ajax-and-javascript-sites/ http://blog.databigbang.com/web-scraping-ajax-and-javascript...