3 ms·

they arent anywhere near as ubiquitous as xml parsers

by daleharvey 18y ago

deleted 18y ago
[deleted]
fizx 18y ago
libxml2 has an html parser. Everything(ish) has a libxml2 binding. The real PITA about scraping HN is that there's hardly any id/class attributes and you end up with td:nth-child(3).
snprbob86 18y ago
For Python, use BeautifulSoup: http://www.crummy.com/software/BeautifulSoup/ http://www.crummy.com/software/BeautifulSoup/ Otherwise, try HTML Tidy: http://tidy.sourceforge.net/ http://tidy.sourceforge.net/
- eli 18y ago
  Hpricot for Ruby: https://code.whytheluckystiff.net/hpricot/ https://code.whytheluckystiff.net/hpricot/ But, yeah, Tidy does a darn job of turning just about any HTML into XHTML.
- weavejester 18y ago
  Ian Bicking seems to consider lxml superior to BeautifulSoup: http://blog.ianbicking.org/2008/12/10/lxml-an-underappreciated-web-scraping-library/ http://blog.ianbicking.org/2008/12/10/lxml-an-underappreciat...
  - snprbob86 18y ago
    Cool! Thanks for this.
icefox 18y ago
Funny story on that. For the Arora browser for a while I was pondering the best way to make a tool to import html bookmarks. I went a handful of different routes, but none of them were that good (I could import all my test data, but I wasn't sure I could input all users data) because of all the different html bookmark files that people generate, but I had an epiphany. I had at my fingertips webkit. So rather then trying to reinvent a bad html parser I just would just let webkit parse it and walk it with javascript. The code ended up being something like QWebView view; view.load("file://local/file/foo.html"); print view.mainFrame()->evaluateJavaScript(javascriptCode); Full source here: http://github.com/Arora/arora/tree/master/tools/htmlToXBel http://github.com/Arora/arora/tree/master/tools/htmlToXBel With QtWebKit working on all platforms and having a really nice api making things like this is a snap and easy to do.

3 ms·

they arent anywhere near as ubiquitous as xml parsers

by daleharvey 18y ago

they arent anywhere near as ubiquitous as xml parsers

deleted 18y ago
[deleted]
fizx 18y ago
libxml2 has an html parser. Everything(ish) has a libxml2 binding. The real PITA about scraping HN is that there's hardly any id/class attributes and you end up with td:nth-child(3).
snprbob86 18y ago
For Python, use BeautifulSoup: http://www.crummy.com/software/BeautifulSoup/ http://www.crummy.com/software/BeautifulSoup/ Otherwise, try HTML Tidy: http://tidy.sourceforge.net/ http://tidy.sourceforge.net/
- eli 18y ago
  Hpricot for Ruby: https://code.whytheluckystiff.net/hpricot/ https://code.whytheluckystiff.net/hpricot/ But, yeah, Tidy does a darn job of turning just about any HTML into XHTML.
- weavejester 18y ago
  Ian Bicking seems to consider lxml superior to BeautifulSoup: http://blog.ianbicking.org/2008/12/10/lxml-an-underappreciated-web-scraping-library/ http://blog.ianbicking.org/2008/12/10/lxml-an-underappreciat...
  - snprbob86 18y ago
    Cool! Thanks for this.
icefox 18y ago
Funny story on that. For the Arora browser for a while I was pondering the best way to make a tool to import html bookmarks. I went a handful of different routes, but none of them were that good (I could import all my test data, but I wasn't sure I could input all users data) because of all the different html bookmark files that people generate, but I had an epiphany. I had at my fingertips webkit. So rather then trying to reinvent a bad html parser I just would just let webkit parse it and walk it with javascript. The code ended up being something like QWebView view; view.load("file://local/file/foo.html"); print view.mainFrame()->evaluateJavaScript(javascriptCode); Full source here: http://github.com/Arora/arora/tree/master/tools/htmlToXBel http://github.com/Arora/arora/tree/master/tools/htmlToXBel With QtWebKit working on all platforms and having a really nice api making things like this is a snap and easy to do.