3 ms·

it would be nice to be able to use an xml parser to screenscrape, i mean, missing alt tags dont really matter, but not having to write silly regexs to strip out

it would be nice to be able to use an xml parser to screenscrape, i mean, missing alt tags dont really matter, but not having to write silly regexs to strip out the data you need over simple xml queries is a plus

staunch 18y ago
Use any decent HTML parser.
- daleharvey 18y ago
  they arent anywhere near as ubiquitous as xml parsers
  - deleted 18y ago
    [deleted]
  - fizx 18y ago
    libxml2 has an html parser. Everything(ish) has a libxml2 binding. The real PITA about scraping HN is that there's hardly any id/class attributes and you end up with td:nth-child(3).
  - snprbob86 18y ago
    For Python, use BeautifulSoup: http://www.crummy.com/software/BeautifulSoup/ http://www.crummy.com/software/BeautifulSoup/ Otherwise, try HTML Tidy: http://tidy.sourceforge.net/ http://tidy.sourceforge.net/
    - eli 18y ago
      Hpricot for Ruby: https://code.whytheluckystiff.net/hpricot/ https://code.whytheluckystiff.net/hpricot/ But, yeah, Tidy does a darn job of turning just about any HTML into XHTML.
    - weavejester 18y ago
      Ian Bicking seems to consider lxml superior to BeautifulSoup: http://blog.ianbicking.org/2008/12/10/lxml-an-underappreciated-web-scraping-library/ http://blog.ianbicking.org/2008/12/10/lxml-an-underappreciat...
      - snprbob86 18y ago
        Cool! Thanks for this.
  - icefox 18y ago
    Funny story on that. For the Arora browser for a while I was pondering the best way to make a tool to import html bookmarks. I went a handful of different routes, but none of them were that good (I could import all my test data, but I wasn't sure I could input all users data) because of all the different html bookmark files that people generate, but I had an epiphany. I had at my fingertips webkit. So rather then trying to reinvent a bad html parser I just would just let webkit parse it and walk it with javascript. The code ended up being something like QWebView view; view.load("file://local/file/foo.html"); print view.mainFrame()->evaluateJavaScript(javascriptCode); Full source here: http://github.com/Arora/arora/tree/master/tools/htmlToXBel http://github.com/Arora/arora/tree/master/tools/htmlToXBel With QtWebKit working on all platforms and having a really nice api making things like this is a snap and easy to do.
jrockway 18y ago
Try cloure-html. It turns invalid HTML into valid XHTML SAX events (which can be made into a DOM with cxml-dom or cxml-stp). The architecture of closure-html and cxml are really nice, even if you don't use Lisp. You should take a look.

staunch 18y ago

Use any decent HTML parser.

daleharvey 18y ago
they arent anywhere near as ubiquitous as xml parsers
- deleted 18y ago
  [deleted]
- fizx 18y ago
  libxml2 has an html parser. Everything(ish) has a libxml2 binding. The real PITA about scraping HN is that there's hardly any id/class attributes and you end up with td:nth-child(3).
- snprbob86 18y ago
  For Python, use BeautifulSoup: http://www.crummy.com/software/BeautifulSoup/ http://www.crummy.com/software/BeautifulSoup/ Otherwise, try HTML Tidy: http://tidy.sourceforge.net/ http://tidy.sourceforge.net/
  - eli 18y ago
    Hpricot for Ruby: https://code.whytheluckystiff.net/hpricot/ https://code.whytheluckystiff.net/hpricot/ But, yeah, Tidy does a darn job of turning just about any HTML into XHTML.
  - weavejester 18y ago
    Ian Bicking seems to consider lxml superior to BeautifulSoup: http://blog.ianbicking.org/2008/12/10/lxml-an-underappreciated-web-scraping-library/ http://blog.ianbicking.org/2008/12/10/lxml-an-underappreciat...
    - snprbob86 18y ago
      Cool! Thanks for this.
- icefox 18y ago
  Funny story on that. For the Arora browser for a while I was pondering the best way to make a tool to import html bookmarks. I went a handful of different routes, but none of them were that good (I could import all my test data, but I wasn't sure I could input all users data) because of all the different html bookmark files that people generate, but I had an epiphany. I had at my fingertips webkit. So rather then trying to reinvent a bad html parser I just would just let webkit parse it and walk it with javascript. The code ended up being something like QWebView view; view.load("file://local/file/foo.html"); print view.mainFrame()->evaluateJavaScript(javascriptCode); Full source here: http://github.com/Arora/arora/tree/master/tools/htmlToXBel http://github.com/Arora/arora/tree/master/tools/htmlToXBel With QtWebKit working on all platforms and having a really nice api making things like this is a snap and easy to do.

jrockway 18y ago

Try cloure-html. It turns invalid HTML into valid XHTML SAX events (which can be made into a DOM with cxml-dom or cxml-stp). The architecture of closure-html and cxml are really nice, even if you don't use Lisp. You should take a look.