4 ms·

When I need to extract data from HTML I use XPath. But to do so I have to use the combination of following tools iconv: necessary only when the page is NOT enc

by reirob 15y ago

When I need to extract data from HTML I use XPath. But to do so I have to use the combination of following tools iconv: necessary only when the page is NOT encoded in UTF-8 tidy: used to convert from HTML to XHTML which is XML. I call it as xmlstarlet: to extract data from the XML file using XPath. I find XPath a much better and much reliable tool for HTML data extraction.

masklinn 15y ago
> When I need to extract data from HTML I use XPath. But to do so I have to use the combination of following tools I just use lxml.html (handles 99.999% of the HTML out there, add beautifulsoup's UnicodeDammit[0] for wonky encodings) and then use lxml's built-in xpath support on top of that. Plus extra bonus, if the datamining paths are simple enough you can use CSS queries instead of XPath. [0] http://lxml.de/elementsoup.html#using-only-the-encoding-detection http://lxml.de/elementsoup.html#using-only-the-encoding-dete...
- reirob 15y ago
  Thanks for pointing this out. I forgot to precise that I am using XPath on Unix/Cygwin command line or in Shell scripts. Here an example of extracting the Hacker News title;url;points;user data: curl -s http://news.ycombinator.com/news | tidy -quiet -asxml -numeric -utf8 -file /dev/null | xmlstarlet sel -N x=http://www.w3.org/1999/xhtml -t -m "//x:tr[x:td[1][@class='title']]" -v "normalize-space(x:td[3][@class='title']/x:a)" -o ";" -v "x:td[3][@class='title']/x:a/@href" -o ";" -v "str:tokenize(following-sibling::x:tr[1]/x:td[2]/x:span[1], ' ')[1]" -o ";" -v "following-sibling::x:tr[1]/x:td[2]/x:a[1]" -n | xmlstarlet unesc
  - sukuriant 15y ago
    Could you put some newlines in that? For some reason, my browser (IE9) is rendering that construct as a VERY long line and extending the page to match it. (no code-specific scroll-bar for me)
    - reirob 15y ago
      curl -s http://news.ycombinator.com/news | \ tidy -quiet -asxml -numeric -utf8 -file /dev/null | \ xmlstarlet sel \ -N x=http://www.w3.org/1999/xhtml \ -t -m "//x:tr[x:td[1][@class='title']]" \ -v "normalize-space(x:td[3][@class='title']/x:a)" -o ";" \ -v "x:td[3][@class='title']/x:a/@href" -o ";" \ -v "str:tokenize(following-sibling::x:tr[1]/x:td[2]/x:span[1], ' ')[1]" -o ";" \ -v "following-sibling::x:tr[1]/x:td[2]/x:a[1]" -n | \ xmlstarlet unesc
      - sukuriant 15y ago
        Thank you.
- yuvipanda 15y ago
  It's also much faster than BeautifulSoup, and uses much less memory. Matters if you're running in a memory/CPU constrained VPS like I was :)

by reirob 15y ago

masklinn 15y ago

> When I need to extract data from HTML I use XPath. But to do so I have to use the combination of following tools I just use lxml.html (handles 99.999% of the HTML out there, add beautifulsoup's UnicodeDammit[0] for wonky encodings) and then use lxml's built-in xpath support on top of that. Plus extra bonus, if the datamining paths are simple enough you can use CSS queries instead of XPath. [0] http://lxml.de/elementsoup.html#using-only-the-encoding-detection http://lxml.de/elementsoup.html#using-only-the-encoding-dete...

reirob 15y ago
Thanks for pointing this out. I forgot to precise that I am using XPath on Unix/Cygwin command line or in Shell scripts. Here an example of extracting the Hacker News title;url;points;user data: curl -s http://news.ycombinator.com/news | tidy -quiet -asxml -numeric -utf8 -file /dev/null | xmlstarlet sel -N x=http://www.w3.org/1999/xhtml -t -m "//x:tr[x:td[1][@class='title']]" -v "normalize-space(x:td[3][@class='title']/x:a)" -o ";" -v "x:td[3][@class='title']/x:a/@href" -o ";" -v "str:tokenize(following-sibling::x:tr[1]/x:td[2]/x:span[1], ' ')[1]" -o ";" -v "following-sibling::x:tr[1]/x:td[2]/x:a[1]" -n | xmlstarlet unesc
- sukuriant 15y ago
  Could you put some newlines in that? For some reason, my browser (IE9) is rendering that construct as a VERY long line and extending the page to match it. (no code-specific scroll-bar for me)
  - reirob 15y ago
    curl -s http://news.ycombinator.com/news | \ tidy -quiet -asxml -numeric -utf8 -file /dev/null | \ xmlstarlet sel \ -N x=http://www.w3.org/1999/xhtml \ -t -m "//x:tr[x:td[1][@class='title']]" \ -v "normalize-space(x:td[3][@class='title']/x:a)" -o ";" \ -v "x:td[3][@class='title']/x:a/@href" -o ";" \ -v "str:tokenize(following-sibling::x:tr[1]/x:td[2]/x:span[1], ' ')[1]" -o ";" \ -v "following-sibling::x:tr[1]/x:td[2]/x:a[1]" -n | \ xmlstarlet unesc
    - sukuriant 15y ago
      Thank you.
yuvipanda 15y ago
It's also much faster than BeautifulSoup, and uses much less memory. Matters if you're running in a memory/CPU constrained VPS like I was :)